That's not be(e)ing very nice, Chris.
If I'm at a BBQ, I've been invited to it. The person who invited me to it absolutely knows that I am vegetarian and have been vegetarian for 28 years. Choosing not to accommodate that would be a pretty horrible thing to do.
I mean maybe, maybe not. Completely different argument though. You don't have a moral or ethical opposition to eating food that is vegetarian or supporting it as an industry. (Or, at least, you shouldn't and if you do then I have questions)
But they'd make sure that there were gluten free buns if there was someone who couldn't eat gluten, which is a more apt comparison.