I'm vegetarian and I often feel like this at a restaurant.
The only time it is a problem is in Italy, when saying "no thanks" to the waiter's magical special of the day is an insult to his family's honor. I don't dare say that I'm a vegetarian in Italy though, or they will decide to skip the delicious pasta or gnocchi or risotto or pizza or whatever and bring me a plate of vegetables. ๐