thanks everyone for the responses.
Osso-- yeah, I thought "huh?" too...I
am very pale, but that is my Scotch-Irish heritage and I'm proud of it dammit! But he was perfectly convinced that he can see a grey pallor in a vegetarian's skin.
Oh, and I'd say yes, there
is such a thing as vegetarianphobia, I see a lot of it because I live in a small town where there are lots of ranchers and cowpokes who just can't imagine it. I've known several people who have never actually met a vegetarian before me so it seems really unusual to them.

Nice people like you, osso, who tend to be more accepting of people, are usually surprised to find that anyone would even think of these things...
Littlek-- I'd forgotten you were veggie, nice to be reminded! Hi! I don't meet 'em a lot IRL, as I said... Do you think there really is a vegetarian "look" that's noticeable? I myself have never noticed it, but I have seen a lot of Americans in general who look more than a little unhealthy...hmm...
Olga-- I have often wondered if that kind of response is based on feeling threatened, although since I don't tend to think of myself as "virtuous" it seems a bit ridiculous!

But I am interested to hear that this is an impression that you've gotten too; it's something I've wondered about but hardly dared to believe since it would imply a little bit of doubt on the carnivore's part... Sad how often we reveal our insecurites by being aggressive towards others.
So, on the whole, would everyone's advice be to just ignore it? That's my natural response to things like this, but then I get so agitated about it later and feel stupid for not having said anything. I had been under the mistaken impression that I wouldn't really ever be around this guy again, until I heard about this two weeks ago, aargh.