Gawd, that happens to me ALL the time! I think it's because I'm fairly quiet most of the time, so it surprises people that when I do come out with something, it's always a joke of some kind- they expect me to be too demure and shy to be kidding with every word that comes out of my mouth.
And, yeah, I have had the experience of feeling too bad after someone takes me all seriously to tell them I was just joking...
I also have coworkers that just don't have senses of humor, that's definitely part of the problem for me. I was looking at an editorial cartoon at work the other day, and I took it to show one of my bosses and her mom-- biggest mistake I've ever made at work...
It showed a huge violent mob protesting the Mohammed-in-a-cartoon thing, all with signs saying horrible things like "Kill them!" and "Cut off their hands!" and "Cut off their heads!" --and then two men in the foreground saying, "Just watch some political cartoonist twist this around to make us look bad!"
First they asked which part they were supposed to read (it was a ONE PANEL cartoon!) and then they took about three minutes to read the damn thing, and then after all that time, they both just nodded all seriously and said "Yeah, I heard about that." Jeez...