@chai2,
I don't see it quite that way. I don't see anything at all wrong (in fact I think it's a positive thing to do) to approach somebody and say something like:
"I didn't say anything at the time, but you know what? It really hurt my feelings when you said X last year on my birthday."
It's best to clear the air, I think. Even if that person says, in response, something like:
"Yeah? Well, let me say it again. I still think X."
My policy is to get the cards on the table and see where people really stand. Best if done at the time, but, better late than never if it's bothering you.