you're right, it's definitely living with yourself.
it becomes odd when you take the long view on things; what does it really matter giving to (most) charities? everyone will die one day, and a vast majority of charities are only affecting how Comfortably people are living.
The only charities i think that one might be morally obligated to give, at least a little bit, to are health ones. if we possess the capacity to prevent others from premature death in exchange for giving up some comforts, then i think we should. though, you could just as easily say that we're all on borrowed time anyways, so what's the point in giving up enjoyment to delay the inevitable.
I guess, i'm just curious where other people think the balance lies between the two.