For me it all comes down to who gets hurt and how much.
This is it for me, right here. I dislike Letterman because i've seen him make a fool of a guest or an audience member, sneering at them. Many, perhaps most, perhaps even the overwhelming majority of jokes achieve their humor at someone's expense, but for a written joke, which is not topical, there is no actual harm done to any real people. Increasingly, i am uncomfortable with laughter at the expense of others, and the older i get, the more compassion i feel. I first began to get uncomfortable with this sort of thing when i was an adolescent. We had a girl in class who was an obvious victim of abuse and neglect as a child, and who was sixteen and in the 8th grade. She came into the classroom one time, horrified because she had cut herself on the playground (i was in there for punishment, i had to write an essay on propagation). Like many children in her situation (mentally, or perhaps i should say emotionally, she couldn't have been more than seven or eight years old), she reacted to pain and sorrow much more than other children do, probably precisely because she could never expect sympathy or comforting. The home room teacher mocked her to me, would do nothing for her, not even to take her to the office. Then she turned to me an laughed and said: "You know, i think she might even understand what we're saying." I said something to effect that she didn't need to include me, and i left the room, for which i was never punished--i suspect the teacher didn't want the story repeated in the office. I probably shouldn't repeat this, but i am (more or less) anonymous here--but that incident recalled to me several occasions on which my mother had laughed at the distress of small children in situations which they didn't understand and which frightened them, or in their tears and sorrow. I was raised by my grandparents, not by my mother. There's a lot more baggage there, which ain't nobody else's business. However, i think that is the origin of my discomfort with "specific" humor, i'll call it. Laughing at the distress, misfortune or misery of an identifiable sufferer. I especially ache for small children and domestic animals who suffer, because in many situations, they just don't understand what is happening to them, and any understanding they do have of the situation is likely to be terror--which elicits horror from me.
Although i do appreciate other people's comments on this topic, do spare me any pop psychology on what i've said about my own situation.