@JML,
Maybe it's just me. but I don't think you are over reacting. Something in his tone was off-key to you. If you thought it was off-putting, trust your gut. Sometimes you really need to pay attention to what he says instead of creating all sorts of warm fuzzy's to excuse what he said. I'm sure he thought it was a compliment, but somehow I still hear 'You don't sweat much for a fat girl". He may be all sorts of wonderful, but he's not more wonderful than you.
It reminds me of a old conversation with co-workers at lunch back in 1976, One of the men (a drunk with three day stubble 20 years before it was deemed 'hip or cool', asked me if it was true that my husband and I had separated. I answered honestly 'yes we have'. Then he said 'Yeah, that's what I heard, I was going tot ask you out, but I heard you had a kid, I don't date girls who have kids'
I responded, 'that works out just fine because I don't go out with bums'. He was still a little hung over, so he wasn't quite sure how to respond. Trust me on this, even drug addicted women wouldn't spend time with him, but it never occurred to him that he really didn't have all that much going on. Hell, he wasn't even funny, and that's my major weakness. I love men or women who can make me laugh.