I can definitely see the issue of a tradeoff. You don't often see people for whom there is no Summer, unless they are clearly very sick (e. g. cancer patients) or very elderly or extremely thin (read: anorectic). Everybody else seems to wear shorts when it hits 90 (or the equivalent) or if they're forced to wear a suit they are clearly really uncomfortable. We may quibble about the actual setting of the thermostat but I think most people think 90 and think, man, that's hot. 80 is often tolerable unless the humidity is outrageous, but 90 just never seem to be anything but hot unless you fall into the elderly, sick or thin bucket.
But cold! It's just ... different. I don't put on a jacket to get the paper, either, and I often don't put on much in order to get the garbage to the curb, but that's about my limit. Pretty much anything more than that, if we're talking 30s or below, I want covering beyond my street clothes. Something with wool or fleece or silk. The shorts-wearing people, even when they're working out, strike me as not just insane, but not of this earth.
They jog among us.