@iamsam82,
Really, the question is why have women lost more hair in the course of our evolution than men have? If we go back to our evolutionary origins and think of the fact that we share ancestry with apes and monkeys, what is unique about humans is that we have generally lost hair. Hence, some of us hear the expression ‘the naked ape’. It’s actually not quite true to say we are hairless or the naked apes because we’ve kept hair in many parts of our body and partly because we haven’t actually lost the hair. It’s just become highly miniaturised, almost certainly because it gave an adaptive advantage in particular, helps keep cool, possibly related to having more upright posture, possibly relating to being much more active and running.
Now the question being asked is why aren’t we all equally hairless? Well to some extent, of course, people across the world vary enormously and then of course there's a big difference between men and women, and the answer most likely is to do with sex or selection. This is the second of Darwin’s great mechanisms, that selection doesn’t just work to fit people in the natural environment. It also comes about because individuals select the traits of the people they want to reproduce with, and probably, it’s the case that loss of hair, a key human characteristic, becomes more important in selecting for women because it makes them more attractive. So while all humans have been selected to lose hair, the process of how you choose a mate has extended much more in women than in men. So the real question perhaps is, why should that be attractive and I'm afraid that's probably something of a mystery.
link.
I was going to just say balls but this answer is better. Facial hair is a result of male secondary sex characteristics based on testosterone levels. Balls make testosterone. So, short answer: Balls.