@iamsam82,
My speculation on the matter is meaningless but if I were to hazard a guess...as women's role as keepers of the hearth, they didn't go out on the hunt - stayed at home base and protected and nurture the children. If women were at the home base/cave/shelter, they were where it was warmer as there was shelter and/or a fire. So evolutionarily, it was a useful characteristic to have a smoother face.
Here's another possibility: if while on a hunt or in the field, men had beards, their faces reflected less light and would be less visibe to a predator or another attacker.
As womens role was NOT to go out on the hunt, the 'need' of a beard was far less. And, as it was lighter and had far less coarse hair, evolution eliminated it as not needed. Out on a limb here, perhaps, but maybe men folk found females with smoother faces fairer and more desirable. After all, there was no battle over the razor.