This phenomenon is one of the joys of life. I have experienced it many times. It is good to be able to have your own perception of someone's beauty that may contradict conventional or widely held notions. I once read a magazine article about a bunch of rather thoughtless young men who used to go to London night clubs and dare each other to approach the "ugliest" girls they could see, and ask them for a dance. Extra points were awarded if the girl gave her phone number, and even more if a date was arranged. The journalist interviewed one guy from this group who said that he had dated an 'ugly' girl in this way and now they were engaged and were going to be married in a few months. He said "I don't see those guys any more.". There was a photo of this couple and you know what - she was really pretty. Trust your own feelings.
The French have an expression "jolie laide" which translates roughly as "beautiful ugly" and is used for a woman who is unconventionally beautiful. Of course there is much that is sexist in this notion, but as the literary critic Daphne Merkin put it, jolie laide is "a triumph of personality over physiognomy, the imposition of substance over surface." I think this is what I am trying to express. Substance rather than surface. An amazing thing to be able to perceive. A woman often thus described is Charlotte Gainsbourg, who I have always thought was gorgeous in every way.