What's wrong with custom made lounge pants?

I bought some custom made lounge pants, slightly in the harem like style. Nowhere nearly like the MC Hammer pants though. All are of a thin, cool summer weight kind of fabric (ie silk, rayon, etc). Been told guys shouldn't wear stuff like this, they should wear cotton kaki's or shorts. And it should always be cottons, not silks, rayon or other thin fluid fabrics. Those are "feminine" fabrics.

I don't understand the concept of silk or rayon being feminine. Silk is silk; a natural fiber that's thin & fluid. So? The silk lounge pants don't make me feel womanly. Rather, they feel very cool, carefree and comfortable. And being thin & light in weight, the slightest breeze blowing makes them even cooler to wear. After wearing heavy denim jeans at work (requirement), it's like I am not wearing anything at all with the silk lounge pant. Rayon lounge pants are slightly heavier than the silk pants are. But still light in weight. I happen to like those fabrics. They have a nice feel to them.

Yet, on rare occasions when I feel brave enough to go out somewhere for a nice long walk I get asked some ridiculously dumb questions. "Why on Earth are your wearing that?" "You know how silly you look in those very baggy pants?" "Do you get your jollies from wearing breezy baggy pants?" That question is a total mystery to me. I have no idea on what they are referring to.

In mild breezes, yes, I do like the breezy feel as it brings out the soft feel of the fabric & the coolness to the garments. However, very stiff windy weather can be quite annoying as it sends the clothes into a terribly blustery & windblown state. Looks fine on women, but the feeling of comfy light weight summer clothes tormented by strong winds is just wicked.