@tsarstepan,
Either that or the texture. Socks might be too thick, wash clothes too rough, etc. If you want to experiment, next time you buy some new underwear, throw them in before wearing them. If the scent is driving it, he will leave it be, but if he likes the feel, texture, etc he will take it.