No, young man is rarely derogatory. I suppose it has been used by an adult scolding a child as in "Listen here young man", but it is not a derogatory term in general. It is often used as a sweet term as in "he is such a nice young man".
I would love to be called young man. (It has been a long time).