Confrontation and respect

Hi , first of all i am a 30 y.o man , single , PhD and an engineer. My problem is i am a pacifistic person just like my dad is contrary to my mother . Me and my father are not the kind of men who would stay strong and never back up when in conversational confrontation with a man . I have felt it when we worked with a contractor that was somehow not an easy person and radiated that vibe of a tough man if i may say. We used to back down pretty easily and this frustrates me . In every confrontation, i try and find friendly resolution with men and i am easily convinced . I want to change this cuz i feel this is a weak trait and shouldn't be to prominent in strong personalities . However, i am not weak per say , in example i had arguments with my boss who was unjust to me and did not back down from my rights .
I have also noticed that men who hold eye contact are more likely to make me back down even more so when they install silence in the interaction. I think manly men in general exhibit this type of behavior which i should adopt. Also respectful men laugh and smile less , now i know doing the contrary should in theory make people like you but i believe it will bring them subconsciously to disrespect you later , as evident by people throwing jokes at me . I have had many bad episodes of bullying in my childhood and even now as a grown up man i get picked at , i am a trustworthy hold to my word nice guy why do i get disrespected often ( is it because of my physique slim guy with 1.70 m height ? or is it because of how i carry my self ?

Thank you