@studydreams,
You can't stop people talking as
they want to.
That last bit is to point out that you can choose friends who don't want to - who are genuine, and who care - and who are true friends.
You can lessen the affect on you by growing as an individual - growing your self esteem, your self respect, your individual power, your compassion, your empathy, your assertiveness etc...as you grow, others opinions that disagree with how you see yourself are considered, but not taken to heart (as in detrimentally), in how you wish to grow from here.
That said - we are all social creatures, and achieving the noticeable effects of the above is difficult (but not impossible).
Another way is to learn how to confront such rumours - respectfully.