Pride plays a part in it, too, as separate from misdeeds -- the last half of Survivor was all about that. Hubris, pride goeth before the fall, whatever. 3 or 4 in a row proclaimed a variation of "I'm in the driver's seat!", and were then voted off. Hah!
It wasn't so much that they did something wrong and deserved to be voted off, as that their pride was their downfall.
Hubris is an old, old thing (gosh I'm eloquent this morning

) and I think it probably serves a function in terms of keeping society from being too too stratified. Pride goeth before the fall, so be humble. If you're humble, you don't put on airs, don't have a house too much fancier than you need, share your wealth if you have it. Those are all things that benefit society.