@maxdancona,
I was taking this a bit further, if you could hurt other people if you think their annoying what would happen ...
Alice would say to Bob "you are annoying". Then Bob would say to Alice "no, you are annoying". Alice would throw a punch, Bob would respond.
At the end, since Bob is bigger than Alice... Alice would admit that she was the annoying one, and then she would just shut up around Bob, or just say what Bob wants to hear, for the rest of time.
This would repeat indefinitely until Zack, the biggest and strongest badass of all, won. He would set up a pecking order and would tell all the other people the proper way to behave so that he wouldn't ever be annoyed.