There is a group here (Setanta, Glitterbag, Izzy, Chai et. al.) who like to think they run things. When they get upset with a user, that user will get down thumbed on even non-controversial posts. Anyone who has been targeted by them definitely notices it.
I strongly believe that Setanta and Chai do not
participate in any votedown abuse.
I would guess that Glitterbag does not participate in votedown abuse either, although I don't have strong feelings on that.
I believe that izzythepush (note that there are other izzys here on a2k, so it is better to be specific) used to participate in votedown abuse. However, I am unsure if he still does this.
I believe that most of the votedown abusers are not people who post actively.
I of course have no evidence for any of this. But I claim higher than average skill at drawing conclusions.
It is certainly reasonable to conclude however that the same things that irritate the "in crowd" as you call them, also irritate the downvoters whoever they are.