This is really hard to answer without a specific case.
However, I think that, usually the one that ends up hurt the worst is looked at as the victim, and the one hurt least is the perpetrator; UNLESS, it's a self defense case where the girl was coming at the guy with a butcher knife and he ends up hurting her in order to prevent his own death.
There's also the point of having to look at who crossed a boundary line first, and what was the reaction.
In a fight between Brothers...if one lands a punch, and the other does not retaliate. Then the one that got hit is the victim. If he retaliates and punches his brother back, they are both victims and perpetrators. Now, there's also the fact of what lead to the physical altercation. maybe the one that got punched was being very verbally abusive. At that point, he is the perpetrator; When the brother decides to punch him, he crossed a further line, and thus becomes the perpetrator and the other a victim.