@Angel24,
Honestly, I'd just let him "win" this one.
He probably can't stop you from going and he probably can't get you thrown out of the game but he surely could make things miserable for your kids if they have to witness their parents fighting in front of their friends.
I know you want to go and see them play but there will be other opportunities for that.
And please don't be all "I can't go because your dad's a dick" but just tell your kids it's probably best that you not be there while everyone is so upset but that you'll expect a play by play when they get home.