I think mismi's approach is best.
In effect, thank you but unable to attend. Briskly - so there's no opportunity for questioning.
Foxy's right. Anything other than going to the party is other plans, so if questioned - other plans. Again, briskly in a way that any comment/question from the other parent isn't encouraged.
I've got a colleague who responds to questions about other plans in a way that makes it clear the questioning is rude, without being obviously rude herself. She's got a gift I wish I had. Something in her tone just discourages any question/comment. Amazing.