I figure you've got four possible choices:
- Cancel. Oopsie, something came up at the last minute. Make it good, but also reschedule for another time, this time you'd be inviting (which means you'd be expected to pay, I might add)
- Go and either enjoy yourself as well as you can, or stew. E. g. deal with it.
- Express your objection, but be prepared to be told something like "they can't get a sitter" (so they should stay home, but I don't make these decisions) or "this is the only time we get to see them" (then that couple should've rescheduled their time with you) or "we have a coupon for dinner for seven" (in that case, they're kinda cheap but they could invite anyone else) or
- Something else, although I'm tapped in terms of suggestions
#1 is probably the easiest. You tell a little white lie, you go away for the evening or your mythological Aunt Sue is ill or whatever.
#2 is a wild card. It might go really well; the baby might just sleep, and the younger couple might be lovely people. Or it could be a festival of screaming, and the four of them telling in-jokes that you don't get, etc.
#3 is another wild card but it's a lousy position to be in. Your friends will undoubtedly feel that they have to give a reason in order to justify their actions. It also sets up that dynamic, that you say no, they tell you to reconsider because of X, and around and around you go.
Unless the dinner is intended to be utterly private, e. g. you're talking business with your friends or something like that, then either #1 or #2 is acceptable. If the dinner is really private, or you just can't stand their kids or the baby, then only #1 is really going to fly.
Surely you've got an Uncle Ferdinand in Poughkeepsie who needs your expert care, right?