@lutherm,
If Adam's not there he'll be next door, that may be enougn to freak Ellie out.
You know what kids are like when there's a party going on, he'll be leaning over the fence etc.
Ellie's mother is dictating who you can and can't invite to your son's party. Invite them both, and if Ellie's mum wants to keep her away that's on her.
Why not ask your son to have a word with Adam? He could say something like you're only allowed to come if you're nice to Ellie, if not you can go straight home. (Which should be fairly easy if he's only next door.)
Your son will know Ellie has been invited so it's not you keeping her away.
I'm a single father as well, and I would make sure some women are there too.