@Misskitten,
I don't think you're overreacting at all.
Here's an idea.
Make plans to go away to, say, a bed and breakfast or a beach or some not too terribly expensive vacation that is geographically between the two of you. So, for example, if you were in Boston and he was in Philadelphia, it would be somewhere between the Connecticut coastline and the Jersey shore. NYC is probably too expensive for what I'm thinking of.
This isn't meant to be a blowout vacation. It's more like a weekend or a three-day weekend.
Propose this trip to him but don't make the plans for him. You're giving him the right to have some input.
Observe how he behaves.
Red flags:
* OMG we'll be spending the weekend without ___, whatever shall we do?
* Oh, let's get a room for ___ to join us!
* Nah, ____ doesn't like ___ (whatever place you picked out).
Probably not red flags:
* Sorry, I can't; I've got to __ (some sort of errand that takes time, like working on his car).
* Gotta work that weekend, no can do.
* I promised my mother (or father, sister, etc.) I would ___ (something like clean the gutters, take them to a show, watch their kids).
Definitely not red flags:
* Sorry, but I'm not a fan of ___ (the town, the beach, the hotel), so can we do ____ instead?
* That's right in the middle of ___ (some big push at work, e.g. inventory, tax season, the Xmas rush, our big annual sale, the regional manager is inspecting the plant, etc.), so it's not a good weekend for me. How about ___ instead?
* I'm watching my spending right now (or calories). Can we do something less expensive/less fattening?
* I'm sorry if you've got your heart set on ___, but I've been there and it's terrible/it got terrible reviews on Yelp.
Someone who rejects every single plan is one thing, but someone who offers an alternative (even if you're not a big fan of the alternative) is making an effort.
See what happens. And if the little vacation happens, don't take it as an opportunity to talk about this gal or your relationship. Just go and have fun.