@dmo1124,
Here's an idea - and it may be difficult to do, now that the genie, as it were, is out of the bottle. But hear me out.
I think you both jumped into bed far too quickly, and I think it is messing with both of your heads. For him, the effects are pretty obvious (and the usual caveat applies, see a doctor, etc.). For you, it may be pulling you into a deeper infatuation than, perhaps, is called for at this stage of the game.
This does not mean that you will not last.
But my suggestion is, maybe, time out, move away from a sexual relationship, at least for now. The pressure, I think, is to perform and be wonderful. I think that's making him nervous and it's, like I said, possibly making you feel more than you should.
So cool it for a while. It does not mean you cannot see one another. Rather, explore the other aspects of coupledom. Go out on dates. Talk. Have fun. And see where it goes. You are looking for long-term, so you can afford to be patient, yes? And he was, at least originally, just looking for fun. So let it be fun for a while. If it is meant to get serious, it will. But in the meantime, I think you're both jumpin' the gun here.