Don't worry, things like these happen and someday if you happen to run into them, you will smile and shake your head thinking "what did i see in him?"
Honestly, I don't think you have anything to be embarrassed about. People have sex, sometimes the girl wants out, sometimes the guy wants out, you get my idea. In your case, I think he wants out. Let him go and if he comes back, be friendly but not too friendly.
Oh and please stop being the one to contact him, especially at crucial moments like after sex and when making the next date, it should be the guy who's doing that.