If you're not in a committed relationship it's none of his business. It sounds like he wants it both ways, he can shag who he wants but you can't.
And I can think of plenty of other excuses as to why I'm late bringing a car back other than having sex. That's not even a legitimate excuse, if I lent my mate my car and he brought it back late because he stopped off to shag some bird I'd give him a slap. If he'd got stuck in traffic or had a puncture fair enough.