I don't think he is trying to end it... what he is saying is far more difficult than simply saying "I want to end it". I also don't think Punkey is correct. You were OK with casually dating, there is no reason he has to say anything for this to continue. It sounds to me he is trying to be honest.
My advice... you should figure out what you want. Do you want to have serious relationship with him? Do you want to just continue with the casual relationship you have now? Will you continue to be OK with him dating other women. If you are clear about what you want, it will be easier to have the conversation.
Then I would be direct... you can say "I want something serious, but you need to stop running around". You can also say "We can keep seeing other people if you need space...". Or you can say "I'm done here buddy, have a nice life".
You deserve to know exactly where the relationship stands. The way to figure it out is to talk directly with him.