As a single man who dates, I have a few thoughts.
You should be very clear with him about what you want. If you don't want to date him, for any reason, tell him clearly "I don't want to date you.". If you would like to keep a friendship, tell him that too. It is your decision and your responsibility to make a clear decision.
If you tell him, "I don't want to date you, please stop or we won't be friends any more", he will very likely back off. If he doesn't, just cut him out of your life.
A lot of problems in dating can be solved by clear direct communication.
You have three choices
1) Cut him out of your life. This solves the problem... but it means losing a possible friend.
2) Accept him as a friend, but make it clear you aren't interested in more. This often works. It means he will have to accept the limits on your friendship and you will have to make them clear. But many times this works.
3) Be open to dating him. Your post suggests that you are skeptical of any relationship, and that you have a list of "dealbreakers". It is possible that these things are preventing a potentially wonderful relationship from happening.
If going to the gym is really a deal breaker, you are cutting out what... 80% of possible partners, many of whom are in excellent health (you can get a lot of exercise without ever going to the gym). The more "dealbreakers" you have, the less likely you are to find a really great relationship.
You get to chose who you want to date for any reason you want. Everyone else has to accept your decision (it really isn't up for debate). That being said, if you are looking for a good relationship it means being emotionally vulnerable and it means opening yourself up to people that might not meet your standards of perfection.