Thanks for your reply.
The reason I've hesitated with this course of action is because the comments he makes are very sporadic. Sometimes he wont make a comment for a few visits then it'll come out of no where. Also, it's made to be a joke on his end but his fascination with the topic makes it clear to me that there is something else there and it taints the whole relationship for me. It's a very difficult topic to approach.
I was considering seeing him less, waning his presence from my life but not cutting him out from it. Maybe I am being sensitive about it but I can't lie about it making me uncomfortable and it'll always cause me to feel some amount of mistrust from now on, even if he stopped.