Emotional affair

I am currently involved in an Emotional Affair with a married man who says he is in a sexless marriage. I have tried leaving but he simply can't let me me go. When we started talking his intention was to divorce within two years. His children are all at college but living at home. He and his wife discussed the divorce with their children who were furious and so they went to counselling. He now says his wife has stopped being argumentative and critical but he is hoping to fix the sexless part of the marriage. He also told me he was no longer physically attracted to her. I told him it was time for us to stop talking and he was upset. He has become such a special part of my day. I told him we had to part but he gets upset. I don't know what to do.