I made my bed and now I have to lie in it....Newbie Here!!

I’m not sure where to start from such a long drawn out story so I’ll try to keep it as short as possible. I’ve been having an emotional/physical affair with another man for a little over a year. He is engaged and has two children, I am also in a long term relationship of 4 years and my child is grown. We are co-workers so that makes it even more risky and to add insult to injury, this is not the first time this man and I have had something between us. We first started our emotional affair 5 years ago. It lasted 2 or so years until his then girlfriend found out. She almost left him; he almost lost his family and home because of it. We went no contact, cold turkey and never spoke again. I was fine with it all. Then at end of 2011, he transferred to my building with his job and within a month, we were talking again. Things were fine at first, just friendly and no crossing boundaries. We were both playing ‘hard to get’. Then one day, it changed. We went from 0 to 60 and took a complete 180. We started having our sexual conversations like before and opened a shared email between us both to correspond and send pics/videos. We have not slept together, the physical contact we’ve had is just kissing like teenage kids. We’ve skyped when he was away on a business trip and the videos/pics have left nothing to imagination. We have written letters to each other and gone into great detail, but we are both too scared to consummate our ‘relationship’. Here is the issue….. Things are very up and down, we are both emotional wrecks most of the time and we’ve many failed attempts to stop whatever it is were doing and work on our own relationships. It never works. One of us always breaks the NC or we stick it out for a little while and one of us conveniently runs into the other and it starts all over again. He claims he is happy in his relationship and makes it very clear that he is not leaving his family. He has told me over and over that he doesn’t know what his life would be without his children (both under the age of) and that he has worked too hard for what he has, he’s scared to lose it. Never once has he told me he loved his fiancé. NEVER. He is very ‘wishy-washy and he goes back and forth, one day he is very pushy, the next he isn’t. Me on the other hand, I’m very unhappy in my relationship and have been for a very long time. I have jumped hurdles for the man I’m with and tried so many ways and times to get us back on track, but he refuses to work on us. Ours is more a relationship of convenience than love. It’s sad but true and I know what I must do to fix that, but it’s also scary for me to leave because I still hold on to hope that maybe one day he will see that I wasn’t a horrible person and that we could have been great together if he had given me a chance. Yes I am a liar and a cheater and I’m well aware of that, I’m not justifying my behavior at all but a person can only take so much before they seek attention elsewhere. Now my affair with the other man has reached a very difficult point. His fiancé started to see that something was different in him and that things were getting distant so he ended it again at the beginning of the year. We went 5 weeks NC, the longest time ever. Then he contacted me again. He claims that he values my friendship and wants me to remain in his life, but the rest has to stop. I’m not sure if I can do that. I think I fell for this man long ago and I’m just too scared to say it out loud. I’m intensely frightened by my attachment to someone I haven’t even been fully intimate with and the thought of letting him go physically makes me ill. He won’t admit to me what he feels about me, just says that he doesn’t want to complicate things more by saying something thats not relevant if it can never be. I’m lost….I really have no idea where to turn or what to do. I don’t know how to live with either of these men in my life at this point and I feel like I’m suffocating. Any insight or thoughts would be helpful since I’m going thru this all alone……