Married and heartbroken after affair

So I will start out saying I have been married to the best guy anyone could ask for for almost 20 years...he has been my everything and my best friend. We have 2 kids age 5 and 13. I have always thought we had the perfect marriage..no fighting, good sex, friendship. I work in a coffee type shop in a small town and everyone knows everyone. A daily customer who is also married whom I have known for the last 20 years started flirting with me and I flirted back. This led to him coming into the store multiple times a day to see me and talk and flirt. We had fun with each other and this continued for awhile. We exchanged cell phone numbers innocently (he owns a local business that I use and I needed to send him a reminder text) I sent the reminder and he replied and things started getting flirty. We continued to text nonstop for weeks along with still seeing each other daily at the store. We talked about everything and anything when texting including our marriages, family, kids etc. We also started sexting and talking a lot about sex and getting together. In almost 25 years together with my husband I have never even looked at another man before now. This guy makes me feel things I have never felt before..my knees literally get weak when I see him...I'm so attracted to him its crazy but even more I think I have fallen in love with him...his heart, his soul and his body. I am 46 and he is 53. He is unhappily married and I am not. He is a respected man in our community and a businessman and he has ties to my family via business and friendship including a casual friendship with my husband. After weeks of texting we decided to do a quick meeting after hours at his office so we could be alone. We met did some talking, hugging and kissing and mild over the clothes groping. It was amazing and left us both wanting more. We decided meeting at his office was too risky and a week later we got a room to again be alone but we had agreed no sex. We met for 2 hours and ended up having sex...the most amazing sex that felt way more like making love than sex. The next day we took a huge risk and met again at his office because we were desperate to see each other. We continued to text nonstop when we could. We would stay up all night after or spouses went to bed to just text and talk. He was the first to use the love word but when he did I told him I loved him right back. This scared us both and the intensity of our feelings shocked us. A couple of days after I started noticing him backing off and becoming distant...not answering a text right away and just weird. When I finally asked him he said he was confused and needed time. I backed off and didn't text him or see him for days. When I finally did text him and asked if we were over his response was maybe with no explanation. I cried for days and walked around with a broken heart which was very hard to hide. Then we started texting again and he said he was talking to someone else and confused. This of course made me angry and hurt and he heard it from me and I walked away with no intention of talking to him again but I found myself unable to not answer his texts. During this time he had told me he had started filling out divorce papers and had told his wife he wanted a divorce. (I know it's true because his wifes facebook page is open to the public and she posted about it). During all this time he kept telling me I was where I needed to be with my family, nice guys finish last that sort of thing. I finally said no more it was too hard to talk to him and I was done and every time I do that he asks me not to go. When I ask him about the other person he is talking to he wont talk about it. If I mention how I feel about him he gets mad, if I ask questions he gets mad but if I say I'm done and try to walk away he wont let me and says he wants to be friends. That usually works for a day or so before one or both of us says something that crosses the line or makes the other one mad. I suspect or maybe I'm just fooling myself that there is no other person he is talking to or seeing and that he is scared and worried about the hurt this will cause both of us and our families not to mention our reputations in a small town. I had previously told him I would leave my husband for him and I would but I think the guilt may be too much for him. We decided to stop all contact with each other and have not been texting but when he drives by the store I work he can clearly see my car and he continues to come in..there are 2 other stores in town he could stop at. we try to be normal with each other but it's so hard to see him and look into his eyes. We always end up looking in each others eyes and saying a lot without talking. I'm so confused by this man...I know what I want and it's him but he keeps sending mixed signals...maybe I'm just an idiot that fell hard for a player but thats not what my heart tells me. I don't want to be without him and my heart is completely broken. I would walk away from a good life for him if he asked. How do I move on and resume my life like I don't love him...how do I live with the guilt of it all....I don't lniw what the truth is for either of us anymore...I only know I hurt like never before. Thanks for listening to me pour my heart out and please no name calling...I know I'm going to hell.