Married and having an affair

So me and my husband have been having a few problems it's been going on for the last 9 months. We have had job loss, house move and also problems having a baby. During arguments my husband has many times told me he wishes he was never with me even before our marriage he used to say it, when I question it later on he admits that at the time he meant it. He's always moaned at me for spending time with friends if I was out longer than a couple of hours, male or female. On more than a few occasions he's tried to stop me going out with certain friends or rings me all the time when I am out. At the beginning of the year he told me that he no longer loved me but wanted to try and make it work so we stopped together. I started chatting to a guy i know and it turned out he was attracted to me. Mine and my husbands sex life started to get very poor, I feel like he has to make effort to sleep with me and there is no passion what so  ever I even have to ask him to kiss me. So as you guessed I started sleeping with the guy I knew. At first it was just sex because I was missing that kind of connection. The sex was amazing..... and still is. It is now 7 months down the line and I'm in love with him, I still get the butterflies when he holds and kisses me. I don't just want him by me I need him. I have never felt like I do about him with anyone else. I cannot explain it but it's just different with him. He's never pressured me to leave my husband but I do know given the chance he would spend the rest of his life with him. Over the last 9 months my husband has been pushing me away and I think when he told me he didn't love me was when my barrier went up and kept my distance. My husband has now turned round to me and told me he's falling back in love with me and knows he needs to make changes, but I really don't think I want to be with him. That love I felt for him a year ago has gone. I have so many what ifs what if it's a mistake walking away, what if I can never have a family because I've left it too long, what if it destroys him. I've been on the other side and it hurts so much. There are many things just little things that he has said that make me feel like we need to try and get along and like each other and it will never just be a natural connection. Things like maybe if we have a baby it will rekindle what we had, the kind of love that makes your heart jump doesn't exist, you never feel the same love for someone as you did when you first met, if he's saying them things surly he isn't truly in love with me?.....I just need advice my head is a mess and I do not know what to do