Cheating on husband with guy at work...

Here's a little background, I feel like I'm married to someone who's not the right person for me. Do I love him? Yes. But I'm not in love with him. He's honest, loving, and caring. But he's also very criticizing, very blunt, and very controlling at times. I'm a very sensitive person, so our personalities clash. I also feel like we were rushed into our marriage because I got pregnant so quickly once we started dating. Fast forward to now, I've been unhappy at home for quite some time. I left him about a month ago, stayed with my mom for a week, and he ended up talking me into coming back home and trying to make things work. But the weekend before I came back, I hooked up with this guy at work that I reeeeeally like a lot. He's always flirted with me, so as soon as I was "single" I called him up and we were both really excited to get together. The night was amazing. Well, 2 days later is when my husband talked me into coming back home so I told the guy at work that I couldn't see him anymore. He was pretty upset. I couldn't stop thinking about the night we had together, and it bothered me that I upset him so I told him that I still wanted to see him. He was all for it. We hooked up for the second time today on our lunch break, and it was just as amazing as last time. I can't get this guy out of my head. I know it's so wrong to do to my husband, but I'm scared that if I leave him and try to be with the guy at work then things aren't going to work out and I'll end up alone. The guy at work doesn't even know that I would love to be with him (not just for sex) and I don't want to tell him that unless I'm sure about leaving my husband. Him and I just click so much, unlike my husband and I ever have. I'm just scared to be alone. My husband gives me the security I need, but I'm so in love with this other guy. I think I would scare him off if I told him all of this. Does anyone have any insight or experience with something similar?