Really need advice

So over a year and a half ago I had a complete one night stand with someone that I've known for many years. This turned into texts abs pictures afterwards, with no intent on sleeping with him again. First let me tell you what happened. I'm a military wife. My husband had been very militant with me for very long. Our daughter who is five now, was very much planned for. As soon as she was born my husband freaked, and stated sleeping in the spare room because I was up with the baby, v abs he couldn't sleep. He left his mother in law to do everything for me after my c section. He was like this on many levels. He was cruel, dogmatic, and very harsh after our daughter. We moved to California for two years, where things were the same but I managed because we'd met some great people and friends there, but he still embarrassed me at times with his cruelness. He is very anti social, and if I were to have people over, he'd their a huge fit and make t miserable for me. Once he even stayed up in our room during party I'd thrown, and wouldn't come down. We were Louis lane and superman that year, so u costume was useless without him. I cried the entire party. Fast forward to our next move, ft Knox. Well my 17 year old sons father came to visit, and again my husband was rude to him, and embarrassed both me and my son. I will spare you the details, but he really showed his ass again. So I get a phone call from an old friend whom I was also friends with his family. After getting positive attention, and feeling important to someone we'd met for drinks and then you know what happened. Afterwards I was hooked on his attention. I didn't want to sleep with him again, I just wanted the nice positive attention. I love being a Mom, and I'm very much a good mother and I take care of my household and myself very well. I didn't want to be with this person, I just didn't want to stop talking. Eventually after texting, and phone calls, he who is much younger and single was getting upset because I was unavailable. So after he too began to be mean and rude to me, we called it off. He would on and off text me. I found out I was pregnant during all of this, and told my husband about the affair and that I was so unhappy. After a few days, he says he agreed how horrible he was being, and asked if I'd go to counseling. I agreed, since I had to right to point fingers after my affair. I chose to get am abortion which was a very heart wrenching decision since we are Christians, but I chose that. Fast forward and we still go to counseling and we are doing so well! The guy has still been texting me like crazy, and I keep turning him away, and I found out that he had a live in pregnant girlfriend. After he wouldn't take no for an answer, I started to get scared that he wouldn't stop, and was upset that I'd worked hard to fix my marriage and he's had zero repercussions. I called his gf and told her. He then told his sister all about us, and our affair and she will have nothing to do with me any longer. We were really close. I just need advice. I feel sick that she won't talk to me, I feel like now her whole family, that thought I was such a good person, now thinks I'm a whore. Please help me. Im really happy in my marriage now, and we've worked hard to get here. I have a good family, I just feel so sick.