Love/ life advice

I need relationship advice and would appreciate an honest answer. I had an affair on and off the last 3 yrs (not that it's okay to have an affair, but I didn't have sex in years w my husband and he wouldn't go to counseling). I was in love w the person I had an affair w but it's been difficult. I broke up w him the last time because he refused to pick up the phone when I was physically assaulted because he didn't want his friends or family to know we were back together (they don't like me because I had a mental breakdown in public a few yrs ago when I found out he was lying and cheating on me). He came back in my life and he said that he wanted to make me a priority and should never be embarrassed to be w me because of his family..we were doing great, but when he came up for the only 2 weeks I would see him over the next few months, he saw me 3x in 10 days and napped/ did online work the last 2 times while he partied w his friends 4 of the nights for hrs. I got pretty upset when I asked him to hang out 1 day and he chose to go to the pool w a friend he saw 4x last week and was going to see a few more times the following wk instead of me. He said I was suffocating him and was a jealous, toxic person who didn't want him to talk to his friends or family. I personally didn't ever feel jealous when he is w anyone but don't feel like spending time w me really mattered unless it was convenient. He stopped talking to me completely and I begged for him back and told him I was sorry for getting upset, but he ignored me (which has happened several times in the past for months). I also found out he had a few online dating apps while w me but not sure if active. I just feel awful and have lower self-esteem than ever. Maybe this is karma, but I feel so broken. My husband wants me even after the affair, but I'm still in love w the guy who doesn't want me and need to fix myself.