@Wonder1,
Your old boyfriend is selfish and manipulative, possibly even borderline abusive if you're even considering leaving your husband for him.
Get some counseling, alone, to figure out why you have such low self-esteem that you ever even considered it to be a good idea to leave someone responsible, loving, and kind (albeit not without problems) for a freeloader who only wants to sponge off child support.
And get marital counseling, and maybe a full medical checkup for your husband. Maybe something physical is affecting his behavior, performance, etc. You don't have to stay with your husband if you really don't want to, but you are by my calculations at least 35 and are way too old to be hanging around someone who is essentially a teenager - and not a nice one at that.