@tuesday2018,
1) This wasn't accidental. You're adults and it was deliberate. Own it.
2) Your 'good times' with your husband DO NOT outweigh physical and verbal abuse. The sweet-talking side is another facet of abuse. It gaslights you, to get you to believe that (as you are saying), it's "not so bad". IT IS.
You can bullshit yourself all you like, but don't bullshit us.
3) Staying in a marriage for the kids is a horrible thing to do to not just you, your affair partner and your husband, but also your kids. Why? Because it teaches them that love doesn't matter (and apparently physical autonomy doesn't, either), in the face of the almighty baby. It also gives them a responsibility for your marriage that you (and your husband) are relinquishing. Please don't pin inertia about making necessary changes on them. It's unfair to them.
Please, I
urge you to talk to a counselor and get tools to leave this marriage. Abusers do not get better without treatment. They are ill and need a doctor's care. Your love and subservience will not cure him. Instead, they will only make him bolder.
Do you honestly want your children to witness your husband smacking you around? And what sort of a lesson do you think they will pick up from
that?
Notice I am saying nearly nothing about your affair. It's not meaningful in the grand scheme of things. More important is for you and your children to be
safe.
People have left with less provocation and less money and resources than you have. And they have turned out all right, as have their children.
Get out while you still can.