@feblue88,
Well, Feblue,
It really doesn't matter if your lover isn't happy compared to your legal spouse. The end result is,,,you're not going to be happy when your husband files for divorce and seeks 100% legal and physical custody. You'll pay Child Support and see your children every other weekend.
If you're ok with that, then by all means, continue what you're doing. If not, break it off and do not have contact with him.