@charming,
I'm sure you're familiar with this expression (or one like it): having your cake and eating it, too. He has the best of everything right now.
If you want to leave, then leave, and retain a lawyer and make sure your children are provided for financially. If you want to stay, then lay down some ground rules as to what is acceptable. Decide what the penalty will be if he breaks those rules. You are, of course, not the only person in the relationship. He may have rules of his own. But if, say, sexting is out of bounds for you, and he keeps on doing it, then he's not showing you or your family a lot of respect.
There are also couples that open up their relationships, but that is evidently not an easy thing to do. It's rather rare. Often, when a couple does this, if they have not thought it through, it's because it's a last-ditch effort to right a sinking ship.
There is also the question of overall support for your children, and I am not just talking about the finances here. Have you got someone to help you care for them? Is there a fallback position for you if things end with this guy? Are you expecting to go back to work as soon as possible? I am not suggesting that you must stay together, but you may need to do a lot more preplanning in order to get good child care, particularly if your parents or siblings are remote or disabled or work full-time or are no longer among the living.