@carlafrade,
You know. He should never have moved in after only dating you and become exclusive after 16 weeks. I assume he is of simular age. You two never got the chance to date and have fun, this should go on for months before moving in together. Instead you got into a marriage.
The thing is, you can and have seen that it isn't working out as a "family" situation and so it won't.
Your kids are your life, I'm sorry of your past. But always put them first, they deserve love in a family and some guy that's been around for less than a year controlling the house is not providing that love.
As others have said stick to your guns.
Date, and date a long time... If I was to be able to give you future advice? Date and date a long time. Keep your home and family to yourself until you know 100% that from outings with the children and you as a "family" it could become a family before allowing someone to move in with you.
Be strong...
Even if you two got on "ok" if he moved out. You already know he can never move in again. So he's not the right guy for you and your family.