.... Suddenly in the eyes of all others – his catholic family & ex-wife I became the other woman. ...
It wasn't sudden, and it wasn't something you just became. You already were. Please own up to that.
Be that as it may, what I want to know is why you've hung around this long.
This is a trainwreck. If it goes down the road you want it to, with a marriage, etc., you are going to have problems with his family and in particular his children. They have practically driven a stake into your front lawn and put up a big, flashing sign saying, "WE'RE GOING TO HATE YOU"
This guy dithers around and seems to be letting others dictate his life for him. And, sorry, but he's cheated already (as have you). What makes you think that, if the going gets rough, he won't potentially turn around and do the same?
Don't live your life waiting for the other shoe to drop. Don't waste your time waiting for this damaged prize.
Go date some more. And get some therapy. Find out why you settled for not even half a loaf, maybe 1/1000th of one. Why you felt your happiness wasn't worth it and why everyone else had to come first, not only in the affair but also in your marriage (your daughter would have been fine; kids know a lot more about what goes on in a marriage than the parents ever want to believe).
Find out why you have and had thought it was a hot idea to live like this and hitch your wagon to this unreliable star.