Am I being unreasonable to feel this way

Hey all, am new to to all of this but thought thought would post and see how I get on. So it's a pretty long story but here goes, I started chatting to my new man in September 2019, instantly there was a connection between us both, he was married, not together but not divorced or anything. Nothing physical happened between us for a long time it was just simply chatting and getting to know each other but we both fell head over heels in love. His home hadnt been a happy one a long time before he left.

He still supported with the house/bills and children etc. We agreed we had to take it slow so it wasn't a leave her and jump straight into another relationship kind of situation. However we were together, but he asked me to keep us a complete secret because he didn't want his ex wife to find out, I accepted this and trusted him when he said it wouldn't be for long just while he sorted out his contact with his children. Time went on, while being in a relationship with me, staying at my house etc, he was still spending a lot of time with her, he would cancel seeing me because she needed him, she would phone and text the whole time while he was at mine with me (he would be replying) he would leave me earlier than planned because she had asked to see him.
  She said she wanted to "try" and mend the marriage so he agreed however told me it was just for show so she wouldn't think he had just given up straight away as if she thought that was the case she would make it difficult for him to see his children. He said he had to do things right so she didn't stop him seeing the children (which I understood as have children myself) however it became that he would go round hers of an evening for a take away and a film to "show" he was trying with her. Would send me a text beforehand saying he wouldn't be able to speak to me for the evening due to being round hers. He paid her rent, bills, food shopping, credit card for 11 months after leaving, spent the week of Christmas with her, staying her families house for days etc. Not speaking to me because he was with her.
  All while I am being told I had to remain a secret and couldn't tell anyone of our relationship, later finding out he had told a few people about me just people that wouldn't be able to tell his ex. This went on for quite a few months. She would request him, her and their 2 children spent lots of time together as a family even though they weren't together, she asked if he wanted to go to the theatre with her on a few occasions, she would message all of the time, he would drop the children home to her and she would ask him in to "discuss" moving forward, this discussion was requested all of the time, most weekends and he would go in her house and do this. Again this went on for a few months.
  The first lockdown then happened so he was off work, he would spend week days and 4 nights with me then go back to his mums (who lives right behind his ex) so he could have his children for the weekend. His ex asked where he was during the week and he told her he was staying with someone from work. Baring in mind by this time my children knew about me and him, my family and friends now knew due to my requesting but his ex, his family and his children were not to know about me.
  I couldn't cope with it being like this so told him I would like them to start knowing the truth, this was argued about and delayed for a while then eventually the ex wife was told as he collected the children from her and she asked what work friend he was staying with and he blurted out about me, she went mad. The children were still not allowed to know, this went on for another few months and I said I would like them to know because I was struggling with the fact that everyone on my side was allowed to know about him but the important people on his side couldn't know.
  He would talk to me and my children about his children all of the time, but he would leave the room if they phoned him during the week just incase someone made a noise and they asked where he was. My children started to question why this was and why them and mummy couldn't meet his children like he had met mine etc. So again a while later the children were told, the ex wife went mad again over this, calling him late at night telling him he was a bad father for being with somebody else and it would mess the children up if they met me, but a couple of weeks after this, I met the children, all was great, I loved them, they loved me and things have been lovely with that ever since.
 
  My partner now lives with me, his children have a bedroom at mine and stay with us at weekends. However the ex still causes grief regularly, pretty much weekly and it lasts for days at a time, her family have contacted me to tell me what they think and I get referred to as "paramour". She is is somebody else now I must add. The ex wife got hold of my email address and sends horrid emails to me, she sends my partner emails claiming I have "low self esteem" and I make him hate her, she has emailed before saying she has found out own am very money oriented which is one thing I am defintly not, she accuses all sorts. She filed for divorce when she found out about me (nothing had been done before this), he amended it for her reasoning and she then refused to sign for months.
  Without me pushing him, things don't move forward. He has lied to me quite a few times over the year to do with her or in fact his mum because he says he doesn't want me to feel upset so tries to save the upset. The lies he has told are very "good" and the stories he has made up to do with it I have completely believed and believed it for months until something gets said or a slip up happens and I've put 2 and 2 together and realised. Which makes me feel incredibly stupid and manipulated. I do believe he loves me, things are better now with him, he doesn't do things for her like he used to, he doesn't drop me to support her, he has nothing to do with her now and is genuinely annoyed if she contacts. What my issue is I am really struggling with the past and letting go, I struggle so much with the fact he used to just drop me for her whej they weren't actually even together and hadn't been for a while, when i waited months in silence and hid myself for him, I was a massive secret for so long and I struggle to understand why as he wasn't even with her. I struggle with believing that it was all for "show" and did something else happen because in my eyes if you were trying to "show" someone you were trying to mend your marriage then surely you would do couple things and be touchy feely etc. Although he assures me nothing like this happened. He would go and spend days at hers "for the children" and not speak to me all day because of this, and this was when we had confirmed we were in a relationship and my children/family knew, so was so hard to then have to pretend I didn't exist because he was "busy" .

I struggle to understand why he would spend time alone with her if it was for the children's sake that he was "trying". Watching films, having taken aways, leaving mine early to go and help her with her work. I saw her message him once saying how important "love and marriage" are, we were sat together on my sofa and she sent that. I can't seem to deal with how bad things were and how I was made to feel. I do want to be with him though, does anyone have any advice or am i being stupid staying?