Leaving for a new state

I have been seeing this married man for more than 6 years now. Initially we were (if I may say) in loved with each other. But as time went by, the relationship was putting too much pressure on both of us, so we agreed to keep it at a level where expectations, love and commitment was kept at bay.

I tried to turn my back several times but I just cannot do it 100%. Because I love him. I do not know how much he feels about me. I never asked after we had that serious conversation about keeping everything at bay.

We have talked about going together in another state and starting over...but then after a while he told me that he did not want to go anymore. He also asked me not to go as well. And I agreed.

But now, I have decided to leave for another state. I was already able to find a job and home there. One of the reasons I am going is because my child with him is already growing, he looks so much like him. He doesn't want to have anything to do with this baby even if we are still seeing each other.

I feel like if I tell him that I am leaving he will get mad. But if I just leave he will also get mad. If someone else told him that I am leaving he will get mad. I am not sure where I will stand...or where I should stand.

=( Please give me any advice you can. I know I did the wrong thing for loving him this much and for having a child with him. But...I want to start over for myself and for the child.