being the other woman

Hi,
I'm new to this. I really needed somewhere that I could go to vent my personal situation, and hopefully get some advice.
I'm not proud of what I am doing, and I know everyone says it, but I'm completely in love with a committed man. I'm 23 years old and he is 37. We have been friends for years, lived together when I was with my ex (his old flatmate). I have a 2 year old daughter and split from her father Jan this year. We went from friends to crazy about each other overnight. It all happened quickly, couldn't control our feelings, and ended up sleeping together etc etc. 10 months down the line we are both completely head over heels, but the relationship is so up and down. He lives with his partner they have a mortgage together, he's tried to leave and she's says she will take her life. He's tried again but he went back out of pity. (Or maybe so he says). He loves me, there is no question about that. We work together too so see a lot of each other. We see each other about twice a week alone, and we have sex,cuddle, chat etc. It never used to be this loving but it's getting more and more intense as time goes on. He's not letting me move on, I don't want to either. But how long do I wait? He's scared of leaving, scared of what people will think, scared of taking the next step, but also scared of losing me. Part of me just wants to walk away, tell him it's over. But I physically can't, I ache for this man, he makes me feel wanted and special, sexy and beautiful, I do believe we were meant for each other. He says he can't bear to think of his life without me. In my head if I leave him, and if he loves me like he says then he will leave and be with me but I can't think straight without him. I spend my nights alone while he is at home. We chat a lot when his partners not around (she's away a lot). I feel like we're in a relationship and I don't look at anyone else because I want him. I'm so confused.