he decided to stay with his family
My big piece of advice would be to stop rationalizing his decision as something he did for his daughter.
Plain and simple, he decided to stay in his marriage. Full stop.
In a few years, or maybe a decade, you might be able to be friends with him.
Right now you need to focus on yourself and not on how to have a relationship with a man who has chosen to stay in his marriage.
Coffee twice a week is not acceptable at this stage.
You need to stop using language like "trying to walk away". You need to walk away.
I understand that your daughters are friends. The one message you need to get to him is that he can not come to your home to pick up/drop off his daughter and that he needs to stay away from you at school. He owes you that much.
He was able to cover up your affair with his wife - he'll be smart enough to make other arrangements for his daughter.