"Dating" a married man

I have been seeing a married man for a year now. It started at work. He pursued me for a while before things started to happen between us. It got to the point where I was sitting in his office for at least an hour a night talking. Talking about anything and everything. I even stopped seeing a guy I had been dating for him... We also had a physical relationship. He would rarely text me on weekends (unless it was work related) when he was at home and that was fine with me. 3 months ago, he decided to leave the company and wants to go back to school next year. So currently he is not working or going to school... meaning he is home a lot. Contact between us has become minimal. We went from seeing each other everyday to hardly speaking. He has a key to my apartment and can come over anytime he wants... . When he first left work, I would see him once a week. Then it was 3 weeks before I even heard from him. Last time I saw him, he told me he would be going out of the country in 2 weeks for a month with his family. I told him to try and stop by before he left. I haven't heard from him. I don't feel like I have put too much pressure or have asked to much of him. I just don't understand... is it just because he has less free time and being at work together was convenient? Is he trying to pull away? Or does he just need a break. I want to ask, but at the same time I don't want to be like a nagging wife.. he already has one of those... Do I need to just accept this for what it is and try and find someone for myself?