Depression after night together

I am feeling at a loss so thought I would ask advice on here.

So a bit of back story, I first met him earlier this year, I knew he was married and he knew that I was in a long term relationship. I also knew the kind of person he was yet I did not stop him from perusing me, I encouraged it. For months we flirted over messages and a few times we arranged to meet but the timing was never great so it took up until last month until we finally met. He called me up one day to say that he had booked a room and that he expected to see me there, I spent a week in turmoil and tried to talk myself out of going but I knew that I had every intention of meeting him.

The day came, I met him in a hotel room during the day and proceeded to have the best sex I had ever had. Afterwards we left and he was going on holiday the next day with his wife and two children so I was not really expecting much contact with him over the week, even though I knew he was away, it was one of the longest weeks I have ever lived through, I felt as though I was going crazy and I could not go longer than a few minutes without thinking about him.

The day after he returned from holiday he text me and whilst it was just a message saying "Hey" I felt as though the weight of the world had been lifted from my shoulders and I could breath again. He explained to me that he had been thinking about me and needed to see me again, I told him that I couldn't meet him again so soon but we would arrange something, he told me that he wanted to spend the entire night with me so I made up a story to tell my partner and booked a room for the following week. We met again in a hotel and if I thought our first time was amazing, the second time was unbelievable and I don't think I am alone in thinking this. We spent the entire night together but the next morning he felt a bit distant, being someone who does not like to look needy, I distanced myself from him and when he left he told me to message him later to say I had got to work ok, I never messaged him.

It has been 4 days and I have not heard a thing from him and I am going through sheer turmoil again. I can't even explain how I am feeling as I have never felt anything like this before, it is as though I am on the edge of a sinking depression and extreme happiness and it all comes down to if this person sends me a message or not.

I know what most if not all of you will be thinking about me and this situation, I am a foolish person who deserves any pain that may be felt due to my inability to be loyal to someone who has never shown me anything but pure love. I know this but when I think about this other person, who is quite clearly not a 'good person' and is using me for one thing, I do not care about anything but seeing him again.

I don't really know what the point of writing this out is, I just need to tell someone what I am going through as I don't know how much longer I can deal with this pain...any advice is welcome