@Martie,
I think there is a lot at stake when someone is married and having an affair, finances, children and I've never heard of one man, to date, that has left his wife, regardless of what he tells his "mistress".
In a way, he has his passion, pleasure, finances, children, family, home. At some point it gets too risky, especially if the "mistress" falls hard for him and they always do, because what they are missing they are getting.
But, then, whilst he goes home to his wife, you go home to an empty love.
I often wonder why if you can't repair a marriage not leave that marriage and be on your own. The strange part of it all is, is that guy you had the affair with is also out there somewhere single and you can sleep over his house.
Passion does cause a problem with people but an affair is secretive and I sometimes think that, that adventure that you craved, crave takes over.
I am betting he does love his wife, just didn't get all that he wanted from her, they prey on people in the same boat, such as yourself. Off course there are "some" feelings there, but ultimately they get their cake and eat it too whilst you get a broken heart.
If you can't fix your marriage then move on, if you can, work on it now.