@Christine1,
If two people are un-happy of the opposite sex, they talk about it and in depth, next thing you know there are common grounds. Discussions go back and forth of un-happiness, next thing you know there is a "sexual" spark, yet there is always one, (one) that will not leave his/her significant other half for what ever reason.
It's an escape from reality.
Where as reality is simple. You've received the advice and you know it to be true, if you can not work your marriage, can not seek counselling together to work the marriage can not get anywhere yet you've tried and tried and tried, then you walk.
What you experienced with the married man you can experience with a single man only better, you get to sleep next to him at night time, wake up in the morning go on dates.
If someone doesn't have adventure and that's what sleeping with someone you can't have is, you don't need to go on that adventure to make yourself happy because whilst you are doing that, someone else is un-happy imagine her thoughts when she finds out and she will.
Just walk. From both. Start your life over.