@lostandhopeless,
You were in a stalemate and felt un-loved, un-wanted but I don't think that the defence of that is to give the other person even more , to try to feel wanted. Seems you couldn't talk to him at all yet you say there was honesty, but obviously no communication.
You asked him to leave but he refused however, you feel that he just didn't take you seriously..
So you slept with someone one night and felt needed. You are relating intimacy with being needed. Having sex is not being needed, it's sex. Intimacy is different it's togetherness.
In reality, the way I see this is that you proclaimed then to love him which doesn't make sense because you felt so un-loved but the moment you thought you truly would lose him, you proclaimed love and then went even the more extra mile to make him happy. Someone who doesn't love a person and cheats doesn't react that way.
In my opinion you have self esteem issues and therefore can't handle perhaps even the slightest neglect. You can't be a yoyo, oh I have attention now I am in love, I have no attention I am no longer in love.
He's kept you going since then for sometime, in-fact 3 years. Ask yourself. What has been different over those 3 years "apart from you doing even more for him than you did (figures are wrong, 4 years together, 1.5 years of fun, 3 years ago you cheated) so anyway than say the first year?
If I was to guess, I'd say you've just been together. You've worked to do anything and everything for him and felt something better and therefore, again called it love.
You cheated once.
He has been having an affair for 3 years and tagging you along with it, in it. Sleeping with both.
That's not a mistake, yours may have been a mistake, you state you are ashamed / was, felt guilty. His was straight out intentional which means he has no respect for you, yet has strung you along for 3 years.
What do I think? Time to get help to find out who you are, what you deserve in life, respect yourself, love yourself. And, time to let this one go and go it alone for a good 12 months whilst you work on you so you can have the life you are wanting. The one of love and peace and happiness. And, leave your one night stand in the past in the closet.