@Sarah boo,
You know Sarah, there is no point doing the blame game what is done is done, what is important is that you know the direction to go in.
This thread suggests that you truly want to end it but the crying from the guy in question is stopping you.
Ask yourself what does "he" have to loose? Just what he is enjoying? Emotions, physical, communication, care, something he doesn't have maybe or just crying because he doesn't want it to end.
No where have you mentioned "love" either within yourself or him, rather, you feel you've made a mistake. Ok, lesson learnt.
Is he married? Did he give you a sob story about his non sexual relations?
Do you think you can work this with your husband as in your marriage?
You say it's the worst feeling that suggests that you are ready to end this and maybe found out more about yourself than you really knew, you can use that and grow as a person.
Have you changed in a good way?
If this is getting to you emotionally you should end it.
It would be good to know a bit more, I get the feeling that you think he will tell your husband is that what you are frightened of? If you end it?