Torn over affair

I have been married for 28 years and have a 26 year old son and a 20 year old daughter. I love my family very much. I have always worked full time and have taken care of my family. My husband and I have always made time for eachother, dining out weekly, vacationing alone. But... he never really wanted to have kids and frankly didnt try very hard to be a good dad. He was distant with our children and not very involved. He did only what he had to. I felt like I was raising the kids alone most of the time. Like they were my kids and not his. But he was always a hard worker and he did all the maintenance at our house. It was big old house that he kept meticulous. It was as if I had the kids to take care of and he had the house. My husband was never an affectionate or expressive man.He only showed his “love” in the bedroom. I probably shouldn’t complain as many of my girlfriends have little or no sex in their marriage. But I got bored and lonely and frustrated with the stress of the kids. Any time there was any issue or problem with the kids it as mine to deal with. And of course somehow my fault. I always tried to take care of myself, belonging to a gym, coloring my hair, dressing nice and my husband wouldn’t really acknowledge. We sold the house last year and are living in a high rise apt with our daughter who commutes to college. My husband is extremely OCD and has a lot of stress over living with our daughter. He is almost constantly on her back. They do not get along and it is very tense at home. Needless to say I have not felt happy for some time now.
There was this younger man, the mail man at my job. Very very handsome, ex - professional model and very sweet. We’d joke every day and one day he kind flirted with me so I flirted back thinking it was harmless. He’s younger and HOT what would he want with me? August 15, 2016 I slipped him my number and we’ve had an affair ever since. He has two grown boys from a previous marriage and an 8 year old boy from his last girlfriend that he absolutely adores. He is a devoted and amazing dad. We were seeing eachother about once a month outside of work but lately it’s been months at a time. He says he’s very busy and he keeps reminding me that I’m married. I have fallen in love with this man. I know it’s wrong. I know we can’t have a future because I will not leave my husband for him or anyone and I do not wish to ask my husband for a divorce. I still care about him and feel like I’d destroy his life if I did. He has no friends, family and I am his whole world. I’ve tried ending my affair, although we haven’t been physical in 3 months, we talk daily and often it’s sexual. This is the hardest thing I’ve ever experienced. I need advice.