Why can't I be with her?

I have tried to be in a relationship with this woman twice now. Both times we broke up it was because whenever we would be together for a certain amount of time, after a while I would feel a pull, like I needed to get home and do my thing, alone, whatever that thing was. As we got closer, the stress of my need to be alone got worse and worse, even though I thought the world of this girl, and still do.

Not that there weren't problems. To be honest, she was not the easiest person to please. I felt like she didn't appreciate the effort I was making, and sometimes when I would go out of my way to try to make everything perfect for her, still she wouldn't be satisfied. I felt like I could never make her happy no matter what lengths I would go to. Also, sometimes I had the feeling that we just weren't connecting the way we should be. That being said, now that we broke up again, I can't stop thinking about her and wondering if I just lost the best thing that ever happened to me.

We just broke up for the second time very recently. We are both middle-aged. After we broke up the first time, I felt a mixture of relief and heartbreak at first, which over the following months slowly turned to regret and longing to be with her again. Once we got back together, I almost immediately felt like I'd made a big mistake, but I felt like I might possibly be in love with this woman and that the reason I was ambivalent was that I had lived single and autonomously for most of my life before her, and that that was causing me to have this struggle between wanting to be with her and wanting to have my old freedoms and my old responsibility-less life. So we tried to make it work again, and she gave me room and space galore, much more than I could have asked for. Still, after a while, she wanted us to get closer and spend more time together again like when we were together the first time, and that was so scary to me I just couldn't do it. So again, I broke both our hearts.

Now here I am again, a few months later, hating myself for having hurt her, and wondering what the hell is wrong with me that makes it impossible to be with her.

I have come up with three possibilities, and maybe it's a combination of all three, or maybe it's something else that I haven't even considered. The third one here is the one I'm most worried about, because if it's that one, then I've thrown away the only woman that has ever loved me for nothing.

1. I'm too set in my ways. I've always lead a pretty simple life with no real encumbrances or serious attachments. I basically go to work and come home and watch TV during the week, and squeeze my social activities in on the weekends. My main focus is to try to have as much free time as possible in which to basically do nothing. Maybe that is a part of my problem.

2. I simply don't love her. I don't know about this. I've never been in a real relationship before this woman, and when I was with her the last time, I could not imagine being with her forever. The thought scared the hell out of me, to be honest, and one of the kickers that lead to me breaking up with her was an incident in which we had to worry about the possibility of her becoming pregnant. The chances were very very slim, but it made me picture that scenario and what I would have done had that happened. It was in my mind just about the worst thing that could ever happen.

3. I smoke too much pot. Pretty much on a daily basis. I have a good job and a stable life, and most people I know, including all my family and most of my closest friends, don't even know I smoke pot at all. It is my life's big secret, and has been for several years. This woman knew about it and had no problem with it, at least none that she ever voiced. But I'm thinking maybe being high so much of the time dulled my ability to feel my emotions or something. Is that possible?

Well, I guess that's my story. When I'm not with her, all I can think about is how much I miss her, and when I'm with her I can't wait to get away from her and go home. I keep replaying only the best memories of my time with her and the horrible times when I hurt her. Mostly I keep replaying the night I broke up with her, wallowing in the guilt and hating myself for hurting such a sweet wonderful woman whose only transgression was loving me, the unlovable, too much. I can't keep torturing myself like this, and so I've come here hoping for some insight or some resolution.

Thank you for your input.