Good, you acknowledge that these feelings have crossed your mind before this woman. Because, as you may be beginning to understand, a lot of people cling onto other possible mates/loves/sex partners as a way to distance themselves from their marriages when the reality is that they should be long gone from their marriages prior to even thinking about someone else.
Your inertia is not helping anything, but I can understand its attraction. Waiting it out (until what? Until someone dies?) is a lot easier, emotionally and financially, than dealing with a messy divorce, what with its arguments, bad karma and division of community property.
Yet deal with it you must or, at least, you should. If you care about your wife at all, even on a friends only level, surely you can see, once you get beyond the haze of your own issues, that hanging around is not doing her any favors. Y'see, and not to be overly blunt here, but unfortunately it's true, women lose status in the mate market for every year they are alive. It sucks, it's absurd, things should not be like that. But it's true. A 30 year old has far less chance of finding someone than a 20 year old. A 40 year old, even less so. A 50 year old, even less. Part of this is because of menopause, part of it is because of a lack of men after a while (particularly as the end of life comes nearer and nearer), and partly because many men (not necessarily all, but many) are hard-wired to want a young chippie. I say this as a 43 year old woman, by the way, and a happily married one at that. But I ain't kiddin' myself. I am well aware that someone who does not know me for my sparkling personality and the like (
) would be hard pressed to justify me as a mate when there are younger and more attractive gals around town.
So your inertia is not helping your wife for the future. And I know that's a tough thing to think about, what things would be like for her with you not around. But for people whose best interests we are concerned about, that kind of unselfishness can really happen.
My advice, essentially - and it's always my advice in these situations, and I really should just start copying and pasting it already - is to make up your mind. You want to be with this other woman? That's fine, just let your wife go so that she can pursue other interests - interests that may or may not include a second marriage. Don't jerk your wife around with your inertia. Or be with your wife. That's fine, too. But that means giving up your gal friend's hugs and hand-holding.
Oh, by the way, if your wife finds out about the hugs and the hand-holding, she will see it as adultery. Most would. She may or may not be forgiving, particularly considering as there have been no sex acts committed. But it will change things. So you need to see that you are playing with fire. What happens if you and your friend are hanging around in the stairwell, holding hands, and a coworker walks in on you? Will all of your coworkers keep quiet? Will they lie for you? I doubt that seriously. And one thing is a certainty in all of this.
Divorce and the like stink. They are a raw deal all around. But at least if you come to your spouse on your own terms, and you split amicably, it doesn't suck anywhere near as much as it does if you were to get a divorce because you were found out for cheating or near-cheating. Consider that before you and your lady friend hold hands in the hallway, stairwell, cafeteria, conference room, parking lot or wherever.
PS You don't say whether kids are involved, either in your marriage or in hers, but their interests and needs have got to be paramount. A messy split will devastate them. Think about that. Inertia is attractive and fun. I am as lazy as the next person. But the pain of a divorce for fault is far worse than the escape velocity you need to get out of the inertial state.