The reason I have such a high opinion of you, without knowing much about you, is this - selfish, unkind and insensitive people never get into situations like this!
It is usually the kind, caring, sensitive, bright, introspective, evolving souls that get trapped in such quagmires... (Just think of a selfish ditz you know - would she ever put up with this?)
There are two immediate challenges you must face:
1. Overcoming your fear of losing him due to -
a. him thinking you are "abandoning" him in his time of "need" (which you may believe will make him not love you in the future)
b. someone else stepping in to "care" for him while you are taking a break from the "relationship"
2. Overcoming your own sense of "duty", "loyalty", etc. (a type of "noblesse oblige") to stand by the man you love in his time of need, to support him, etc.
If you allow your common sense, your sense of self, and your dignity to be your guides, you will be able to overcome #1 (both parts a and b). You must realize that an honorable man does not use a loving woman the way he is using you (I know, every time someone says something negative about his treatment of you (like I just said about his misuse), your memory may race to find 2 positive things he does for you - I have really been there, but please try to be guided by reason, not emotion, and yes... I know this is hard).
Would you ever treat him the way he has treated you? I think not... at least not judging by your posts... and you are also in need, just as much as he is (if not more), his situation is not really any worse than yours.
So... he is getting divorced, his family is not happy about it, blah blah blah... you have been through worse (at least from what you have told us) and you have managed to do it without stomping all over someone else's dignity.
You are a divorcee and a single mother, you have had to struggle and work hard for everything you have, you are a prominent member of a community and are now being ostracized by a significant number of people for things that you have NOT done. He is a guy, who is tired of his wife, is getting a divorce - taking his own sweet time, while having a great time being out with friends and family (who are much more willing to make you the bad guy than him), AND... on the side, enjoying the love, care and sex of a wonderful woman - YOU... without having to do ANYthing for you.
Why are you holding the bar so low for him? Because he is getting divorced? A number of us here have been divorced (including you) - and I will let others speak for themselves, but at least in my case, it didn't kill me, neither did it turn me into an insensitive and nonchalant donkey who treats other people without respect or consideration... and I didn't have someone's gentle arms holding me through the ordeal... He is a big boy, his need is not greater than your need. Do not worry about him not wanting you in the future because you "abandoned" him in his time of "need". If he loves you, he knows that he needs to be there for your needs and until he can do that, he has no right (moral, emotional or ethical) to expect anything of you.
I am not even going to address the worry about someone else stepping in to take your place. As irrational as this is, I know I have held that fear in the past (those were the bad old days), but I think I cannot tell you anything you cannot think of yourself. Please be rational... if that's how shallow his love is for you, then you don't need it (no matter how much you may think you want it)...
The second point (about your own feelings of "loyalty", "decency", etc.) I have already touched upon... do not hold the bar so high for yourself when you are willing to put it on the ground for him. Be rational, use your common sense, listen to your gut (not to your fears, but to the voice of the woman you really are, the voice that is always within you)...
I am definitely rambling today - lack of sleep will do it... sorry...
You said "I am so afraid if I do [take a break] he will feel I have let him down"... Please, please stop worrying about what he feels for a little while and worry about what YOU feel. Do you think he worries nearly as much about what YOU feel? Doesn't seem like it to me... You have an obligation to yourself and to your children, to be and remain a whole, healthy and functional human being - no one, not even the greatest love, the Romeo of your life, is worth self-sacrifice or the sacrifice of your children's mother (that is where your obligations are - with you and your kids)...
Start putting a little distance between the two of you - find new ways to occupy your time so you are not so available to him, avoid seeing him as much as possible, especially in private. When you are at a public function together, YOU should be the one being cool and generally ignoring him (remember, he is not an available man - neither legally nor emotionally, at this point). I am certain you have not enjoyed yourself at any gatherings which you have attended because you have been entirely consumed by thoughts of him (where is he, when will he come over, why is he not looking at me, etc.) and you should really try to change that. Remember the person you were before all this? That person was stronger, happier and in a better place.
In my estimation, LOVE should raise you higher, should make you happier, should fulfill you in body, mind and spirit. Is this love doing that for you?
I will probably disappear for a while - I will be working a lot of hours in the coming month, but I will try to check in every now and then...
P.S. I hope this is somewhat helpful to you and others in search of this sort of advice. A little confession (selfish), this is helping me keep my mind off the married man I love (no clandestine relationship there, I was lucky enough to find the strength to stop myself from going down that road, but my thoughts are still on him much much more often than they should be.)