In turmoil

I have been married for 11 years and have been unhappy for the last 6 years. I have two young children so I have stayed with my husband. I have recently started an affair with a colleague who is also in a relationship but not married.
We have met numerous times in the last 2 months. We talk a lot and he was honest about his girlfriend from the beginning. He has told me how much he likes me and we are going to spend a weekend together. He also wants to make plans for October which I was surprised about. We have had sex several times which was great.
The whole thing is making me extremely anxious . I can't sleep or eat and have lost a significant amount of weight. If I don't get a text from him for a few days I panic that the whole thing is over. He has reassured me several times this is not the case.
I really don't want to feel like this as is is making me snappy with the kids etc. I have not heard from him in 24 hours now and feel terrible. I know I am probably blowing things out of proportion but I can't help it.
Should I believe what he tells me? Should I end it to save myself further heartbreak ? I can't understand why I am not coping with this and obviously can't talk to anyone.