Really need some advice please

I am new to this forum and to the program, but I have chosen to take the path to working on the recovery of my marriage. Every day seems to confront me with a new obstacle-whether its within myself or with my spouse. We have been together since we were 15 years old. That is half of our lives already. There is not a memory that comes to my mind that does not somehow involve us together. I understand that I know no one here but honestly feel as though just getting all of this out will somehow bring huge relief.

Everything started 8 years ago when I myself had an affair with a co worker. My first mistake was letting someone of the opposite sex become a close friend. The friendship lasted for about 6 months until it developed into a more personal attempt. At the time I was very young and immature- and I made some very regretful decisions- I had an affair. Needless to say my husband and I divorced. After 3 months of sorting through and working things out, we decided to remarry and continue a new life together. I was very open with him (gave him access to everything and worked on building the marriage we both wanted). As soon as we were able I even quit my job.
A year after leaving my job, we had our first child. The experience was so amazing and I remember how happy we were for such an amazing blessing. 3 more years passed and then we had our second child. life could not have been more happy. we were happy.
Two years after our second child was born, my husband began a new job. It was a few months in when changes began-he started making comments about us spending too much time together or not wanting to have lunch together anymore. I immediately sensed a change in some of his actions, but he continuously denied. Right around this time, he introduced me to a female co-worker . There was a group of us girls who began hanging out and exercising together (at the time i was a personal trainer). She and I, along with a couple of other girls even went to out of town events together. There was always something about her that I questioned due to certain actions on her part- but everytime i would bring up anything to my husband he would simply tell me he thought I was over reacting or looking too much into things. Another red flag came when he all of a sudden asked me to stop hanging out with her. (she was an older woman who somewhat lived a "single" lifestyle- there were certain events i also witnessed with other men where I could see wasnt good for me to be around). I honored my husbands request. As time passed I had a gut feeling that there was something going on between the two. Over and over again, for two years , we would argue about these feelings and he would make me feel as though i had done something wrong or that i was obsessed with trying to make him hate her. the constant arguing during this time only seemed to make us drift away, but we continued to remain married.
Three weeks ago, he finally admitted to me that they had slept together 4 times before the time of him encouraging she and i to hang out. they still work together. there is still email contact (shes a receptionist) and meetings where they are both present. i was furious and so hurt that he would wait until two years later to tell me all of this-let alone let me be around her too.
My husband has taken the initiative to tell me all the details, he seems truly regretful , but there are times i feel he wants to just not talk about it anymore. In a way its almost like to that since I did something 8 years ago and he forgave me- thats what he expects too. and just start over. Is it wrong for me to think that the pain is worse because we are at a different time in our lives (6 years after remarrying, asking for gods forgiveness, having children together). he tells me the affair had nothing to do with what i was doing at the time- but that things in the past he feels did contribute to his curiousity. he tells me all that matters to him now are me and the kids. I have told him that i think the only way for us to truly move on is for him to leave his job. I feel horrible because 7 months earlier he was promoted to manager and we have been financially better than we ever have. He told me the affair has been over for 2 years but the guilt was driving him crazy. there is just still alot of hurt present and some days i feel i worry myself to death because he is still in the same environment with her only a few offices away. its almost like torture. how can he be okay with communicating with her or being around her in the same room?