Husband and the neighbor

I have been married for over 20 years. And am decently attractive and in good physical shape. Many years ago my husband cheated on me after the birth of our second child. He cheated with a YMCA receptionist who worked at the location we frequented. It took much work, but we reconciled. Last year my husband began talking with our female neighbor, who is divorced daily. He would go over to her yard, before coming in after work. She would wait on her steps if she saw him pulling up after work. I caught him over her house on the steps at 6:30 AM and I asked him if he had feelings for her. He said he did have feelings for her, and then later tried to change his statement and said he had a crush on her. I put my husband out for days, until we talked. He now claims he didn't know why he said what he said about our neighbor. I tried to tell him I was worried about his communicating with the neighbor in advance of the 6:30 AM incident, but he blew me off! My question is: should I divorce my husband, even though there was no sexual stuff going on? However, I do believe he had an emotional affair. I have to see this neighbor regularly since we live right next door and it is tough..