Affair with a close friend and neighbor

I have fallen for my neighbor after five years of being his close friend. My husband and I accepted him into our family as a close friend. He had dinner at our house every night. We supported him through several relationships. Some of the women and myself were very close friends. A year ago he married. Being such close friends we discussed everything with one another and went to eachother with concerns or issues. The connection extended to our parents, siblings, etc. Recently, I called him for support when I felt alone. My husband was working out of town and I needed someone to talk to. I called him and he came to my rescue. It started innocently enough, yes we had a physical attraction always(enough that my husband wondered) but never acted on it in the 5 years. That night we hugged, cried, and ultimately escalated the friendship by kissing. We did not have sex. The following day, both of us being a bit confused talked about the previous night. The feelings were validated by basic touch, again not sexual. We then began to arrange times to meet and be together. It escalated to a emotional and sexual relationship. We have now chosen to be in an emotionally and sexual relationship. We have set somewhat clear boundaries, neither of us intend on ending our marriages but we continue to find time to spend together alone. We engage in talking, handholding, cuddling, kissing, etc. making it difficult to stick to the idea of it being a stricktly physical relationship. What direction could this possibly go? I have been married to my husband for seven years. We have two daughters together. Our neighbor has watched them grow up and has cared for them as his own. Our lives have been intertwined for many years making it difficult to differentiate if this is meant to be or a period of time that we are unhappy in our current relationships...