Am I setting myself up for hurt?

I'm in a relationship with someone for 7 months. At his suggestion, we have decided to live together starting in a few months. We are older so I'm not certain if the recommended one year rule applies. However, while he'd be moving into a newer and larger apt, I'm coming from my large home to a different town 50 miles away.

Here's what I do know about him....he is who he appears to be. By that I mean, he has nice adult kids, a brother he is close with and many nice friends, all of whom seem well grounded. He is a lawyer and does not bullshxx or otherwise mislead. However, he seems hypersensitive about his ex wife. They were married for 34 years and been divorced for 3.5 years or so. She asked for the divorce but she is alone and do not believe she has gotten through it. He has told her he is in a relationship with me and apparently, she has been worse to deal with than before. Their son is engaged and having an engagement party (the son and fiancé are giving it just for their friends and their parents and not the parents' friends). He told me the other night about this and said I'm not included because his ex would be too upset and she's emotionally fragile from the divorce. This was on top of a mutual friends' wedding for their son out of town that I was excluded from and the wedding shower b/c he told me “it was be too uncomfortable for his ex“. I told him I'm his partner and his ex is just that, an ex! By the way, he refers to her by name and never uses the “ex“ term.

I further told him that the only people before me should be his children always but after that, it needs to be me. Further, I told him he compares me with his ex even telling me we both dress nicely...told him that is totally inappropriate and I never need to hear her name unless absolutely necessary. He was understanding and not defensive. Told me he loves me and would not be wanting to live with me unless he wants to have me in his life. Of course, moving is an advantage to him since it's in his area of town and he won't be spending any more money but will have a larger unit, albeit sharing it with me. I, on the other hand, have to sell a large home and move to another town. I do really like the new area but there's much more involved for me so I told him I need to know I'm not going to be 3rd fiddle and moving forward, I need to be part of his family's celebrations. If his son and his fiancé aren't comfortable with me at this party, that's fine but the reason shouldn't be b/c his ex will feel uncomfortable and may make a scene. I also told him he should be focused on my happiness and not how his ex is feeling - she needs to get therapy or whatever help and I won't be blocked from being involved b/c she always may feel uncomfortable. Am I being reasonable?

He did remove the old photo of him and his ex with their kids and added one pic of us together. Is this enouhg? Should I wait to live with him just in case?