left out again

I have been a mistress to one man for 20 years. I have spent my youthful years with him, and am now white-haired, a senior citizen and retired. He recently joined a dance group (he's 75 years old, 8 years my senior), that are invited out as a "group", travel as a group and practice as a group. Outsiders such as myself cannot attend any of their functions. Now I find out, about 3 ladies in the group, know of his wife. Soon parties will ensue, and I've been told it would not be feasible to include me to any of these if they happen to occur, he says its too risky. I'm ready to call it quits. He calls me selfish, jealous and overreacting.