I have been dating a married man for nearly 4 years but I am getting jealous

We live miles apart and he only comes down every month for one or two days at a time. He seems to quibble about the cost of the hotels and meals, despite the fact I cant work and pay for half of everything. He has a highly paid job. It makes me feel terrible and spoils the day(s). He is currently on a family outing for 5 days where he has rented a cottage for him and his wife which isn't cheap and does this thing where he tells me an hour before going, and starts the conversation with "you are not going to like it" which he seems to think makes it OK if I don't like it. Is there any way to make him understand how I feel, he wont leave his wife despite two years ago promising he would. He just said she is a companion I just cant leave! Thank you.