It's understandable to feel this way for both parties. I have had female friends my wife didn't like and vise verse. The thing is that people just aren't 100% compatible, sometimes we have more fun talking about things with other people. Is it something to be worried about? Yes, but only if there is a possibility that he could be unfaithful. If you feel that he's closer to her than you, then sit down and talk to him. Try and be non-judgmental, which is very difficult sometimes.
If your uncomfortable with their relationship, ask to see his phone and check messages and call logs. You can ask in front of him and if he hands the phone over, look at call logs and txt messages, if you feel something is said that is weird, ask him.
In the end, just don't let jealousy or be over reactive about his friendship. Try and just see it for what it is (unless it is something to worry about). But when you have your sit down, talk to him. Express how you feel and why you feel that way. Make sure to figure out why it makes you uncomfortable, if you go to him and he asks why you feel that way and you just respond with, "I don't know?" or "Because!" then he'll just get mad and frustrated. Take the time and find out if the friendship is more like a relationship and make sure when you talk to him, that you know how to answer his questions too.
We have friends for a reason, people like different conversations, different views on politics, law, etc. It could just been she's fun to him, but that doesn't mean he doesn't love you still, could just be that something you two don't agree on, but they do.
That's my two cents worth, I really hope it helps, but most important is to stay calm and don't let anger, frustration or anything else negative affect your conversation on the matter.