I'm not sure what to do in my relationship

I'm new to this website and I'm here because I'd really like some advice. My boyfriend and I have been together for nearly 3 years. Our relationship is very good, but lately things have been weird. We have been together since our sophomore year of high school and we're in college together now. We are young, but we've always had a very mature and realistic relationship, and we value each other very much. I guess the most recent thing that triggered these feelings happened the other day he went hung out with a girl. They went to her house, got coffee together, and hung out for a long time. He didn't tell me that he did it until he got home. He didn't cheat on me, and I know that he wouldn't, that's not what I'm worried about. We've both been cheated on before and I know he's not the kind of person that would do that. However, what I'm worried about is that he is seeking something else and not wanting to admit it, maybe even to himself. I've always had this worry in the back of my mind that he's going to want something different from what I can give him. We are very similar, but I feel like other people are more passionate about the things he is interested than I am. I have talked to him a bit about this but I haven't really resolved anything. Of course I want him to be happy, and I let him know that he is not a bad person is he wants something else, and I just want him to be honest and do what will make him feel best. He says that he doesn't want to leave me and that I still make him happy, and I don't want to leave him, but things just feel different. It seems like I'm not as important in his life as he is in mine. Him and our relationship are the most important things in my life, while I'm only one little aspect of his. I am not expecting him to revolve his entire life around me; the only reason I bright this up is because it seems like it used to be that way for him too. He used to tell me all the time that I was the most important person to him, and I'm only upset because that has changed and I don't understand why, and I worry that someone else will take that place. (please note that I'm not trying to say that he should spend every second of the day with me and only me and never spend time with other friends or do anything else. That is not what I'm saying.) It just seems as if I used to be his number one, favorite person and I'm not sure that that's how it is anymore, and I'm worried that it is because we're drifting apart. I know that I have flaws and I am difficult to deal with sometimes. I have anxiety and depression and I have a very hard time connecting with people, so I can be dependent on him sometimes. We've discussed this before, and he has even told me that he occasionally depends on me too. (I have other friends and other things in my life that make me happy, I'm not solely relying on him for my happiness.) I know that it can be a burden for him, though. I'm always sure to stress to him that I'm not looking for him to "fix" me or my problems, I just want him to be there when he can. But while he is dealing with his own problems, I think he feels pressured when presented with my problems as well. I'm worried that I'm pushing him away and that he will develop closer relationships with other people. I told him that it felt like he was living two different lives, one that was our relationship and one that was everything else, and he agreed. I think he feels the need to act differently around me, and whether or not it is my fault, I'm worried that it's irreversible. I don't want to feel like I'm dating one version of him, while the real version comes out with his friends or even other girls. I love him so much and ending our relationship would be devastating. Our relationship has always felt very special. Like I said, I know that we are young. But we aren't stupid. I'm just feeling very stuck and I'm looking for another opinion. Thank you for taking the time to read if you did. Sorry if it seems out of order or like I'm rambling.