How to handle a boyfriend being friends with his ex

Okay, so I apologize in advance but this is going to be a bit of a story.

I started dating a guy about three and a half months ago. Everything was absolutely perfect for the first two months. He surprised me with flowers and gifts all the time, wrote me love letters, took pictures with me, put me as his phone background, made blogs about me... etc. We are in sort of a long distance relationship. I'm at college an hour and a half away and he is from my home town. I say sort of because the first month he came to visit me every other weekend and for the second I went home three out of the four weekends to see him. And now for the third month, I am home for the holidays. We've seen each other fairly often throughout our relationship.

So, like I said everything was perfect until about the first week of November. He became really withdrawn from me in every possible way. (No affectionate names, taking a long time to text me back, no longer calling me on his work breaks, etc.) It just so happens the day this started, he started spending a lot of time with his ex girlfriend. To the point where they were seeing each other every day for a week. (She cut his hair, they went in a group to go see a movie, she visited him at work, she went out to eat with him and his friends, he took a picture of himself with her phone and she posted it online) Him and I started bickering a lot over little things and leaving a lot of things unresolved before starting another argument. It seems like I tried everything to save our relationship but he wasn't putting any effort towards it so nothing was working. I was so hurt and frustrated that I decided I was going to break things off with him. I picked him up and we sat in my car as I started my break up speech. I told him I didn't deserve the way he was treating me and I didn't know what else to do because it wasn't working out and the only reason I was hanging on was because of how he treated me in the past, not how he was treating me during that time. However, he ended up apologizing and asked me to reconsider and sleep on my decision because he didn't want to end things. He promised to be a better boyfriend and we both ended up crying as we were held onto each other. So, we didn't break things off and the next day I saw him and he did as he promised. I could tell he was trying once again. But, this behavior slowly faded and about a week later we started to bicker again. This time however, we stopped tackling the surface issues and started to communicate in a calm, rational way what was really bothering us. We had our first resolution and it gave both of us confidence in our relationship once again. For the next two weeks, we focused on rebuilding our relationship and we communicated effectively with a lot of things. Of course I had a lot of questions about him spending so much time with his ex girlfriend simultaneously with him being distant towards me... His response requires some background information though:
He broke up with his ex girlfriend about a month and a half before he started dating me. They dated for about eight months. They met through mutual friends which was one of the reasons he gave me for hanging out with her so much. He told me it was a mistake dating her because he never had strong feelings for her and he started dating her before he was fully over his other ex (which for sake of clarity, I'll label ex2) He described his ex2 as very manipulative and self-seeking and she knew that he still wasn't over her while he was dating his ex. He ended up in a situation where he was trying to please both girls and he ended up cheating on his ex with ex2. He told me all of this due to us being open about our pasts and this was before him and I had any problems. He grew up with a father who taught him strong morals about treating females right so this first act of infidelity really did hit him hard. I think it damaged his self-concept of being a good person and trusting himself to do the right thing. He said he felt horrible about himself every day it was dragged on and he would never do anything similar in the future because it was such a dark time for him. All of his friends ended up finding out about what he did and for a long time he was ostracized from his friend group. He said he prayed about a month before he met me to find a way out of all of the drama in his life and he said he saw meeting me as a way he could was able to do that due to a clean slate. I told him I forgave him for his past faults and I believe him to be a better man than what his past actions have shown, which I do. I'm not quite sure if he has forgiven himself though.

So, he said the reason why he became so distant from me was because his ex girlfriend and him got over their breakup and ill feelings for one another and decided to be friends which was around the time they started hanging out. He said because it was such a similar situation as before (where he was hanging around a past girlfriend while still dating somebody) it made him afraid of the possibility of drama in his life again and he was considering avoiding the situation all together by getting out of our relationship. He said once he realized that his ex girlfriend wasn't the same as his ex2 (manipulative and self seeking) he began to come around to me again and he gained confidence that the same situation wouldn't unfold again.

So, ever since all of this happened, like I said we resolved things by communicating to one another and we have been on really good terms. I got back from college about a week and a half ago and problems with his ex girlfriend started popping up again. The second day I was back I was hanging out with him and he was going through his call log to see if his friend called him and I noticed that his ex called him pretty frequently with one call being that morning. I asked him when the last time he hung out with her and he said it wasn't for at least two weeks. I'll admit, this did bother me because I didn't realize how often they talked. He said they text on occasion too. Then the following night he told me he was going out to eat with his guy friend so I didn't invite myself and instead I made other plans to hang out with my friends to go see a movie. When I got out of the movie, I looked at my phone and he texted me and said he was at the restaurant with his friend and his ex girlfriend. (which I know is true because his friend posted a picture of the three of them online). I saw the text about twenty minutes after he sent it and I tried to call him but he didn't answer. (It was about 1:30am) I called his phone at about 330am to see if it was still on, which it rang once then I hung up. The following day he told me his phone died which was why he didn't answer, which I know wasn't the truth but I never confronted him about it. Two days later, I went to visit him at his work to surprise him with a snack and drink and his ex-girlfriend was there. He told me she got there about 15 minutes before I did. At first it was really awkward, but I introduced myself to her and after that she was really friendly to me and we started small talking and sharing stories. She used to work in retail, which is what my boyfriends job is, and she got to know all of the people he works with while they were dating. So, while she was there she wasn't just talking to my boyfriend, she was folding clothes and helping out with the store and hanging out with the other employees. She works as a hair stylist specializing in men haircuts and she was putting together a portfolio and asked my boyfriend if she could cut his hair again and take a picture of him and my boyfriend agreed. So they did that the next day and afterwards they ended up seeing a movie together. (it was him, her, and one of her friends).

I guess where I'm stumped with all of this is how to handle a boyfriend still being friends with his ex-girlfriend. I know you cannot have a relationship without trust, so I do my best to trust him but I'm not sure where I should draw the line.

Please help.