Should I just give up on him?

There is this guy, say Joe, we both had been hitting on each other for about a year but because I was in a relationship Joe chose not to pursue it. When this relationship ended we started hanging out more and finally after talking and beating around the bush we both established we really liked each other. On our next date we were hanging out and out of the blue Joe kissed me this went on to turn into a hot and heavy make out session. This was about a week and a half ago. The next day we were texting about it and we both agreed we really liked each other and had strong feelings. We have one more date where we hung out with Joe's sister and cousin and played board games, I know them from work. I text Joe two days later and we have a very short conversation and he abruptly stops texting me. 3 days later I asked Joe if he wanted to go out to dinner sometime over the weekend. Joe goes on to say, "No, I'm sorry, I can't." So I go on to ask if we are cool. He says yes we are cool. I mention that I'm glad. A few hours later Joe tells me that he had a crush on me and enjoyed my company but just doesn't, "trust you are right for me." He goes on to apologize for not mentioning this sooner.

Since this has happened I sent him a couple of texts, which was most likely a mistake. I worked with him today and I just completely ignored him.