Need advice..He says I dont deserve friends

Hello Everyone,
I will straight away start my story. I joined this college in 2015 for my postgraduation. We were 3 in one batch. This male friend(?) of mine is my senior from 2014 batch. Initially we didnt talk much. Last year in march we started talking and within 2 months we became very close friends.We would stick around each other during the college and hang out after college.It was a routine that we used to talk till 3 am in the morning on phone. One day he said that a guy and a girl cannot remain just friends and he is afraid that something will happen between us since he wanted to kiss me in the college and he doesnt want that since he is married …and we should distance ourselves…we tried but both of us couldnt and then we got physically intimate.We became from friends to best friends to more than friends…these were his words. I was confused of my feelings.But then we decided to stop all this and just remain friends. Then came a turning point he had a fight with his 3 friends from his batch ( they are 4 in 2014 batch 3 girls and 1 guy thats him) and stopped talking to them and then slowly he started distancing himself from me…i couldnt understand why..we had fights..and then i went for a vacation to my hometown.It was 2 or 3 days before my vacations were over when he messaged me that he is having panic attacks. I do meditation(for my anger)so I told him that he should also start doing it . After coming back I told him to go to his wife and family in his hometown. He went for 2 weeks and was improving with meditation but then the night before when he was supposed to comeback his condition worsened & he extended his stay..this happened 2-3 times and he would extend his stay.Everyday 24 hours I used to think about him..worrying and since he was at home he contacted me less…then one day i became so anxious and angry that I called him and his wife answered the call and I said the worst thing I could “Where is he? ..why are you not sending him back?…you know that hospital facilities are better here…he is also missing his college…I want to talk to him give him the phone.” Of course she didnt and after the call they had fight as she was asking who am I to tell her to send her husband….and she was right. Next day I apologized for my behaviour. He said that he has forgiven me and asked me for a help to talk in the office regarding his extensions. When he came back he sorted out the fight he had with his batch mates and started ignoring me. I tried to talk to him but all in vain.Finally after crying and begging for a month I also decided to ignore him. After about a week he came to me to talk and we started talking again.Everything was slowly being back to normal when one day a message popped in his phone and as i was sitting next to him I read it. It said “I love you too”…and it was not from his wife. When I asked him he said that she is his ex-girlfriend and that she is the only one he loves and will forever love. His father was against their marriage thats why he married where his father wanted him to. But he cannot forget her and they are still in contact even though she is also married. This shocked me. He said he didnt tell me cause people start judging and he did not want me to break the friendship. But I was so shocked I didnt know how to respond.I said this is wrong. But he said its his personal life …I shouldnt be interfering..I am a friend and I should know my limits. For next few days he was behaving mean to me and I also felt irritated. But then everything again slowly started improving. He was very helpful and caring…but I dont know something irritated me… I started picking up fights over small things…And since he has anxiety problem whenever we used to fight he couldnt sleep at night and this made him angry…and he said that he doesnt want this friendship any more..I apologised …promised not to do it again but I again repeated the same thing…we fought again and again…and our fights were really abusive and he warned me in dec that if I pick up fight with him again he will destroy me. Everything was going fine and I did it again on 7th jan irritated him over stupid thing and we fought.He slapped me in my face 3 times. said I dont deserve friends….blocked me from facebook and whatsapp…not talking in the college…I know I have no right to apologize and i should just give up. I dont know why I am doing this….I never opened to anyone like I did to him..he was the first ever person I got emotionally attached to…but why I didnt try to understand him…even though he has this anxiety problem…why I couldnt for once controlled my anger…I guess I am a horrible person.Now he acts like a complete stranger.What worse is he his last words that he said that I have not even a single positive thing about me. And that he regrets that he was ever friends with me.What should I do?