Why won't he leave me alone?

There was this guy I met when I just moved to a new city. I was stranded so I did not know my way around and I almost got lost. He approached me to give me a ride to wherever I intended to go. I was sceptical at first because we all know we should never enter a stranger's car but I had no option because I was stranded by the roadside. I voiced my concern and we had this back-and-forth of him trying to reassure me it was fine to hop in while I displayed reluctance. He got irritated from the look on his face and tried driving off at that point I figured it was better to let him take me home than be left alone at night in an unknown area, so I entered.

He told me he was harmless and he winded the car window down in case I wanted to shout for help if I got scared. He told me he stayed around and pointed to his mechanic shop he also greeted a few policemen who smiled at him on the way and called his name.
I knew my house but I did not know the route to get there since I was new to the area and the houses looked similar. He was patient and helped me ask for directions from passersby. We got lost a couple of times and with the help of people we got redirected and got on the right path. I was embarrassed because I was stressing him out yet he told me it was fine and kept on smiling at some point, He patted me on my thigh. I was startled so I jumped since I did not expect or see that coming I told him to keep his hands to himself. He apologised and from that moment he kept mute throughout the journey. I could sense he was upset so I told me he could drop me halfway home but he insisted so he dropped me off somewhere close to my house. He asked for my number and called me when he got home but I was already asleep and saw it the next morning so I sent him a thank you text later in the day. He replied with a "you are welcome sticker" and did not call me again.
We crossed paths three weeks later so he started calling and said whenever ladies are rendered assistance they automatically assume the man wants a relationship and that was why he did not call me after I gave him my number. He said he was trying to calm me down the other day and I flared up at him. I apologised and he apologised too for making me feel uncomfortable with that gesture.
We started communicating again though sparsely. Once a week he would text and we would talk. He called and told me when I come over to his place he would cook me dinner though he did not ask to see me or invite me over directly however when I call he doesn't fail to plead that we meet up yet he will not outrightly pick his phone to suggest it instead we just talk and ask how I am faring.
I finally agreed to see him and he suggested I come to his house. I got scared and told him I don't go to a man's house if I was just meeting him for the first time. He said he was no longer a stranger considering we have been talking for three months that he is harmless and won't hurt me but I still felt it was inappropriate having the first meeting in a man's house. He told me I went to another man's house( he found out when he called and I told him I was at my friend's house) but I do not want to visit him at his house to which I explained the situation. ( He was a friend of my brother's whom I knew before him and I recently discovered lives in the same city though I met him recently) even though he did not, believe me, I told him we could go out and later if I feel comfortable around him I would visit him in his home. He kept on trying to convince me to visit him at his house and I got upset. He said he would call the next day to fix a date to go out and that was the last time we spoke till two months later when he called me up. I did not call him during that period too

I felt he ghosted me so I responded nonchalantly and unfriendly
I later felt bad and called him to call me later when he has the time to talk because I wanted to talk to him. He said he would but he hasn't. But he has a habit of coming and going so I know he would still call me. Why doesn't he just leave me alone if he doesn't want to be friends with me?

Answer: Because you never told him too.