What you are describing is creepy. There is no acceptable reason for a man to follow you that way. That is not normal and I would be worried if someone did that to me.
You are correct to seek balance. It is statistically rare for women to face violence from strangers in most cities in the United States (we hear the headlines, but when you look at the data it is significantly more likely for men to face violence than women). That doesn't mean you shouldn't be careful, but you shouldn't be paranoid.
My son was pepper sprayed once by a woman who saw him as a threat. He was 13 at the time... the problem likely was his skin color. It was traumatic for him, and I was pretty upset.
There is a danger in overreacting to fear... it hurts real people. The fact that you are thinking about this, and working to find a balance is a very good thing.
That being said... In the case of someone following you even when you stop and change direction, I don't think you overreacted.