Don't be confused, yes, you are being abused. He is manipulating you, threatening physical violence, gas lighting you and causing physical and emotional harm. No, you didn't ask for this, it's not your fault he's an abuser.
Please file a police report about the incident and start the procedure for a restraining order.
Start building a support system around you. Notify your landlord (to see if you can change the locks & not allow him into the premises), neighbors (so they can help/notify you or the police if they see him), family and friends what's been going on. He's counting on your silence, confusion and lack of support.
Keep an emergency phone in a location that has a locked door. If he does enter, lock yourself in and call the police. Try not to escalate the situation, he wants you to react so he can keep the abuse going. Try to remain calm, and answer his questions but don't overly compensate. Again, he wants a reaction, don't give it to him.