He seems to think he still owns the property. Hence I've got questions about that so-called gift. Perhaps he only intended for you to live there for a while, or gain some income from it briefly or whatever.
Be that as it may, he sure as shooting thinks it's his now.
How about asking him - "Gramps, I thought this place was mine." And he'll say no or yes or whatever.
Is it worthwhile to pursue legal options? Possibly. Real property of course is $$ but at least in the US oral promises aren't going to mean diddly without paperwork. Not to mention the fact that taking him to court likely means harming, perhaps irrevocably, your relationship with him and possibly also your mother.
What to do? Think long and hard about what it's worth to you. Is this your only place to live? Your only source of income? Are you already estranged from him and your mother? Then it might be worth it to sue, but keep in mind that your chances of prevailing are pretty slim here in the US.
Or let it go and tell yourself that, if there is ever a next time that you find yourself in a similar situation, insist on getting something in writing. It is not a lack of trust and you are not being a jerk. It's to make sure that everyone remembers everything correctly and with complete agreement.