See, this is the thing about people who mistreat us.
We don't owe them a damned thing. You don't owe him friendship, etc. If you want to be friendly with him, then it is your prerogative, but understand that you need not be involved. You're more than allowed to be angry enough at poor treatment to tell him to go scratch and delete his contact info.
PS Given that he works, how much older than you is he?