@awkwardcase,
What do you want to do? Do you want to face him or not? Would it bother you or inconvenience you in anyone to meet him?
You are in control - if you do not have any interest in meeting him, then say thank you I appreciate you getting my stuff back to me, but it would be tough for me to meet you right now, could you just drop it in my mailbox instead?
You don't have to give a reason why it is difficult - whether it be you just don't want to see him or you are busy.