@spidergal,
Based on how aware he was of time's ya'll had met, and based on his volunteering about having "seen you at the mall", I'd say there's definitely enough evidence to warrant your being cautious.
I'd suggest that you call back your HR rep and mention, "you know, I've been thinking about the exchange yesterday and it concerns me that he volunteered how he'd seen me at the mall. This behavior obviously goes beyond the workplace, and I'd like it noted in the records you're keeping."