Are you paid for your lunch time? I did work at one place where they did pay for your lunch hour so I know that some places do, however, for most places lunch time is unpaid. If it is unpaid, it is your personal time and I can not imagine that a work place could impose restrictions during your personal time. Also, is this lunch " at a company café or outside? Again if it is outside the company and during your personal time, I can’t see how they could restrict you legally.
A couple of things to consider: is they may be doing this to prevent any sort of harassment type of issues. Even during personal time, if you are with other employees, especially as a manager, you do need to be careful what you say and how you act. Another item, are you restricting any employees from attending lunch? This can be viewed as favoritism " in other words, you are spending quality time with some people reporting, but not all.
Have you taken an HR legal/harassment type of course? Most big companies do so. These types of issues are outlined and explained in these courses. It may be why HR places such restrictions. And seeing that HR advises against this (rather than actually restricts it), my guess is to prevent any inappropriate appearances. It would probably be best for you to follow their advisement.