@Tabs,
I honestly don't know if this is just innocent affection or not, but a common precursor to sexual assault is grooming. Grooming typically involves all the things in the original post.
This man is a bit young to be a grandfather and the children he is interacting with are not related to him. I wonder if their parents are even aware that their children interact with him and would be comfortable with it. We also have to respect their parents regardless of whether this behaviour is wholly innocent or not.
It would be horrible if social services careened in and damaged this man's life without any evidence. I don't think that would happen. And surely, it's a worse possibility if a child is harmed. Better to be safe than sorry.
A common mistake is people think pedophiles are aware of where their actions are taking them. They often aren't. So, potentially this man could be going down a path that will lead to a crime being committed. Better to catch it early and save potential victims and the potential perpetrator before it's too late.
What can complicate things is that the OP had, at one time, an emotional connection to this man that became so damaged that they broke up. You might think that disqualifies her word. It doesn't. Social services are able to distinguish the facts from pure speculation. I'd recommend she goes to them and not delay any longer.