Playing with Magnet causes pain in my palm

What you may suffer from is an over-active imagination.

Seriously, I'd bet the pain you've experienced is probably there at other times, too. You're focusing on it more when you're holding those magnets because you're alert to see about any possible differences at that time.

Try holding something like another non-magnetic type of ball or sphere in your hand instead of the magnets and it may come back then - just as often.

The tiny microscopic amount of iron coursing through your veins at any point in time and the small magnetic field will not cause pain or even a noticeable physical effect or problem.

+1 what Ragman said.