can someone legaly say they are power of attorney over someone who didn't give consent?

I'm not a lwayer, scatman, but my layman's understanding of the situation is that your grandfather has not been judged incompetent by any court. If that is so, then where he chooses to live is his choice and nobody has a so-called "power of attorney" over him. Why do some of his relatives say they "have to sign him out"? Is he locked in or something? Can't he just pack up and go live with you if that's what he wishes? Those who claim to have this "power of attorney" would have to go before a judge to obtain such power.

Perhaps one of the lawyers on this forum will show up to either back me up or contradict me.

I am not a lawyer, either. You claim he is still mentally competent to make his own decisions. If so, he can still choose to live in the home or with you or wherever. That solves the immediate issue.
But from what I read there are family disagreements that are going to have to be resolved while he and his wife are still capable of making the decision about who to give power of attorney to.
It could get into a tugging match with the old folks in the middle. I don't know what to suggest.

How did he get into this home? Did someone take him there or the hospital place him there for care? Someone from the family would have talked to the nursing home staff. They want a family member for their records.

You need to ask these relatives to show you the papers that say they have power of attorney. Your father may have signed something giving them this power.