I don't know, Robert. Based on my experience it takes two people to play this game.
Sometimes in custody disputes like this the non-custodial parent gets to be what Mr. B calls "the glamour parent". The glamour parent is the fun parent who doesn't really have to enforce any rules, etc, which makes the job of the custodial parent very difficult.
It's really hard to know based on the limited information we've been given.
The fact that the kid cries when he has to leave the father tells me that something is going on. He's either being a glamour parent or something bad is happening at home between the kid and the mother -- in which case the poster has an obligation to the child to find out what it is.