@savari07,
The situation you're describing really isn't emotional neglect/abuse, as the law defines it--it's not as extreme.
http://www.americanhumane.org/children/stop-child-abuse/fact-sheets/emotional-abuse.html
But what's going on with that child is shameful, and no "difficult situation" those parents are in excuses or justifies that sort of inattention to the needs of the child. The father is an irresponsible parent, but since he's seemingly doing the absolute bare minimum the law requires of him, and not more actively emotionally abusing the child, it's not a situation in which CPS could even do much--except to,
maybe, tell the mother not to leave him with the father all day every day, but I'm not sure they could even force a man like that into parenting classes, which he certainly seems to need.
Fortunately, for the child, he's in school most of the year, so this is really a temporary situation--and I hope it is only a temporary situation. And, even more fortunately for this little boy, you are looking out for him and looking after him, so you're helping to mitigate his father's neglectful behavior. If you don't mind doing that, please continue doing that--you're doing it for the child, not the father. You are a very caring person and you are helping the boy.
Talking to the mother is a good idea. If she can make another arrangement for the child, whether at the daycare at her workplace, or with another relative, or in some recreation program at a YMCA, or school, or community rec center, or church, it would be better for the child. If she is genuinely struggling with a difficult situation and needs some help managing her life and providing adequate care for her children, encourage her to seek some social services help at community agencies offering help to families--help her find one, if necessary--there are many supportive services she might benefit from.
I agree with you, it's a frustrating situation to have to deal with. I feel angry just reading about that father's behavior. He may be dealing with all sorts of problems of his own, but he's not behaving responsibly toward his child, he's not acting like much of a parent at all.
You can always call CPS, not to lodge a complaint, or to ask them to investigate, but simply for advice regarding what else you might do, or suggest to the mother regarding child care. They are a good and useful resource and they are generally willing to try to help prevent abuse/neglect, rather than simply address it after the fact as they are most often called on to do.
Let us know what happens.