Failed to get the humor: what is little numbers game on the side?

You are partly on the right track, but you do not get the joke.

Explanation of joke:

The cartoon depicts a 1930s-1950s stereotype American gangster or "hoodlum". He is addressing a man sitting at a table attached to a wall (hence oriented "sideways"). The man is accompanied at the table by three numbers 13, 64, and (I think) 7 in physical form. They are like large versions of the numbers house owners fix on their front doors. They appear to be playing a card game. One number is wearing an eye shade (he is probably the dealer in the card game). The hats, clothes, décor etc are cliché type visual clues.

A "numbers game" is an illegal lottery played mostly in poor neighbourhoods in the United States, wherein a bettor attempts to pick three digits to match those that will be randomly drawn the following day. These games are run by gangsters.

A senior gangster would be angry if a junior member of the gang were to run his own numbers game "on the side", i.e. as a subsidiary activity, secretly, unofficially and without permission.

The joke is that we are seeing a "game" in which "numbers" physically take part, which is visually sideways. Two double meanings.

In fact the game is "on the side" of the room.

That is, the game is a scam devised by gangsters.
Without the gang leader's nod, any member of the gang can't play, esp. secretly, the game.

You got it.