How do I rank entries with given average grade and different number of occurrences?

Probably the easiest way is to require a certain number of entries in order to get a full average. If you said the average was provisional until five grade were scored, you could just use the average.

Professional sports use engineer's suggestion for things like quarterback rankings. The idea is you can't come off the bench for one game, have the game of a lifetime, and be considered the top quarterback in the league. So, there's precedent for that approach.

On the other hand, assuming you're a teacher with a real-world ranking problem, you might ask yourself what behavior you're trying to encourage and come up with a ranking system that rewards that behavior. In your example, it would appear that X is a (smart?) slacker who needs to complete more assignments. One alternate approach would be to rank by grade points (total, not average). In this example, X has 5 and Y has about 121 or 122. Of course, this would rank X alongside Z who completed 5 assignments and scored 1 on each of them.

Another approach would be to discount average grades based on how much of the expected work has been completed. For instance, if X should have completed 3 assignments by now, he gets a score of 1/3 * 5 = 1.667. If Y should have completed 30 assignments by now, he gets a score of 26/30 * 4.68 = 4.056. With this approach, if somebody gets ahead of expectations, he could be rewarded with a score above 5. For example, if Y should have completed 20 assignments, his score would be 26/20 * 4.68 = 6.084