@oristarA,
The subject of the sentence is not the light of day, it is all the potential people who will never be born. It is they who outnumber the sand grains of Arabia.
The unborn ghosts are all the possible people who might have been born....they have never been born, thus, like ghosts, they do not exist. It is a poetic way of emphasising the point. He is making the point that of all the huge number of people who never came to be, there would have been potential genuiuses in every field, likely outclassing such wonderful poets as Keats ( a very well known Romantic poet in English) and geniuses like Einstein, Darwin, Newton etc.
Yes, the unfertilised ova and sperm who do not fertilise are part of what the author means....also, one supposes, all the people who might have been born of people who died in childhood, in war, of pestilence and starvation etc.
The prior state referred to is the state of non existence. He is saying how dare we whine because we must die and cease to exist when it is a miracle and wonderful privilege that we ever came to be in the first place.