Without matter there is no gravity
Primordial particles and even energy, have mass, and it's the mass which causes the curvature of space (not just matter) that we measure as gravity.
So even though there was no solid Baryonic matter in the early universe, there was still the force of gravity, and there were still distortions in localized curvature due to energy and quark distribution.
The real question is why is there variation at all? Why wasn't the early universe just flat and homogenized? There are some good theories to explain this already, but I don't think any one of them yet tops the heap.