You are making the erroneous assumption that everything these agencies do serves the public. Simpy because agencies like the FDA and EPA say they are protecting us, doesn't mean they do so with every action they take, and every program they implement.
I don't have a problem, in principle, with regulatory agencies. I think they are necessary. I never said that everything the government does should be in the hands of the private sector. I do believe though that competition would improve the efficiency and efficacy of any and all agencies, but I recognize that inserting that force into the process of regulation would probably not be wise.
A private police force would, just like a public police force, protect all of the people who pay for their salaries and equipment. Private police force models are not limited to an arrangement where only certain citizens pay for their own protection. A town or city could employ a private police force and require it to protect all citizens. Police and the military are not functions of the government that I believe, for several reasons, should not be privatized, but quality and distribution of service is not one of them.
The US Postal Service is a dinosaur on the verge of extinction, and their level of service has deteriorated considerably. While the Post Office sole source of funding is the fees it charges for its services, it obviously doesn't charge enough because it is $15 billion in debt.
Government may not seek profits but that doesn't mean they operate at lower costs than private companies. There is little to no competition with government services and so no incentive to cut costs. All of the agencies are driven to seek increased funding every year. It's their nature of the system. No agency wants to spend less than their budget lest their budget be cut, and all seek more money.
Again, there are very good reasons for government, but very, very few people are arguing that it be done away with altogether.
Don't mistake the intended purpose of government agencies with the actual ways they operate.