Does the office have a group of its own? If so, then I would suggest moving all discussion there and locking down the privacy settings. If not, I would suggest to your boss to make one. After all, does s/he really want the competition to know your company is having trouble making its sales quotas or whatever?
Another option, if the company is small, is taking it to a Facebook Message group.
Either way, once you tell your boss the facts of life about Facebook privacy (which stinks), the problem might start to resolve itself.