itself/themselves

I would go with 'itself', if only because 'family' is singular.

If we were speaking of family members, then "The family members kept mostly to themselves", is what I would use.

There are contexts where the plural verb is more natural: My family are always fighting among themselves. When the individuals in the collection or group receive the emphasis, the plural verb is acceptable. Generally, however, in American English, collective nouns take singular verbs. In British English, collective nouns are more often treated as plurals that take plural verbs.

Thanks, roger and centrox.

I should have remember that you were orientated on British usage.

Thanks, roger. I use British English.