If I read this right, these children are your younger foster brothers and sisters (as your mother made the decision to care for them). Do I nderstand this correctly?
It seems to me that they are your mother's responsibility. It doesn't sound like you should be the one to set limits for them.
I agree with you that using violence is the wrong thing to do. The answer is that these kids need clear limits with consequences for crossing them. They also need someone who is strong enough to care about them, and support them when they do the right thing while they are setting limits. This is a difficult thing to do.
If these kids are really out of control, then social services should be involved. Social services may be able provide counseling for the kids, and for your family.
But really, your mother needs to take the lead on this. It isn't fair for her to make you take the responsibility.