How old are these children?
Before anything else, you must put a stop to the choking behavior. That is very serious, and very dangerous. A choke hold from behind can easily damage the windpipe of the other child, or cut off the oxygen supply to her brain, and this can be fatal. Even professionals, when trying to subdue someone who is agitated, are taught that you never grab the person from behind with the arm across the throat. And trying to strangle her is certainly equally dangerous, and equally disturbing. He cannot be allowed to behave in this way either toward his sister or toward any other child. And, quite honestly, when he chokes her so hard that he has left red marks on her neck, she is being subjected to legally abusive behavior, in addition to being endangered, and you and her mother must put an immediate stop to it because you are responsible for her welfare.
Regardless of the provocation from the girl, you have to make it very clear that the boy is not to put his hands on her or touch her, and she is not to touch him either. There is not to be any wrestling around, even in play. There has to be zero tolerance for this. You cannot trust either the boy's impulse control or his judgment, and you have to draw a firm line when it comes to physical contact or a physical reaction on his part. Give him other options--i.e. he can leave the room if she is bothering him, he can tell you or his mother about what is going on, he can respond verbally to his sister and tell her to stop annoying him, etc. But, under no circumstances can he put a hand on her until you have this situation well under control. If need be, either do not leave the children unsupervised when they are together, or have them in separate rooms if someone cannot keep an eye on them.
Most of the rest of what you describe sounds as though the children grew up in a chaotic household without consistent discipline, structure, or rules. Why are they throwing food on the floor, or spitting on the floor? Why has this been tolerated? Why don't they put things back in the fridge or freezer or put covers back on jars or close cereal boxes? Truthfully, this sounds like a lack of consistent proper parenting. These children are just doing whatever they feel like doing, without any adequate external parental controls to help them manage their impulsivity. They need to learn new habits about how to behave--family habits that will apply to everyone. As in, no one in this family spits or throws food on the floor, everyone must put food back in the refrigerator after taking some, lights are not to be turned on unnecessarily, trash belongs in the trash basket etc. and these things must be established as the rules of the house--for everyone. And let the children participate in a discussion for deciding on the penalties for breaking such rules, so they feel this is a joint family effort to make life better for everyone. You can't just set rules, you have to enforce them in some way, and consistently, using praise, rewards, penalties, whatever works.
Similarly, let the children help with the dinner preparation if you can, so they are eating something they helped to make, or had some say in planning. Let them help set the table and clear the table. Again, make it a family affair, with everyone pitching in. Talk to them about not wasting food, about how you can all save up for a special family treat of some kind, if you can cut down on the food and money that's being wasted. Again, try to enlist their cooperation and involvement so that meal times do not become a power struggle.
The behavior you describe from the boy-- domineering his sister, lying, saying inappropriate or distasteful things, sounds more problematic. Is this how his father acts, or acted--domineering? You said the mother's marriage had been mentally and emotionally abusive--were the children on the receiving end of this too, or did they just witness it? Is their father still involved in their lives?
Your desire to have some ground-rules for behavior in the home, and to expect these to be followed, is very appropriate, and it sounds as if the children sorely need these rules to help them learn some sense of self control, some sense of order, and some sense of responsibility. And it sounds as though you've already done everything you can to bring that about without much success. I don't think this is your failure, I think you walked into a situation that was already dysfunctional and it's very difficult to get a handle on it.
I think you're in a situation that needs some professional intervention because an awful lot of things seem to be going on that are already somewhat out of control, and some of the things you describe could escalate into very serious problems. I think you and the children's mother should make an appointment with a psychologist who specializes in both child and family therapy. Someone like that can be of enormous help to both of you in terms of simply advising you how to handle various types of behaviors that the children are displaying and how to correct behavior in the most effective way. In addition, that person can decide whether to see the family as a unit, or to see each of the children separately and then meet with you and their mother without them being present.
It does sound as if the boy might definitely be in need of therapy, given the behavior you describe, and the girl might also need therapy since she has been on the receiving end of her brother's abusive behavior. Both of these children may still be dealing with the residual problems of what went on between their parents during the marriage, and these emotional problems should be addressed now in order to prevent more lasting damage and more serious problems down the line. And, obviously, you and their mother can use some guidance in how best to handle the behaviors you find distressing.
Asking for help here was a good start. But I think you need some professional intervention to help get the home situation under control and to determine whether one or both of the children needs or would benefit from therapy. They apparently have lived through a difficult past situation, and their parents are now separated, and you are an additional presence in their lives. These are not easy things for children to process, nor are they easy things for you or their mother to help them handle. That's where therapists can be very helpful, to everyone concerned. See if you can find a good psychologist who specializes in child and family therapy, or simply a good child psychologist. I really think that's your best option, all things considered. You and their mother should probably meet with the therapist without the children for the first session, and talk about the sorts of things you've mentioned here, as well as what the situation was like for them during their parents' marriage, and how they have reacted to the separation.