Do I understand you correctly? Do you really mean that adultery is merely a matter of what the two people in a relationship agree upon?
If you are saying this, then I agree with you wholeheartedly.
Of course, this means that adultery is no different than anything else a couple may or may not have agreed upon. Church attendance or who stays home with the kids are similar issues where one side may have made a commitment they no longer want to keep.
I concede the point that a bad relationship can be worse than divorce. Clearly you are correct about this. However, infidelity doesn't condemn a marriage to this category. There are many marriages that have included infidelity, even where lying is involved, that end up as working relationship.
I disagree with you about social expectations. Society expects a commitment of exclusivity in marriage as the norm even though a significant part of society will break this commitment. It is as if society is setting marriage up to fail.