'The Mistress' - has been unstable.
Well off course, regardless of what you told her, she had hoped obviously, otherwise she wouldn't have continued and sounds to me that you didn't really respect her either, as "the lady" kept calling... would show a different thought there, than "the mistress has been unstable" or actually 'ex'-mistress.
So, all I can say is we make our bed, now we "lie" in it... You have to take what ever comes out of this.
What was the "honest" reason why you would not leave your wife? Because you are going to have to come up with a beauty, in my opinion, this "lady" when she sees you, will get angry because you cut off all communication and dropped her for dead.