@Thomas,
His opinion that conservatives have turned into a cult of lunatics is just that, his opinion and one that is so absurd it hardly bears consideration.
It's obvious to all who frequent this forum that I have little use for Liberals and yet even I wouldn't call them a cult of lunatics.
But let's address your question.
If I came to the conclusion that all of my friends were members of a lunatic cult then in absolutely all likelihood, I had become a lunatic, and so whether or not I "left" the imaginary cult, of which I had never been a member, would be immaterial.
A point in fact is that not all of the "friends" of Justice Roberts have criticized him; not by a long shot.
If he ever considered all Republican or even conservative politicians, pundits and bloggers his "friends" then he is, emotionally, a child; which, of course, he is not.
Subscribing to liberal or conservative ideals does not signify membership in a cult (particularly as Posner obviously defines the term).
If you are a liberal or a conservative and you find that some people who you previously felt shared your ideals (hardly the definition of a "friend") are now lunatics, it is lunacy to jetison your previously held ideals and adopt their opposites.
Depending upon how you view a "normal" human, you may consider Posner's comments in line with "normal" human reaction, but I don't have so low an opinion of humans (nor do I think you do) to consider abandoning one's principles in the face of rather ineffectual criticism to be normally human.
Since I don't believe Posner is a simpleton with the IQ of a very young child, he must be an asshole.
What you should consider is that you are not only accepting his obviously absurd argument because it comports with your politics, but that it has caused him to rise in your esteem.