Let's consider the case where we have
1) A speaker with ideas that are offensive to a significant percentage of the community (whether that be Richard Spencer, or the president of CAIR).
2) An audience, a significant number of people who either agree with the speaker or are curious and want to hear what he has to say.
3) A strong opposition, a significant number of people who are offended enough to want to prevent the speaker from speaking or the audience from hearing.
Frederick Douglas (who was personally invested in these issue) said that censorship hurts two parties, the speaker and those who want to listen. The question is in what cases does the opposition have the write to prevent the speaker from speaking? More importantly, who decides? I strongly disagree with the little I have heard from Richard Spencer. I would possibly want to hear what he had to say anyway. This doesn't mean I support him, it means I want to hear it for myself.
I suspect that most people reading this would say that the president of CAIR should be allowed to speak and that Richard Spencer should be prevented from speaking (by whatever means). If it is a majority rules... there are a number of Universities where a speaker from Planned Parenthood would likely be blacklisted along with Richard Spencer.
But who makes this decision?