I am not a lawyer, but I think the keyword here is "expressive association". According to the Supreme Court's jurisprudence, the First Amendment guarantees a right to organize around a set of ideas, and to exclude from the organization members who don't fit into this set of idea. In practice, this means the Catholic Church has a right to exclude Protestants and Muslims; even the Klu Klux Klan has a First Amendment right to refuse Blacks or non-racists as members.
For the Boy Scouts, one of their defining ideas, expressed in the Scout oath, is a commitment to being "morally straight". The organization holds the opinion that homosexuality is morally skewed, or whatever the opposite of "morally straight" is. The Supreme Court held that the Boy Scouts are entitled to this opinion about morality, and that they are entitled to organize around their idea of morality.
The outcome would not have been the same if the man had been expelled for being the muslim, because the Boy Scouts would have had trouble demonstrating how their mission is inconsistent with being a Muslim.
The outcome would have been different for a business, because businesses don't exist for the purpose of expressive association.