There is no Wisdom in Crowds

I'm afraid we Nebraskan's have no experience with crowds.

Oh, yes, you do! One of my friend who used to live in Omaha took me to a football game in Lincoln, and that was an experience I'll never forget - even though it happend several decades ago. Most everybody wore red, and we sat on one end of the stadium to watch Nebraskans come alive.

Be fair. Lincoln, Omaha, and Memorial Stadium are the only three cities in the entire state.

And you saw Cornhuskers come alive, although I will forgive you since every Nebraskan is also a Cornhusker.

They don't have cattle stampedes in Nebraska anymore?

This is a poor example to use considering that "voting" has only taken place for six days and only by the approximately 200 people who have thus far found their way through the learning curve to make use of that aspect of the new site.

i'm sure crowds can be stupid... but there's plenty of good collective action out there too... how about when village joins together for a harvest? or organized protests to overthrow a dictatorship, or rallies to fundraise for cancer research...etc.?

or are these not crowds?

what does the author of the book refer to as a crowd?

Dag, I'm going to say these are not crowds. Each of the examples have a heavy element of organization. Maybe I'm confusing crowd with mob, but intuitively, I have the same distrust of crowds as e_brown.

Since this topic just has to be about a2k voting, I'll mention that I've finally got my preferences set where I want them. All Posts, all Time. That's how much I trust this crowd to make a decision.

I am specifically taking about Crowd "Wisdom" (which I contend is a myth).

Obviously crowds working together are very effective provided they have some guidance. This is quite a bit different then the proposition that crowds somehow have an "intelligence" that can make rational decisions.

For crowds to act intelligently they need to be controlled, either by set institutions or by a few powerful individuals.

The examples given in this discussion that I agree are affective (e.g. harvesting, Google page rank, and the SETI program) all have a controlling institution that makes decisions that the crowd follows.

My contention is that attempts to put the decision making power in the hands of the crowd have failed. Technology like Digg, I would say is a fine example of a failure of crowd-controlled content. They not only return homogeneous results and filter out the most interesting results . They are also easily subverted... political sites from both sides routinely request their members Digg articles (meaning the most extreme, least nuanced articles are highlighted).

I am most skeptical of the idea that by taking the on-the-spot of opinion of everyone in a crowd-- some of which will be instinct, some have a motive-- and expect that in total it will represent some sort of "wisdom".

Crowds (for all their advantages in terms of sharing work) are even more irrational and less logical than the people who form them.

This thread was inspired by voting-- but I wrote it because the idea piqued my interest.

That being said, I just noticed that this thread is tied for having the highest number of votes of any thread today. Amusing, but I am not sure whether that supports, or discredits my thesis.

so do we know what the book author actually meant? (edit: I certainly don't... so it should be :"Does anyone know what the book author actually meant?")

Crowds are not necessarily wise. It's not that they can be depended upon to make the right decision, just that they can be depended upon to make a decision. Take, for example, a run on a bank -- classic crowd behavior. The bank may or may not be failing, but once there is some reason to suspect that the bank might fail, then each individual depositor has an incentive to withdraw his or her deposits from the bank. Furthermore, they all have an incentive to take out their money before anyone else does. So each tries to be the first in line at the bank, which just means that everybody shows up at the same time. The bank, which may not have been in danger of failing before, certainly is in danger once all of the depositors demand to withdraw their money simultaneously (for an example of how this works, click here).

Now, an economist will tell you that the depositors are acting rationally, and clearly that's correct (a game theorist would tell you that a bank run is nothing more than a type of "prisoner's dilemma"). But to say that the crowd is rational is different from saying that the crowd is right. The consequence of the crowd's action might very well be worse, overall, than if it hadn't ruined the bank.

Thomas's example of language is a good -- and often overlooked -- example of the wisdom of crowds. There is no official mechanism for determining how people communicate, and those institutions which attempt to do so (such as the Acadamie Française) have a very poor track record. But again, just because the crowd makes a determination doesn't mean that it makes the right determination. For instance, it is rapidly becoming acceptable to spell the phrase "all right" as the single word "alright," even though the change drives language purists to despair. Once the crowd accepts "alright," though, it will become standard English, and "all right" will look as quaint as "base ball" or "aeroplane."

Indeed, we have a tendency to "ratify" the wisdom of a crowd's decisions in retrospect. Stock prices go up or down depending upon the collective decisions of hundreds or thousands of investors, and we then declare that a company is worth a certain sum of money based upon its stock price. The company, in other words, is worth whatever the mass of investors says it's worth, which means the crowd is never wrong, and it would be foolish, or at least counter-intuitive, to disagree with it. Joseph Heller, in Catch-22, neatly summarized the paradox: when asked what if everyone thought as he did, Yossarian replied "then I'd be crazy to think anything else." The crowd thinks the bank will fail, and in causing a run on the bank leads to its failure. The crowd thinks a company is undervalued, and causes its stock price to rise to match its higher valuation. In both cases, the crowd makes an accurate prophecy, but that's largely because it's a self-fulfilling one. In sum, crowds are right more because they are crowds than because they are right.

Language is an interesting example... although I am not sure if this can be called "wisdom". The important thing in language is that crowds all change together (i.e. people need to follow the crowd). This is more an example of "flocking" than of "wisdom". Also interesting is the role that controlling elites (especially the media) play in changing language.

You are incorrect in your assertion that investors "are never wrong". Look at Enron, the crowd overvalued the stock... but when logic (i.e. reality) hit, the crowd turned out to be quite irrational (and paid dearly for it). There are plenty of examples of this.

The point is that crowds are quite irrational. Runs on banks is another fine example. If a small group of investors got together with control of how the larger group would act, these irrational and self-destructive stampedes wouldn't happen.

No I'm not. I'm giving you a wide range of examples. How is digitizing books (my recaptcha example) an example of predicting the behavior of a crowd? How is wikipedia's collaborative article writing an example of predicting the behavior of a crowd? How is Linux's collaborative code writing an example of predicting the behavior of a crowd?



I have, but you seem to be deliberately ignoring them.



The system has flaws and lacks for diversity in the data at this early stage. Like has been said so many times, it will only become useful with scale, and there are some pretty critical flaws in the way it works that need to be fixed. That's not a perfect example of anything you are saying about the validity of the concept itself.

I wouldn't say that there are "controlling instititutions". Google doesn't control the PageRank of specific websites, or the results you get when entering a search term. Google controls the framework.

The actual results are results of crowd wisdom.

Now, you can certainly quibble with how effective any given kind of framework that attempts to harvest crowd wisdom is. Is a constitutional republic more effective in harvesting crowd wisdom to govern a country than a parliamentary system? Is the Google Image Labeler more effective than Yahoo's attempt to tie Flickr into its image search results?

The quality of the framework therefore determines very much the results you get from harvesting crowd wisdom. But I don't think that having a poorly thought out framework negates the existence of crowd wisdom.

In the sense that the term is used of course it can. "Wisdom of crowds" doesn't mean that you are supposed to be able to make a value judgment to call the crowd "wise", just that there can be crowdsourced intelligence and that it can have useful purposes.

For example, it doesn't purport to mean that the crowd is wise overall. Many examples of crowdsourcing involve finding ways to filter the wise out of the foolish in the crowd that is collectively providing data.

So yeah, it's easy for you to use word play to portray the situation as a dumb crowd and brilliant individuals extracting something useful (like you did with Google) but that's often how the wisdom of crowds is found.

For example, you say it's only useful to predict how the crowd will work, and often it's used to predict something about you, the individual. For example in the recommendation engine systems, it can recommend things that you are likely to want as opposed to what the crowd is likely to want.

And you just can't write off the Google example that easily, it's the preeminent example of this generation. Google came to prominence based off the simple concept of pagerank. They theorized that when people link to other pages they generally perform a form of editorial selection of the link (choosing better ones over worse ones) and that the collective opinions of the masses could provide relevancy ranking that is useful to the individual.

There is financial incentive to manipulate their rankings, and the scale is intimidating but they use their wisdom to find algorithmic ways to provide the crowd the best of its own wisdom. They don't like to manually tweak results at all, and their smarts are in extracting the cream of the crop from the crowd. How can you describe the pagerank algorithm, which every major search engine now uses a variation of, as not mining the wisdom of the entire internet? And how is the entire internet not a crowd?

In the case of a run on a bank, you have a crowd where each single member is highly aware of other crowd member's actions. Which means that each members actions highly depend on the action taken by surrounding members.

The question is whether this is the best kind of framework to get the best results from crowd wisdom.

Take, for example, the Google Image Labeler. Each member of the crowd is only aware of his own actions in labelling an image. Only when the two members enter the same text for any given image, the 'game' proceeds to the next image.

After finishing a 'game', the players get to look at the labels the other player used for the image, and on with label the two were matched.

Unlike in the example of the run on a bank (which seems to be a rather ineffective 'framework'), this set of rules seems to be much better able to harvest crowd wisdom.

(for reference: the Google Image Labeler)

There is an excellent video of a talk from the 2007 TED conference that helps visualize the concept of scalability in social networks, folksonomy etc., as it is being applied to photographic images from around the world.

http://www.ted.com/index.php/talks/blaise_aguera_y_arcas_demos_photosynth.html

James Surowiecki, the author of the book, does not give a hard definition the term as far as I remember. The phrase is mainly a play on Charles Mackay's classical book title called "The Madness of crowds", which deals with stock market bubbles, which hunts, crusades, duels, and other examples of crowds being crazy. Surowiecki's book is intended as a counterpoint to Mackay's. It isn't intended as a refutation. Surowiecki doesn't say anything remotely like "crowds are always wise".

He only says that they sometimes are. Surowiecki describes, with esxamples, the aggregation of information that takes place in crowds, and how it can make the crowd smarter than its individual members. Again, he doesn't state that crowds are always wise.

For a fairly good impression of what the author is about, I recommend his interview on NPR's Dianne Rehm Show.

Also, the introduction to his book is available on Amazon as a preview. It gives a flavor of what he means, though it doesn't give a hard definition.

That doesn't quite answer what you asked, Dasha, but that's what I wanted to say.

Google Image Labeler is a neat idea... I am skeptical about whether it really leads to better searches.

I have been looking for any research on whether it is effective or not. I am a bit skeptical (especially after playing the game) about whether it can dramatically improve image search results.

Has anyone seen any quantitative research on the effectiveness of methods such as this?