I am smarter than the crowd (and so are you).

Forums: Wisdom Of Crowds, Politics, Folksonomy, Crowdsourcing, Science And Mathematics
Email this Topic • Print this Page

Can the "crowd" do a better job than an individual at making correct estimates?

This claim has been made that by averaging the guesses of a crowd, you are more likely to get the correct answer than by having an individual person make an estimate.

The typical story is as such:

Quote:

In the 1800's there was a cattle baron who argued that the common rabble at a country fair could never guess the weight of a prize bull, but that an educated cattle man could do it.

So they had a contest. All the bumpkins at the fair got to guess at the weight of bull and anyone who got it right would win a prize. They ran the contest all day and had several hundred guesses, but nobody got it right. But when the baron averaged all the guesses, the average was dead-on. So even though no single individual in the crowd got it right, the crowd on average did.

Of course, one story doesn't prove anything. This calls for a scientific study.

I am suggesting a scientific study-- and I am volunteering to offer myself as guinea-pig, to face the crowd. We need 20 questions... with answers that no one knows, but with clear answers that will be knowable in the future. We will then ask the crowd through a vote, and I will give an answer. and we will see who does better.

My suspicion: as things are often not what they seem, the crowd will often be quite wrong (of course it will get it right sometimes).

I will also be wrong somtimes. But, there will probably be some questions where I have some insight (as all individuals do). On these questions I will do better than the crowd. My prediction is that because I can identify the insight I have, where the people in the crowd with the same insight will be voted down by the others, I will have a slight advantage.

But we won't know until will try...

Does anyone want to come up with questions?

Topic Stats
Top Replies
Link to this Topic

Type: Discussion • Score: 4 • Views: 5,395 • Replies: 27

@ebrown p,

Ok, I'll bite. I'll offer you two questions

How much does my dog weigh?

and

What time will I be washing dishes at home today?

1 Reply

@fishin,

You want to take a poll on how much your dog weighs?

Sure. That is not a bad one. The rule is that you can't not make any information on your dog available to me, or to the voting public until after the vote.
The dishes one seems problematic...

Question #1: How much does fishin's dog weigh?

2 Replies

@ebrown p,

ebrown wrote:

This claim has been made that by averaging the guesses of a crowd, you are more likely to get the correct answer than by having an individual person make an estimate. [..] Of course, one story doesn't prove anything. This calls for a scientific study.

I thought you'd claimed that this research had already been done? You know,

Quote:

"Nihm (in his somewhat patronizing manner) is stating, without proof, that voting with a large crowd produces results that are "better" . There are theoretical reasons that these schemes are not good for classifying data (which are backed by research).

(The only contexts I was making that argument in, by the way, was by every time including specific examples of where I personally, anyway, saw it to be working, and how. Probly doesnt count as scientific "proof", but it sure beats the generalised assertions I was responding to.)

Anyway, you dont seriously believe you can do a "scientific study" by starting a survey on a2k, do you? Basically redoing the test you cite, but in a controlled, self-selective, and non-random environment? Isn't this just flippancy? Wouldn't you do better responding to the specific, reasoned arguments responding to your quick dismissals of examples like wikipedia and last.fm on the other thread, instead of just starting anew in a third thread in a row to assert yours?

0 Replies

@ebrown p,

ebrown wrote:

Can the "crowd" do a better job than an individual at making correct estimates?

This claim has been made that by averaging the guesses of a crowd, you are more likely to get the correct answer than by having an individual person make an estimate.

OK, nevertheless - if you're going to go ahead with this, you should set down how you're going to do the comparison. Getting the crowd's take is simple enough - people randomly guess the answer. (Except for the "crowd" in question will lack, of course, both the scale and the random constellation that any actual example of crowd wisdom would involve, of course, which is why the test is bogus from the start, but I already said that.)

Now, which individual's estimate are you going to compare the crowd result with? Yours? Then you should make your guess now, of course, because if you do it only after the guessing has begun, your estimate would be influenced by ... crowd wisdom. Someone else's? Obviously not "whichever individual got closest to the result", because that's a set-up whereby you determine the outcome in advance through the mechanism, and merely shows that "whichever individual turns out in the end to have gotten closest does better than the average of guesses (duh)".

3 Replies

@nimh,

This is talked about in "Stumbling on Happiness" -- with cites -- I'll have to try to find it though. (Darn paper books.)

Has anyone mentioned "who wants to be a Millionaire" yet? The "ask the audience" lifeline (AKA the "wisdom of crowds" lifeline) was/is a pretty trustworthy one.

Quote:

Ask the Audience: The contestant asks the studio audience which answer they believe is correct. Members of the studio audience indicate their choices using an audience response system. The results are immediately displayed on the contestant's and host's screens. This is a popular lifeline, known for its near-perfect accuracy. Philbin once said that the audience's answer is statistically 95% of the time correct.

http://en.wikipedia.org/wiki/Who_Wants_To_Be_A_Millionaire%3F

1 Reply

@nimh,

I agree with nimh about ebrown voting first so he can't secretly leverage the wisdom of us, the crowd.

***

On nimh's point of what's a fair comparison: I take it that ebrown is not trying to prove that he's an especially smart individual. Given that, I think the fairest test is to count the number of individuals who guessed better than the average (there will surely be some) and compare it with the number of people who guessed worse than the average. If the wisdom of crowds is for real, more individual guesses than not should be worse than the average guess. Conversely, if more people than not guessed better than the average, ebrown p has proven his point.

0 Replies

@nimh,

I am closer to the real answer than the crowds average, I win. If the crowds average answer is closer, I lose. That's the claim that is being made; that the crowd's average answer will be better than the individual's (i.e. my) answer-- so this is the fair test.

I agree that I should make my answer before I know the crowd's average answer.

But my claim is that individuals are smarter than crowds... so using my individual intelligence (however I like) is part of the contest.

1 Reply

@sozobe,

Who wants to be a millionaire is a completely different situation...

If one were to define the "Wisdom of Crowds" as the ability to do multiple choice questions on popular trivia, I would accept it. Whether this ability can be considered "wisdom" seems a bit of a stretch to me...

This is not much of a valuable trait for any real questions...

I just thought of an interesting question about the Millionaire show... are people allowed to not vote if they don't know the answer-- and how many people in the audience don't vote for each particular question? If people in the audience are only voting when they think they know the answer, it makes the 95% figure much less impressive.

3 Replies

@ebrown p,

Oh, and we have two questions, not one: the second is when Fishin will be washing the dishes today. Why don't we start with these two? We don't have to wait until we have 20, do we.

0 Replies

@ebrown p,

Quote:

If one were to define the "Wisdom of Crowds" as the ability to do multiple choice questions on popular trivia, I would accept it.
Quote:

Isn't that awfully close to what you're proposing here? Multiple choice makes the difference?

It seems like the most pertinent sort of question (to A2K) is the just the sort of average that was in your original answer. That's what the rating system does -- some will vote things down (guess too low), some will vote things up (guess too high), and the resulting average is -- hopefully, and given enough input -- about where things "should" be (close to the actual weight of the hog).

edit -- I see that fishin's questions ARE both averages, that's good.

Quote:
Whether this ability can be considered "wisdom" seems a bit of a stretch to me...

It's the right answer. <shrug> That's what you're proposing as a metric here... the right answer.

Quote:
I just thought of an interesting question about the Millionaire show... are people allowed to not vote if they don't know the answer-- and how many people in the audience don't vote for each particular question?

When I watch the show, (which admittedly isn't often), it looks like everyone votes. Didn't find a definitive answer with a quick search.

0 Replies

@ebrown p,

First, define "wisdom."

1 Reply

Here is one on the web. You guess the number of NYC cabs.
http://www.pbs.org/wgbh/nova/sciencenow/0301/04-cabs.html

Like they pointed out. There needs to be an incentive for the crowd to want to be accurate otherwise a few wise acres can skew results. This explains why the trading sites that trade futures in election results are so often accurate. It is the wisdom of crowds. Different people have access to different information and use that information to make a bet. The result is that in totality they probably have all the information to make the group decision.

0 Replies

@ebrown p,

ebrown p wrote:

If one were to define the "Wisdom of Crowds" as the ability to do multiple choice questions on popular trivia, I would accept it. Whether this ability can be considered "wisdom" seems a bit of a stretch to me...

And yet, when I dedicated a thread to an impossibly difficult, open-ended task, and how a crowd of incredibly dumb members solved it without them even trying, you objected that this task was too open-ended, so the concept didn't apply. Respectfully, I submit you're moving goalposts here: If crowds can solve the problem, it's a completely different situation because you decree it to be either too open-ended or not open-ended enough.

0 Replies

So, when does the crowd start guessing? Can I guess if I've met fishin's dog in person?

1 Reply

@DrewDad,

Wisdom is the reward you get for a lifetime of listening when you'd have preferred to talk.
Doug Larson

0 Replies

@ebrown p,

Quote:

I am smarter than the crowd (and so are you).

I agree. I think that's why this website's creators built it so we can tailor our own experience.

0 Replies

I had an algebra teacher in jr high school that let us vote on the correct answer to a few problems. His point? Majority may rule, but isn't always right. He made his point.

1 Reply

@ebrown p,

In actuallity the dog weight question poses the larger problem for purposes of the "test" since there are people within the crowd that know my dog or at least know about it. On the other hand, no one other than myself knows anything about when I was my dishes on any given day.

A part of the entire concept of the "Wisdom of the Crowd" concept is that some people within that crowd will have more knowledge on the specific issue than others.

For example, if I had asked the question "What color is my house?" and some of the crowd had visited my house, they may recall the color and not only disclose the color, but state "I was at his house last month and it's red!". Someone else might pipe in and hold up a copy of an email I had sent them mentioning that I was in the process of paiting it red as well. I wouldn't have to make that info available, others with the knowledge could do it on their own.

The more people there are with knowledge of the specific question asked, the more likely it is that the crowd will start being swayed from completely random guesses to the actual (or near actual) answer. That is the nature of crowd wisdom. As the crowd "learns" the guessing ends and the you get more responses that are the accurate answer swinging the crowd's overall response closer and closer to the accurate answer.

On any given question, a single person with perfect knowledge of the topic could out-guess the crowd pretty much every time. But no one has perfect knowledge of everything and the "crowd" has been being surveyed on many topics for years, decades for centuries. For example, our entire educational system is built on what "the crowd" has decided is necessary for our children to know to effectively survive in adulthood in our society.

1 Reply

@ebrown p,

ebrown p wrote:

Quote:
In the 1800's there was a cattle baron who argued that the common rabble at a country fair could never guess the weight of a prize bull, but that an educated cattle man could do it.

So they had a contest. All the bumpkins at the fair got to guess at the weight of bull and anyone who got it right would win a prize. They ran the contest all day and had several hundred guesses, but nobody got it right. But when the baron averaged all the guesses, the average was dead-on. So even though no single individual in the crowd got it right, the crowd on average did.

Of course, one story doesn't prove anything. This calls for a scientific study.

It's been studied extensively since 1860 and this has already been pointed out to you. That anecdote sparked a lot of scientific research that you are ignoring to do your own "science".

http://en.wikipedia.org/wiki/Regression_analysis
http://en.wikipedia.org/wiki/Law_of_error
http://en.wikipedia.org/wiki/Standard_deviation

Quote:

I am suggesting a scientific study-- and I am volunteering to offer myself as guinea-pig, to face the crowd. We need 20 questions... with answers that no one knows, but with clear answers that will be knowable in the future. We will then ask the crowd through a vote, and I will give an answer. and we will see who does better.

You dismiss the ox example with it's 800+ participants as unscientific and then think what you are doing is "scientific"? Laughing

1 Reply

I am smarter than the crowd (and so are you).

Related Topics

Quick Links

My Account

able2know