There is no Wisdom in Crowds

@ebrown p,

ebrown wrote:

I love Wikipedia... however it has well-known problems of exactly the type you are downplaying which have required banning users and locking topics. And, it is still not very difficult to see (or add) incorrect information (I take it you are not a Colbert fan).

I followed the Colbert campaign. You are aware that it didn't have any impact beyond the moment? Sure it's easy to momentarily add incorrect information - as Colbert, rather redundantly, proved. But the whole concept of the "wisdom of the crowd" as it appears on Wikipedia, as I understand it, is in how quickly the crowd will then correct these interferences.

Colbert urged people to change an item, and of course a couple got through; but the bogus additions were corrected as quickly as they were made. Thanks to crowd wisdom -- in which even a concerted attack to misinform is overridden by the sheer numbers of people editing Wikipedia on a day-to-day basis.

It's actually easier to insert misinformation into Wikipedia by doing it under the radar, on some out of the way topic. For example on the page about you, if you're some minor official in something. Or on the history of some outlying territory, in which nobody's interested except those with a political stake in it.

But this doesnt disprove the concept of "the wisdom of crowds"; it actually confirms it. As in: the more popular a topic (i.e. the bigger the crowd!), the faster such misinformation tends to disappear again. The more obscure the topic (i.e. the smaller the crowd), the longer such misinformation may remain online.

1 Reply

@nimh,

There are always malicious individuals who simply delight in chaos and disruption.

But their effects are only felt when they are a statistically significant number.

0 Replies

@ebrown p,

ebrown wrote:

Even so, I am not skeptical of collaboration... the applications I am skeptical of involve semantic filtering and tagging.

Ok, your definition of what you're criticizing seems to be very fluid. It's like every time someone brings up an application of "crowd wisdom" that works, you shift the criteria for what parts of the concept you're criticising. But once you've narrowed down the parts you don't believe in to only a segment of what people mean when they say "crowd wisdom", that does mean you can no longer say that the whole concept is "a dubious idea"; just that it can't effectively do these one or two specific segments you've singled out.

But on to filtering and tagging, then. Simple example. I use Last.fm. On Last.fm, people listen to music on the site and tag it to indicate what genre etc it is. Last.fm also "scrobbles" the info from all the tracks users play on Media Player or Winamp etc, and anyone can tag those tracks and bands too.

The result is that you go to a band's page and (among many other things) find the most used tags to describe what kind of music it plays. The result is more reliably, widely understandable than any elaborate categorisation by a team of pros would have been. You see music labelled as how most people would label it - simple as that. With the one exception of when more bands share the same name, it's a very effective form of harnessing "crowd wisdom".

2 Replies

@nimh,

It can work in various ways and has various effects, most especially when we are talking opinion-based data.

For example, I listen to Pandora. They use feedback from users to determine what genres music falls in, and then create playlists based on finding stuff that sounds like what you initially inputted in various ways. And it usually works pretty well.

But, in some cases, it works terribly, and songs which just don't match at all or are just plain bad songs get thrown in the mix. It's frustrating. The group has apparently decided that my definition of what sounds 'like' the band I initially inputted is apparently incorrect. This has left me feeling like not using the service at all, even though the group has made a decision which must be pleasing to the majority of them.

In some applications, group thought or crowd wisdom works great; but I believe that the more subjective a question is, the less overall satisfaction will be had by the end user. For example, what if the google image identifier asked you to tell whether or not a picture was 'pretty?' My guess is the answers would wildly gyrate all over the place.

Cycloptichorn

2 Replies

@Cycloptichorn,

Quote:

But, in some cases, it works terribly, and songs which just don't match at all or are just plain bad songs get thrown in the mix. It's frustrating.

Like I said before, "taste" is polarizing.

Think for a second, though, about how new music gets characterized.

1 Reply

@DrewDad,

Quote:

Re: Cycloptichorn (Post 3365348)
Quote:

But, in some cases, it works terribly, and songs which just don't match at all or are just plain bad songs get thrown in the mix. It's frustrating.

Like I said before, "taste" is polarizing.

Think for a second, though, about how new music gets characterized.

As 'personal taste' is the model the voting is built upon here, is it not also an inherently polarizing thing?

Cycloptichorn

2 Replies

@nimh,

Quote:

The result is that you go to a band's page and (among many other things) find the most used tags to describe what kind of music it plays. The result is more reliably, widely understandable than any elaborate categorisation by a team of pros would have been.

Magazines and music shows use "pros" to categorize music all the time, with very good results.

Categorization is a well-known problem in the software engineering world (I am a software engineer working in speech recognition where we use these algorithms). These social networking ideas as a useful categorization scheme are not supported by research. I will dig up research if I have time, but if you want to get an informal feeling of what experts in the field think about social tagging, just search for the word "metacrap".

You are stating, as axioms, things that are not supportable by experience.

2 Replies

@Cycloptichorn,

Quote:

As 'personal taste' is the model the voting is built upon here, is it not also an inherently polarizing thing?

Don't know. What's the difference between the kind of personal taste that makes someone vote a thread up or down, and the personal taste that makes someone rate a product on Amazon, a seller on eBay, or a video on Google Video?

1 Reply

@ebrown p,

Quote:

if you want to get an informal feeling of what experts in the field think about social tagging, just search for the word "metacrap".

Experts in which field? Speech recognition? Software engineering? Social tagging is social tagging. It's not the ultimate, best, or only information retrieval method. It has uses, and can be very useful in more homogenous 'crowds', like say third year sociology students, and fairly delimited data pools, like say research readings.

Loved your search keyword idea - it's like recommending some use the keyword 'pinko' to learn about Democrat policy Wink

0 Replies

@joefromchicago,

joefromchicago wrote:

Thomas's example of language is a good -- and often overlooked -- example of the wisdom of crowds.

Thank you Joe! But in the meantime, I found an even better example: ebrown_p himself! If it wasn't for two wisdom-of-crowds effects called morphogenesis and neo-Darwinian evolution, ebrown_p wouldn't even exist to be skeptical about them.

But this thought is quite a tangent from the original topics of web development and group dynamics so I took the liberty of starting a separate thread to explore it. You can find it at: http://able2know.org/topic/121189-1

0 Replies

@ebrown p,

ebrown wrote:

Magazines and music shows use "pros" to categorize music all the time, with very good results.

Music shows categorize music by genre all the time? Really? I know of, you know, MTV or a radio station having a metal/hardrock/loud music program, or a hip-hop program, but that doesnt approach anything like systematically categorizing each individual band.

Magazines though, do it, yeah, in their reviews section. Shops do it too. And no, in neither case are the results anywhere as reliably + widely recognizable, in my humble experience. Just do an experiment: pick ten different music magazines from different parts of the spectrum, or go to ten different music shops in different parts of town (let alone on a global scale), and check how they categorize the various kinds of electronica/ trip-hop/ breakbeats/ electro/ dub/ turntable/ lounge/ downtempo. Every mag and shop fumbles around to establish its own container labels, they're all different, and individual artists are constantly found in a category labeled X in one mag or shop and Y in the other.

Then last. fm brings in tagging and filtering and presto; no longer does your search on where Mr. Scruff or DJ Vadim might be found this time depend on whatever the resident expert has come up with; it can be labelled by several names at the same time, and they're the ones that most people turn out to associate with the music. As someone who has spent some frustration going through various possible sections of a record store to find which category they'd put them in this time, I can tell you that's an easily more reliable indicator of what you might associate it with than the guesstimate of the owner of London Records or conviction of the music editor of Breakbeat youth. That's a very effective application of crowd wisdom.

You're being way to slick with this. You get a concrete example (actually, you got two and ignored the other one, or at least the response to your off-hand Colbert reference), and brush it off without apparently even considering it, stating some commonplace about how music shows and magazines do this all the time "with very good results", thank you very much -- without apparently more basis than the eagerness to brush off the specific example and move back to generalised assertions and an appeal to authority.

I dunno. Maybe it's too soon to say. But in my impression your posts have the hallmarks of someone seeking to prove a point rather than genuinely weigh the evidence. Way it's going now, you'll likely keep cherrypicking your way through the examples and moving the goalposts where needed. Which is a pity, really, cause judging on the posts you've gotten so far you have some real expertise joining into this discussion.

1 Reply

@Cycloptichorn,

Sorry Cyclo, had overlooked your post.

Cyclo wrote:

The group has apparently decided that my definition of what sounds 'like' the band I initially inputted is apparently incorrect. This has left me feeling like not using the service at all, even though the group has made a decision which must be pleasing to the majority of them.

Right - totally. (Clicking on related music in Last.fm is a mixed bag - it works better the better known an artist is, eg the more people have had their user patterns plugged in. For less commonplace artists or genres results can be fairly disappointing. But that kind of speaks for the effectivity of crowd wisdom rather than against it.)

Anyway, I said right, totally, because isnt that what the point is? I havent used Pandora myself, but judging on your description, the collective user input has yielded a result which is most pleasing or relevant to the most users in the group. Which is tough for you I admit, but also the only metric of success I can think of when the question is categorisation of consumer products (in this case, music). A group of experts might have come up with a different categorisation which, who knows, might win more plaudits from the real connaisseurs - but which would have left the result less instinctively usable to most users, then, than the current one. My take, anyway.

0 Replies

@nimh,

Quote:

Then last. fm brings in tagging and filtering and presto; no longer does your search on where Mr. Scruff or DJ Vadim might be found this time depend on whatever the resident expert has come up with; it can be labelled by several names at the same time, and they're the ones that most people turn out to associate with the music. As someone who has spent some frustration going through various possible sections of a record store to find which category they'd put them in this time, I can tell you that's an easily more reliable indicator of what you might associate it with than the opinion of the owner of London Records or the music editor of Breakbeat youth. That's a very effective application of crowd wisdom.

Yup... But it's not only crowd wisdom applied in the tagging system...

It's also that people (including software architects) have come to think in a container structure. Which is really weird, considering that we're talking about information.

But apparently, somebody thought it would be easier for people to grasp the idea that, say, a Word document should be treated like a physical object. Which means, you'll put it into a folder, which you put e.g. on your desktop (see the physical-world-terminology?), and that's where you'll find it again later.

That makes a lot of sense, considering that people are used to this idea of handling stuff. You've got your CDs at home, you want to sort them... well, you'll put them into a specific place, maybe using a specific system. Sorting them alphabetically, or whatever.

---

Now, the metacrap dude (I googled him, found really only one reference.... is it really that Cory Doctorow - the science fiction author? Yes it is.) says that, look, having a global system of organizing things by meta information will never work. Mainly because you put this thing in this place, I'll put it in that place. You'll name this bit a widget, I'll name it a what's-it. That where the mess starts, he says. To organize information, you'd have to get everybody, globally, to use the same system of organizing information. To put the same things into the same containers.

And then he goes off on a rant, finding all the various reasons why that will never happen.

Thing is: I agree.

---

But we're not talking about physical objects. We're talking about information. Unlike the physical object (e.g. a CD, a letter, ...), the information doesn't have to be put into one place.

Filing a specific news article under "Politics" makes just as much sense as filing it under "International News". Both things might apply at the same time. And as we're not talking about physical information here, we don't have to decide to put it either in one or in the other category.

How do you do that? You come up with a new structure. The old model of folders and subfolders doesn't make sense any more.

And that's where labels or tags come in. As in nimh's example above, you can attach several tags to one bit of information. And you can find the information in several "places".

And all of a sudden, it doesn't matter that users attach different labels to the same bit of information. That people lie. Or that people are lazy, stupid, or whatever.

0 Replies

@old europe,

Quote:

Quote:

As 'personal taste' is the model the voting is built upon here, is it not also an inherently polarizing thing?

Don't know. What's the difference between the kind of personal taste that makes someone vote a thread up or down, and the personal taste that makes someone rate a product on Amazon, a seller on eBay, or a video on Google Video?

Isn't obvious, OE? On none of these sites are you using the rating system to change the position of the product out of disinterest, but rather to give others a value judgment about the product itself. Which, if you think about it, has been my point all along: ratings are not typically used as a content management system, but one of relative judgment.

When one sees a thread with -15 being the rating, it is not an attractive advertisement for that thread!

Cycloptichorn

1 Reply

We are talking about Information retrieval let's discuss how to evaluate whether a system of tagging voting is good or not.

There are two main ways to evaluate a set of results; "Precision" is the percentage of results you receive that are relevant (the more irrelevant results, the lower your score). "Recall" is the percentage of possible relevant results that you receive (the more good results that aren't returned, the lower your score).

Crowd generated tags will give you a decent precision (you will get lots of good results). My concern is with recall... interesting posts that aren't tagged, or aren't tagged correctly get left out.

The problem this will cause is homogeneity. The results you get will all be largely the same. Interesting results (with are relevant but not returned by the search) are going to be left out. Worse yet... with filtering in the mix they will be actively excluded.

Nihm (in his somewhat patronizing manner) is stating, without proof, that voting with a large crowd produces results that are "better" .

There are theoretical reasons that these schemes are not good for classifying data (which are backed by research).

This is also based on experience-- Nihm and crew are pshawing the current phenomenon in specific areas here (I will point out again how this is playing in the science thread) ... maybe it is too soon. I guess we will see how this plays out.

1 Reply

@Cycloptichorn,

Quote:

Isn't obvious, OE? On none of these sites are you using the rating system to change the position of the product out of disinterest, but rather to give others a value judgment about the product itself.

How do you know what their motivation really is? People may rank and vote for stuff for any kind of reason; and I doubt that all of them have the Common Welfare in mind when they rank down a product, or a video, or a comment about a video.

In fact, if you head over to youtube right now, go and look at the comments of any given video, and how the thumbs up and thumbs down icons work - it's exactly the same system as with the threads here. I agree that the rating for the videos works in a different matter, but the point is: I don't think the primary motivation of most users there is to give helpful advice to other users.

That might be a bit different on Amazon or eBay, granted. But the point is: it doesn't matter why people rank stuff up or down. It's the vast number of people using the features that make those features valuable to others.

(And hey, that doesn't mean I'm dismissing your other concerns. I'm just saying that, while questioning the motivation behind the voting might be justified, it can still render meaningful results.)(I guess I'm repeating myself by now.)

1 Reply

@Cycloptichorn,

Quote:

As 'personal taste' is the model the voting is built upon here, is it not also an inherently polarizing thing?

I don't have a problem with polarization on an Internet message board.

<shrugs>

0 Replies

@old europe,

Quote:

Re: Cycloptichorn (Post 3365484)
Quote:

Isn't obvious, OE? On none of these sites are you using the rating system to change the position of the product out of disinterest, but rather to give others a value judgment about the product itself.

How do you know what their motivation really is? People may rank and vote for stuff for any kind of reason; and I doubt that all of them have the Common Welfare in mind when they rank down a product, or a video, or a comment about a video.

In fact, if you head over to youtube right now, go and look at the comments of any given video, and how the thumbs up and thumbs down icons work - it's exactly the same system as with the threads here. I agree that the rating for the videos works in a different matter, but the point is: I don't think the primary motivation of most users there is to give helpful advice to other users.

That might be a bit different on Amazon or eBay, granted. But the point is: it doesn't matter why people rank stuff up or down. It's the vast number of people using the features that make those features valuable to others.

(And hey, that doesn't mean I'm dismissing your other concerns. I'm just saying that, while questioning the motivation behind the voting might be justified, it can still render meaningful results.)(I guess I'm repeating myself by now.)

Youtube is a better example of what you meant, but still not analogous to our current situation. For rating a video on youtube doesn't change the users ability to see the product, which is the video itself. On A2K, the product is the thread and the discussion within; changing ratings merely to move threads around isn't the same thing as they are using their rating system for at all! Instead, it's value judgments about the content, not disinterest in people's comments, which prompt a thumbs up or thumbs down. Rating comments over there doesn't move the comments at all. So I think you are incorrect; between Ebay, Amazon, and other sites which use ranking systems, they are almost universally value judgments, and not something you bother with if you are simply disinterested, for voting up or down doesn't really change the ability of the user to find the product itself.

Cycloptichorn

1 Reply

@Cycloptichorn,

Quote:

Youtube is a better example of what you meant, but still not analogous to our current situation. For rating a video on youtube doesn't change the users ability to see the product, which is the video itself. On A2K, the product is the thread and the discussion within; changing ratings merely to move threads around isn't the same thing as they are using their rating system for at all! Instead, it's value judgments about the content, not disinterest in people's comments, which prompt a thumbs up or thumbs down. Rating comments over there doesn't move the comments at all. So I think you are incorrect; between Ebay, Amazon, and other sites which use ranking systems, they are almost universally value judgments, and not something you bother with if you are simply disinterested, for voting up or down doesn't really change the ability of the user to find the product itself.

Good lord. Is rating a Youtube video not a value judgment? Don't the highest rated videos go to the top? Don't you have to work harder to see the lowest-rated videos?

Amazon users rate the reviews, and the best reviews are highlighted. Amazon users rate the ratings.

You're making less sense as this "discussion" progresses.

1 Reply

I might add that A2K is itself a form of crowd sourcing.

People ask questions, and self-appointed "experts" provide information. The users generally monitor the information provided and correct any misinformation.

Even deliberate misinformation (Gungasnake comes to mind) is refuted, time and again.

Give Craven some credit for maybe knowing what he's doing.

0 Replies

There is no Wisdom in Crowds

Related Topics

Quick Links

My Account

able2know