Re Nature's December comparison of Wikipedia and Britannica

Nature mag cooked Wikipedia study

By Andrew Orlowski in San Francisco
Published Thursday 23rd March 2006 03:33 GMT
Nature magazine has some tough questions to answer after it let its Wikipedia fetish get the better of its responsibilities to reporting science. The Encyclopedia Britannica has published a devastating response to Nature's December comparison of Wikipedia and Britannica, and accuses the journal of misrepresenting its own evidence.

Where the evidence didn't fit, says Britannica, Nature's news team just made it up. Britannica has called on the journal to repudiate the report, which was put together by its news team.

Independent experts were sent 50 unattributed articles from both Wikipedia and Britannica, and the journal claimed that Britannica turned up 123 "errors" to Wikipedia's 162.

But Nature sent only misleading fragments of some Britannica articles to the reviewers, sent extracts of the children's version and Britannica's "book of the year" to others, and in one case, simply stitched together bits from different articles and inserted its own material, passing it off as a single Britannica entry.

Nice "Mash-Up" - but bad science.

"Almost everything about the journal's investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading," says Britannica.

"Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopedia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit."

In one case, for example. Nature's peer reviewer was sent only the 350 word introduction to a 6,000 word Britannica article on lipids - which was criticized for containing omissions.

A pattern also emerges which raises questions about the choice of the domain experts picked by Nature's journalists.

Several got their facts wrong, and in many other cases, simply offered differences of opinion.

"Dozens of the so-called inaccuracies they attributed to us were nothing of the kind; they were the result of reviewers expressing opinions that differed from ours about what should be included in an encyclopedia article. In these cases Britannica's coverage was actually sound."

Nature only published a summary of the errors its experts found some time after the initial story, and has yet to disclose all the reviewer's notes.

So how could a respected science publication make such a grave series of errors?


--------------------------------------------------------------------------------

Now is the time for all good men to come to the aid of the Wiki
When Nature published the news story in December, it followed weeks of bad publicity for Wikipedia, and was a gift for the project's beleaguered supporters.

In October, a co-founder had agreed that several entries were "horrific crap". A former newspaper editor and Kennedy aide John Siegenthaler Snr then wrote an article explaining how libellous modifications had lain unchecked for months. By early December, Wikipedia's Jimmy Wales was becoming a regular feature on CNN cable news, explaining away the site's deficiencies.

"Nature's investigation suggests that Britannica's advantage may not be great," wrote (http://www.nature.com/nature/journal/v438/n7070/full/438900a.html) news editor Jim Giles.

Nature accompanied this favorable news report with a cheerful, spin-heavy editorial (http://www.nature.com/nature/journal/v438/n7070/full/438890a.html) that owed more to an evangelical recruitment drive than it did a rational analysis of empirical evidence. It urged readers to "push forward the grand experiment that is Wikipedia."

(Former Britannica editor Robert McHenry dubbed Wikipedia the "Faith based encyclopedia", and the project certainly reflects the religious zeal of some of its keenest supporters. Regular Register readers will be familiar with the rhetoric. See "Wikipedia 'to make universities obsolete' (http://www.theregister.com/2004/09/07/khmer_rouge_in_daipers/)).

Hundreds of publications pounced on the Nature story, and echoed the spin that Wikipedia was as good as Britannica - downplaying or omitting to mention the quality gap (http://www.theregister.co.uk/2005/12/16/wikipedia_britannica_science_comparison/). The press loves an upbeat story, and what can be more uplifting than the utopian idea that we're all experts - at whatever subject we choose?

The journal didn't, however, disclose the evidence for these conclusions until some days later, when journalists had retired for their annual Christmas holiday break.

And this evidence raised troubling questions, as Nicholas Carr noted last month. Many publications had assumed Nature's Wikipedia story was objectively reporting the work of scientists - Nature's staple - rather than a news report assembled by journalists pretending to be scientists.

And now we know it was anything but scientific.

Carr noted (http://www.roughtype.com/archives/2006/02/community_and_h.php) that Nature's reviewers considered trivial errors and serious mistakes as roughly equal.

So why did Nature risk its reputation in such a way?

Perhaps the clue lies not in the news report, but in the evangelism of the accompanying editorial. Nature's news and features editor Jim Giles, who was responsible for the Wikipedia story, has a fondness for "collective intelligence", one critical website suggets. (http://www.aetherometry.com/antiwikipedia2/awp2_index.html)

"As long as enough scientists with relevant knowledge played the market, the price should reflect the latest developments in climate research," Giles concluded (http://www.aetherometry.com/antiwikipedia2/Appendix_3.html) of one market experiment in 2002.

The idea became notorious two years ago when DARPA, under retired Admiral Poindexter, invested in an online "terror casino (http://www.theregister.co.uk/2003/08/05/meet_the_transhumanists_behind/)" to predict world events such as assassinations. The public didn't quite share the sunny view of this utopian experiment, and Poindexter was invited to resign.

What do these seemingly disparate projects have in common? The idea that you can vote for the truth.

We thought it pretty odd, back in December, to discover a popular science journal recommending readers support less accurate information. It's even stranger to find this institution apparently violating fundamental principles of empiricism.

But these are strange times - and high summer for supporters of junk science. ®