Re Nature's December comparison of Wikipedia and Britannica

March 22, 2006

Dear Educator:

Because you're a valued Britannica customer, I'm writing to you today about a subject that has received widespread news coverage — it is a subject that's being taken very seriously by all of us at Encyclopædia Britannica and one on which we have worked extensively with our editors, contributors, and advisors for many weeks.

In one of its recent issues, the science journal Nature published an article that claimed to compare the accuracy of the online Encyclopædia Britannica with Wikipedia, the Internet database that allows anyone, regardless of knowledge or qualifications, to write and edit articles on any subject. Wikipedia had recently received attention for its alleged inaccuracies, but Nature's article claimed that Britannica's science coverage was only slightly more accurate than Wikipedia's.

Arriving amid the revelations of vandalism and errors in Wikipedia, such a finding was, not surprisingly, big news. Perhaps you even saw the story yourself. It's been reported around the world.

Those reports were wrong, however, because Nature's research was invalid. As our editors and scholarly advisers have discovered by reviewing the research in depth, almost everything about the Nature's investigation was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopædia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit.

Since educators and librarians have been among Britannica's closest colleagues for many years, I would like to address you personally with an explanation of our findings and tell you the truth about the Nature study.

Almost everything Nature did showed carelessness and indifference to basic research standards. Their numerous errors and spurious procedures included the following:

Rearranging, reediting, and excerpting Britannica articles. Several of the "articles" Nature sent its outside reviewers were only sections of, or excerpts from Britannica entries. Some were cut and pasted together from more than one Britannica article. As a result, Britannica's coverage of certain subjects was represented in the study by texts that our editors never created, approved or even saw.

Mistakenly identifying inaccuracies. The journal claimed to have found dozens of inaccuracies in Britannica that didn't exist.

Reviewing the wrong texts. They reviewed a number of texts that were not even in the encyclopedia.

Failing to check facts. Nature falsely attributed inaccuracies to Britannica based on statements from its reviewers that were themselves inaccurate and which Nature's editors failed to verify.

Misrepresenting its findings. Even according to Nature's own figures, (which grossly exaggerated the number of inaccuracies in Britannica) Wikipedia had a third more inaccuracies than Britannica. Yet the headline of the journal's report concealed this fact and implied something very different.

Britannica also made repeated attempts to obtain from Nature the original data on which the study's conclusions were based. We invited Nature's editors and management to meet with us to discuss our analysis, but they declined.

The Nature study was thoroughly wrong and represented an unfair affront to Britannica's reputation.

Britannica practices the kind of sound scholarship and rigorous editorial work that few organizations even attempt. This is vital in the age of the Internet, when there is so much inappropriate material available. Today, having sources like Britannica is more important than ever, with content that is reliable, tailored to the age of the user, correlated to curriculum, and safe for everyone.

Whatever may have prompted Nature to do such careless and sloppy research, it's now time for them to uphold their commitment to good science and retract the study immediately. We have urged them strongly to do so.

We have prepared a detailed report that describes Britannica's thorough (7,000 words) analysis of the Nature study. I invite you to download it from our Web site at www.eb.com.

Best regards,
Patricia A. Ginnis
Senior Vice President
Encyclopædia Britannica, Inc.

Nature mag cooked Wikipedia study

By Andrew Orlowski in San Francisco
Published Thursday 23rd March 2006 03:33 GMT
Nature magazine has some tough questions to answer after it let its Wikipedia fetish get the better of its responsibilities to reporting science. The Encyclopedia Britannica has published a devastating response to Nature's December comparison of Wikipedia and Britannica, and accuses the journal of misrepresenting its own evidence.

Where the evidence didn't fit, says Britannica, Nature's news team just made it up. Britannica has called on the journal to repudiate the report, which was put together by its news team.

Independent experts were sent 50 unattributed articles from both Wikipedia and Britannica, and the journal claimed that Britannica turned up 123 "errors" to Wikipedia's 162.

But Nature sent only misleading fragments of some Britannica articles to the reviewers, sent extracts of the children's version and Britannica's "book of the year" to others, and in one case, simply stitched together bits from different articles and inserted its own material, passing it off as a single Britannica entry.

Nice "Mash-Up" - but bad science.

"Almost everything about the journal's investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading," says Britannica.

"Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopedia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit."

In one case, for example. Nature's peer reviewer was sent only the 350 word introduction to a 6,000 word Britannica article on lipids - which was criticized for containing omissions.

A pattern also emerges which raises questions about the choice of the domain experts picked by Nature's journalists.

Several got their facts wrong, and in many other cases, simply offered differences of opinion.

"Dozens of the so-called inaccuracies they attributed to us were nothing of the kind; they were the result of reviewers expressing opinions that differed from ours about what should be included in an encyclopedia article. In these cases Britannica's coverage was actually sound."

Nature only published a summary of the errors its experts found some time after the initial story, and has yet to disclose all the reviewer's notes.

So how could a respected science publication make such a grave series of errors?


--------------------------------------------------------------------------------

Now is the time for all good men to come to the aid of the Wiki
When Nature published the news story in December, it followed weeks of bad publicity for Wikipedia, and was a gift for the project's beleaguered supporters.

In October, a co-founder had agreed that several entries were "horrific crap". A former newspaper editor and Kennedy aide John Siegenthaler Snr then wrote an article explaining how libellous modifications had lain unchecked for months. By early December, Wikipedia's Jimmy Wales was becoming a regular feature on CNN cable news, explaining away the site's deficiencies.

"Nature's investigation suggests that Britannica's advantage may not be great," wrote (http://www.nature.com/nature/journal/v438/n7070/full/438900a.html) news editor Jim Giles.

Nature accompanied this favorable news report with a cheerful, spin-heavy editorial (http://www.nature.com/nature/journal/v438/n7070/full/438890a.html) that owed more to an evangelical recruitment drive than it did a rational analysis of empirical evidence. It urged readers to "push forward the grand experiment that is Wikipedia."

(Former Britannica editor Robert McHenry dubbed Wikipedia the "Faith based encyclopedia", and the project certainly reflects the religious zeal of some of its keenest supporters. Regular Register readers will be familiar with the rhetoric. See "Wikipedia 'to make universities obsolete' (http://www.theregister.com/2004/09/07/khmer_rouge_in_daipers/)).

Hundreds of publications pounced on the Nature story, and echoed the spin that Wikipedia was as good as Britannica - downplaying or omitting to mention the quality gap (http://www.theregister.co.uk/2005/12/16/wikipedia_britannica_science_comparison/). The press loves an upbeat story, and what can be more uplifting than the utopian idea that we're all experts - at whatever subject we choose?

The journal didn't, however, disclose the evidence for these conclusions until some days later, when journalists had retired for their annual Christmas holiday break.

And this evidence raised troubling questions, as Nicholas Carr noted last month. Many publications had assumed Nature's Wikipedia story was objectively reporting the work of scientists - Nature's staple - rather than a news report assembled by journalists pretending to be scientists.

And now we know it was anything but scientific.

Carr noted (http://www.roughtype.com/archives/2006/02/community_and_h.php) that Nature's reviewers considered trivial errors and serious mistakes as roughly equal.

So why did Nature risk its reputation in such a way?

Perhaps the clue lies not in the news report, but in the evangelism of the accompanying editorial. Nature's news and features editor Jim Giles, who was responsible for the Wikipedia story, has a fondness for "collective intelligence", one critical website suggets. (http://www.aetherometry.com/antiwikipedia2/awp2_index.html)

"As long as enough scientists with relevant knowledge played the market, the price should reflect the latest developments in climate research," Giles concluded (http://www.aetherometry.com/antiwikipedia2/Appendix_3.html) of one market experiment in 2002.

The idea became notorious two years ago when DARPA, under retired Admiral Poindexter, invested in an online "terror casino (http://www.theregister.co.uk/2003/08/05/meet_the_transhumanists_behind/)" to predict world events such as assassinations. The public didn't quite share the sunny view of this utopian experiment, and Poindexter was invited to resign.

What do these seemingly disparate projects have in common? The idea that you can vote for the truth.

We thought it pretty odd, back in December, to discover a popular science journal recommending readers support less accurate information. It's even stranger to find this institution apparently violating fundamental principles of empiricism.

But these are strange times - and high summer for supporters of junk science. ®