The neurologist and epidemiologist Cathie Sudlow has written a highly readable and important piece in the BMJ exposing Science magazine’s poor reporting of a paper on chronic fatigue syndrome, (1) but she reaches the wrong conclusions on how scientific publishing should change.
For those of you who have missed the story, Science published a case control study in September that showed a strong link between chronic fatigue syndrome and xenotropic murine leukaemia virus-related virus (XMRV). (2) The study got wide publicity and was very encouraging to the many people who believe passionately that chronic fatigue syndrome has an infectious cause. Unfortunately, as Sudlow describes, the study lacked basic information on the selection of cases and controls, and, worse, Science has failed to publish E-letters from Sudlow and others asking for more information.
In the meantime, three other studies have not found an association between chronic fatigue syndrome and XMRV. (3-5)
To avoid such poor reporting in the future Sudlow urges strengthening the status quo—more and better prepublication peer review. Not only is she trying to close the stable door after the horse has bolted she has also failed to recognise the possibilities of the new Web 2.0 world. The time has come to move from a world of “filter then publish” to one of “publish then filter”—and it’s happening.
Prepublication peer review is faith based not evidence based, and Sudlow’s story shows how it failed badly at Science. Her anecdote joins a mountain of evidence of the failures of peer review: it is slow, expensive, largely a lottery, poor at detecting errors and fraud, anti-innovatory, biased, and prone to abuse. (6 7) As two Cochrane reviews have shown, the upside is hard to demonstrate. (8 9) Yet people like Sudlow who are devotees of evidence persist in belief in peer review. Why?
The world also seems unaware that it is scientifically dangerous to read only the “top journals”. As Neal Young and others have argued, the “top journals” publish the sexy stuff. (10) The unglamorous is published elsewhere or not at all, and yet the evidence comprises both the glamorous and the unglamorous.
The naïve concept that the “top journals” publish the important stuff and the lesser journals the unimportant is simply false. People who do systematic reviews know this well. Anybody reading only the “top journals” receives a distorted view of the world—as this Science story illustrates. Unfortunately many people, including most journalists, do pay most attention to the “top journals.”
So rather than bolster traditional peer review at “top journals,” we should abandon prepublication review and paying excessive attention to “top journals.” Instead, let people publish and let the world decide. This is ultimately what happens anyway in that what is published is digested with some of it absorbed into “what we know” and much of it never being cited and simply disappearing.
Such a process would have worked better with the story that Sudlow tells. The initial study would have appeared–perhaps to a fanfare of publicity (as happened) or perhaps not. Critics would have immediately asked the questions that Sudlow asks. Instead of hiding behind Science’s skirts as has happened, the authors would have been obliged to provide answers. If they couldn’t, then the wise would disregard their work. Then follow up studies could be published rapidly.
Unfortunately, unlike physicists, astronomers, and mathematicians, all of whom have long published in this way, biomedical researchers seem reluctant to publish without traditional prepublication peer review. In reality this is probably because of innate conservatism and the grip of the “top journals” who insist on prepublication review, but biomedical researchers often say “But our stuff is different from that of physicists in that it may scare ordinary people. A false story, for example, “Porridge causes cancer” can create havoc.”
My answer to this objection is that this happens now. Much of what is published in journals is scientifically poor—as the Science article shows. Then, many studies are presented at scientific meetings without peer review, and scientists and their employers are increasingly likely to report their results through the mass media.
In a world of “publish then filter” we would at least have the full paper to dissect, whereas reports in the media even if derived from scientific meetings, include insufficient information for critical appraisal.
So I urge Sudlow, a thinking woman, to reflect further and begin to argue for something radical and new rather than more of the same.
1. Sudlow C. Science, chronic fatigue syndrome, and me. BMJ 2010;340:c1260
6. Godlee F, Jefferson T. Peer Review in Health Sciences. 2nd ed. London: BMJ Books; 2003.
Competing interest: RS is on the board of the Public Library of Science and an enthusiast for open access publishing, but he isn’t paid and doesn’t benefit financially from open access publishing.
I got this from Ves Dimov's tweet this morning and read the blog entry. This has been debated for a long time in different circles about doing better than relying on pre publication peer review as a means of quality control. Ron LaPorte et al wrote in "Scientific Journals are ‘faith based’: is there science behind Peer review?" (see here http://goo.gl/lHULk) that this was due to (I quote verbatim) because of the "almost non-existent use of the scientific method to question and test the publication process itself". This brings back the old debate of using scientific methods to question the process of science publication itself. A good post and worth reading and discussions.
