24th November 2022
Peer-review: Time to get rid of it
‘There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.’ Drummond Rennie.
Somewhat damning?
It supports my considered opinion that medical research died decades ago. It is now populated by the undead to become, what could best be called, ‘Zombie science’. Or, possibly, the walking dead.
I would not be the first to think this. In truth, I nicked the term. Here is the abstract of a paper by Bruce Charlton in the Journal ‘Medical Hypotheses.’ It was written in 2008:
‘Zombie science: a sinister consequence of evaluating scientific theories purely on the basis of enlightened self-interest.’
‘Although the classical ideal is that scientific theories are evaluated by a careful teasing-out of their internal logic and external implications, and checking whether these deductions and predictions are in-line-with old and new observations; the fact that so many vague, dumb or incoherent scientific theories are apparently believed by so many scientists for so many years is suggestive that this ideal does not necessarily reflect real world practice.
In the real world it looks more like most scientists are quite willing to pursue wrong ideas for so long as they are rewarded with a better chance of achieving more grants, publications and status. The classic account has it that bogus theories should readily be demolished by sceptical (or jealous) competitor scientists.
However, in practice even the most conclusive ‘hatchet jobs’ may fail to kill, or even weaken, phoney hypotheses when they are backed-up with sufficient economic muscle in the form of lavish and sustained funding. And when a branch of science based on phoney theories serves a useful but non-scientific purpose, it may be kept-going indefinitely by continuous transfusions of cash from those whose interests it serves.
If this happens, real science expires and a ‘zombie science’ evolves. Zombie science is science that is dead but will not lie down. It keeps twitching and lumbering around so that (from a distance, and with your eyes half-closed) zombie science looks much like the real thing.
But in fact the zombie has no life of its own; it is animated and moved only by the incessant pumping of funds. If zombie science is not scientifically-useable–what is its function? In a nutshell, zombie science is supported because it is useful propaganda to be deployed in arenas such as political rhetoric, public administration, management, public relations, marketing and the mass media generally. It persuades, it constructs taboos, it buttresses some kind of rhetorical attempt to shape mass opinion.
Indeed, zombie science often comes across in the mass media as being more plausible than real science; and it is precisely the superficial face-plausibility which is the sole and sufficient purpose of zombie science.’ 1
Unfortunately, I can only provide you with a reference to the abstract. Because, in what I consider a majestic, universe spanning irony, the full article sits behind a paywall. Nowadays most medical papers are kept safe from the public, or the amateur researchers, or anyone else who is not a millionaire. They can only be viewed by those who have access via their university – usually. I call it ‘censorship by inability to pay.’
You cannot even read medical research that will have been funded by your taxes, or someone else’s taxes in another country. Instead, it sits in a virtual room, secured behind the locked-doors of ‘pay per view.’ Which represents another twitching limb of zombie science. It senses money and reaches out blindly to grab it, with dead, bony fingers. ‘My precious.’
Going back a couple of steps. Who is this Bruce Charlton of whom you speak? Well, he used to edit the journal Medical Hypotheses. But he made the error of publishing an article highly critical of the mainstream narrative on HIV. The article in question contained this statement. ‘There is as yet no proof that HIV causes AIDS.’ Inevitably, a major outcry took place. Glasses of Dom Perignon slipped from chubby, quivering fingers. Foie gras was left uneaten, that and the guinea fowl.
Many will strongly believe, that this statement, and the entire article, must be wrong, and should never have been published. But I would contend that this is absolutely not the point. The point is that anyone who believes articles should not be published because they are ‘clearly wrong’ needs to be gently led away from the world of science. Then booted out of the door and told, in no uncertain terms, to get out and stay out. Until they learn the error of their ways.
‘In science, the primary duty of ideas is to be useful and interesting even more than to be true.’ Wilfred Trotter.
What happened next was depressingly predictable. Elsevier, the publishers of Medical Hypotheses, did exactly what you would expect of the walking dead. They did not defend the right of the editor – of a journal titled ‘Medical Hypotheses’ – to publish contentious articles. They panicked, then piled the blame on Bruce Charlton.
After receiving a raft of complaints, Elsevier had the article peer reviewed under the oversight of editors from The Lancet. Following the peer review, the article, and another by Marco Ruggiero of the University of Florence in Italy, was withdrawn and a reform of the journal was mooted.
“They were withdrawn because of concerns expressed by the scientific community about the quality of the articles, and our concern that the papers could potentially be damaging to global public health,” the publisher said in a statement.’ 2
My favourite comment is below:
‘This journal has published ‘hypotheses’ that are regrettable… “I do not think that the medical community will lose anything if the journal does not continue in its current form.’
And if you want to find a more Stalinist, Big Brother(ist), and frankly sinister comment than the final one, you will need to travel far. ‘Regrettable’ … a word most commonly used by the evil baddie in a James Bond movie. Just before feeding his underling to the sharks waiting below.
Evil bad guy: ‘Your actions, I am afraid, are regrettable.’ Presses button.
Underling: ‘Aaaarrrgggh….’ Chomp, thrashing, blood.
And lo it came to pass that Bruce Charlton was fired. Then, in an even more majestic, metaverse spanning irony, Elsevier decreed that the journal Medical Hypotheses must become peer-reviewed. Bruce Charlton had vehemently disagreed to this – another reason why he was fired.
Yes, a journal dedicated to publishing new scientific thinking was to be peer-reviewed. But who could they choose to carry out such a task? All those ‘peers’ who just happened to have previously published the exact same new hypotheses – never published before. A clever trick you may think.
Of course, they do not mean that. What they mean is that established figures within the field should be chosen to do the hatchet job … sorry, peer-review. The very people who would suffer the greatest reputational (and financial) damage, if their established views were to be successfully overturned. Now let me think about the likely outcome of any such review … for approximately one picosecond.
The simple fact is that peer-review has become a slaughtering field for new ideas, and new hypotheses. It is the perfect place to send a timid new-born hypothesis blinking into the sunlight. I visualise a David Attenborough documentary. The bit where a baby wildebeest plops to the ground, under the baleful watching gaze of a pack of hyenas. You know what happens next. It ain’t pretty.
Do you think my view of peer-review is a bit over the top, a wild conspiracy theory of some kind? Well, here is what Richard Horton, long-time editor of the Lancet, has to say of peer-review.
‘The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.’
Or this quote from Richard Smith, discussing Drummond Rennie:
‘If peer review was a drug it would never be allowed onto the market,’ says Drummond Rennie, deputy editor of the Journal Of the American Medical Association and intellectual father of the international congresses of peer review that have been held every four years since 1989. Peer review would not get onto the market because we have no convincing evidence of its benefits but a lot of evidence of its flaws.’ 3
Listen guys, sorry to disillusion you, but peer-review was never meant to push forward the boundaries of scientific research. It was primarily designed to keep the top guys at the top, and squash anyone with dissenting views. You think not? You think it has been proven to be effective?
‘Multiple studies have shown how if several authors are asked to review a paper, their agreement on whether it should be published is little higher than would be expected by chance. A study in Brain evaluated reviews sent to two neuroscience journals and to two neuroscience meetings. The journals each used two reviewers, but one of the meetings used 16 reviewers while the other used 14. With one of the journals the agreement among the journals was no better than chance while with the other it was slightly higher. For the meetings the variance in the decision to publish was 80 to 90% accounted for by the difference in opinions of the reviewers and only 10 to 20% by the content of the abstract submitted.’4
And yes, since you ask, I have been asked to peer-review papers. I sent one off recently. Hypocrite? Well, hypocrisy makes the world go around. In my defence I believe it’s a good idea for me to recommend that a ‘contentious’ paper on LDL gets published. Otherwise, my sworn enemies get to clamp it within their pitiless jaws and crush it to death. Why do you suppose I get sent papers from time to time? Because the editor knows exactly what I think, and wants the paper published. Hypocrisy! Why, yes.
In reality, peer-review is about as much use as a chocolate teapot. All journal editors know it’s bollocks, most reviewers know it’s bollocks. But it suits everyone to pretend that the ‘all hallowed’ peer-review cleaves the sword of truth in a mighty fist, protecting us all from bad science.
Does it? Just to give you one recent example where you can replace the words ‘peer-review’, with the words ‘chocolate teapot’ I refer you back to the world of COVID19. Where one, now infamous paper, passed straight through the editorial team, peer-review, and every other check and balance, to find itself published in the Lancet, no less. Even though it rested on completely made-up data:
‘The Lancet will alter its peer review process following the retraction of a paper that cited suspect data linking the controversial drug hydroxychloroquine to increased COVID-19 deaths.
In the future, both peer reviewers and authors will need to provide statements giving assurances on the integrity of data and methods in the paper, the journal’s editor Richard Horton told POLITICO.
“We’re going to ask our reviewers more directly, whether they think there are any issues of research integrity in the paper,” he said. This stipulation will apply to every paper submitted to the journal.
“If the answer to that question is yes, that’s the moment where we trigger some kind of data review,” he added.
These changes to the eminent U.K. journal’s peer review policies are a direct result of a paper that used data from the U.S.-based firm Surgisphere, purporting to be from around 700 hospitals in six continents. But as questions emerged over the study, Surgisphere refused to allow a review of its dataset.
It wasn’t just the Lancet paper that had used data from Surgisphere. The New England Journal of Medicine had used the data for a paper.
The paper was retracted at the request of three of its four authors. They claimed that they couldn’t see the raw data because the fourth author — Sapan Desai, the CEO of Surgisphere — refused to hand it over. But the fact that the co-authors hadn’t seen the raw data pre-publication also raised questions for many readers.’ 5
Yes, indeed, the great and mighty Lancet published a paper based on completely fabricated research. Do you think Horton’s sticking plaster solution is going to have the slightest effect? “We’re going to ask our reviewers more directly, whether they think there are any issues of research integrity in the paper.
Yup, that’ll sort everything out, no doubt about it. No … doubt … about … it. Ask a few peer-reviewers to accuse their peers of potential research fraud. I can see no problem with doing that, at all. I can just imagine the frosty silence that will ensue the next time the author and peer-reviewer meet up.
Peer-reviewer: ‘You’re a liar.’
Researcher: ‘No, you’re a bloody liar.’
Hands up those who think that Richard Horton was simply attempting to deflect criticism away from himself, towards the peer-reviewers. ‘It’s not my fault, it was the peer-reviewers. They made me do it.’ Boo hoo. Poor little you.
Some may believe (as would I dear reader) that this utterly fraudulent load of crap sailed through editorial control, and the peer-review process, because it was attacking the use of hydroxychloroquine in COVID19. Claiming that it killed people. Of course, this was very much the party line at the time. Still is. [Not getting into that debate here].
However, I know, and you know, and everybody knows – although those at the top of this particular game would deny it vehemently – that if the authors had claimed the opposite well then. Well then… well then, their research paper would have been scrutinised to within an inch of its life, then rejected. On whatever grounds could possibly be found. A semi;colon in the wrong place. ‘Off with their heads.’
Peer-review. Yes, peer-review… a crude means of discovering the acceptability — not the validity — of a new finding.
Max Plank was the man who published Albert Einstein’s special theory of relativity. Much against forceful dismissals by his peers it must be said. Einstein’s theory was, at the time, very much unacceptable to most physicists. Plank held out against them, which was perhaps to be expected. He was a bit of a free-thinker. As he once famously said:
‘A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.’
Science can never be about acceptability – which is, too often, the purpose of peer-review. It is about the truth. Or reality, or whatever term is the best description to use. Science is about rocking the boat, and upsetting the established views, and informing ‘experts’ that they are talking rubbish.
As Richard Feynman said. ‘Science is the belief in the ignorance of experts.’
Peer-review achieves the exact opposite of what we should want from science. It cements the power of experts. It acts as a brake on progress. It rewards those who maintain the status-quo. It helps to ensure that acceptable papers are published, and unacceptable papers are not.
Yes, I am fully aware that the vast majority of people use the term ‘peer-reviewed’ as a term of praise. A stamp of scientific veracity. It has the exact opposite effect on me. It grates horribly. Just publish the damned paper and let me decide if it is a load of rubbish, or not, thank you very much. I do NOT need a board of censors to decide what I can and cannot see. Lest my poor little unformed and childish mind becomes corrupted.
I also know, I really do know that we would all love to believe in peer-review. Surely it is better than doing nothing. We cannot just let any old crap get published, can we? To be perfectly frank, the idea that we have to do something, simply because we believe something must be done, is an insatiable human drive that is another of my pet hates.
A.N. Idiot: ‘Something must be done.’
A.N. Other Idiot: ‘Here’s something, let’s do that.’
Me: Sigh. ‘With or without any evidence that it works?’
Further Idiot: ‘Evidence, we don’t need evidence. It is obvious that this will be effective.’
All idiots together: ‘Well, that’s good enough for me.’
Here is the contrary standpoint. If doing nothing is just as effective as doing something, then I always recommend we take the ‘doing nothing’ option. Apart from anything else it frees up time to do other things that are clearly more beneficial. Such as getting in a bit of whisky tasting or picking your teeth.
In fact, doing nothing is part of my broader ‘don’t just do something, stand there’ initiative. Unfortunately, almost everyone else seems to favour the ‘Work, work, busy, busy, chop, chop, bang, bang.’ philosophy. ‘Looks how busy I am. I must be doing good.’ To quote Bing Crosby:
‘We’re busy doin’ nothin’
Workin’ the whole day through
Tryin’ to find lots of things not to do
We’re busy goin’ nowhere
Isn’t it just a crime
We’d like to be unhappy, but
We never do have the time
I have to watch the river
To see that it doesn’t stop
And stick around the rosebuds
So they’ll know when to pop
And keep the crickets cheerful
They’re really a solemn bunch
Hustle, bustle
And only an hour for lunch.’
I love that song.
Having said this, I also do believe we should try to ensure that research papers are not complete rubbish, based on fraudulent research (see under the Surgisphere paper on hydroxychloroquine – published in the Lancet). For science to work, we should be able trust what we read. As far as this is possible.
But the peer-review system, as it currently exists, does not achieve this. It allows utter made-up rubbish to be published. Worse, much worse, it stops a great deal of potentially valuable research dead in its tracks.
‘If mankind is to profit freely from the small and sporadic crop of the heroically gifted it produces, it will have to cultivate the delicate art of handling ideas.’ Wilfred Trotter.
Therefore, gentle reader, I have a suggestion. Journal editors should make their own decisions about what should and should not be published, based on how interesting and valuable it seems, then publish. Do not hide behind shadowy peer-reviewers, who have their own agendas to pursue.
At which point you use the Internet for what it is good at. Get a bloody good discussion going. Make the article free to view, for anyone, for the first two or three months – or longer. Invite a broad scientific audience to get involved.
Make it easy for people to attack it or praise it. Hit the upvote button. There are very many, very smart people out there. If they can’t find a problem with a paper, fine. If they can, get the authors to argue their case. Publish the best responses. Expose the discussion to the world. Pull the paper, if needed. Slap various addendums on it, such as ‘readers should note that this paper is a steaming pile of…’
Would this work. Well, it was certainly not the Lancet editorial team, or the peer-reviewers, or even the authors of the paper, who recognised that the hydroxychloroquine paper was fraudulent. It was other researchers from around the world who pointed out that the data were made-up.6
So, in my view, we need to allow the entire world to be reviewers and get rid of peer-review. Other than use it to provide helpful suggestions as to how to make the paper better. Just to add that the helpful elf who edits my blog ramblings, had this to say about this blog:
‘Like it – what you’re suggesting is a TripAdvisor like free scientific paper web site that can be commented on by anyone … ‘ Which is a bloody good summary.
I lay this suggestion before you with all great humility. Next, I hope to discuss the FDA, and the other regulatory bodies around the world. Let me see. What comes after hyenas? Vultures, great white sharks, vampires, leeches … let me think.
1: https://pubmed.ncbi.nlm.nih.gov/18603380/
2: https://www.nature.com/articles/news.2010.132
3: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005733/
4: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005733/
5: https://www.politico.eu/article/lancet-review-process-following-covid-19-saga-coronavirus/
6: https://www.the-scientist.com/features/the-surgisphere-scandal-what-went-wrong–67955