The headline caught my eye, as it was intended to: “One in five chemists have deliberately added errors into their papers during peer review, study finds.” It’s introducing an article in Chemical and Engineering News reporting on a new article in the journal Accountability in Research by Frédérique Bordignon, who surveyed research chemists about their experience of the peer review process. The article’s abstract echoes the news headline, saying, “Some authors yield to reviewer pressure knowingly introducing changes that are clearly wrong.” That’s a fairly eye-popping result — peer reviewers are pressuring scientists to introduce changes that are clearly wrong into our descriptions of our research?
Well, here’s the funny thing: If I’d been a reviewer on that paper, I’d have said that statement in the abstract was an error. I’d probably also have said that it was a dangerous one.
I suspected right away that this description of the survey results was incomplete, at best. Indeed, the text of the C&EN article has a distinctly different spin than the headline or the paper’s abstract: “When asked if they felt they were forced to modify their manuscript with text they thought to be incorrect, 22% of survey respondents said yes.” Authors making a change they thought to be incorrect is not actually the same as introducing an error! It may simply reflect peer reviewers disagreeing with an author’s interpretation of results in the paper under review, and the editor siding with the reviewers in that agreement.
This is the degree to which authors can be “forced” to make changes they don’t like to a paper — when expert reviewers ask for a change and the editor who oversees the review process decides that the paper will only be accepted with that change made. Authors are free to take their paper to another journal if they feel strongly enough about the disagreement, though this is not the path of least resistance. It’s a pain to start review over from scratch at a new journal, and if you want to publish sooner, your quickest route is to try to do what reviewers ask even if you’d rather not.
Does Bordignon’s paper have a softening from its abstract to its actual text that parallels the C&EN shift from “errors” to “changes they thought to be incorrect”? I have, unfortunately, no subscription access to Accountability in Research. Fortunately, Bordignon followed best practices and provided supporting materials on Zenodo that include the survey questions. The relevant one is Q13: “During the peer-review process of your own papers, have you ever made changes to your manuscript that you thought were incorrect (i.e. introducing an error), but you did them in response to pressure from reviewers?”
That is indeed more complex than the interpretation in the paper’s abstract. There’s a difference between “changes … you thought were incorrect” and “introducing an error,” despite the parenthetical offering the latter as an example of the former. Speaking as an author of peer-reviewed papers, “introducing an error” would certainly be a change I “thought was incorrect.” But! I have absolutely had reviewers ask me to make changes I considered incorrect, and made them — because the reviewers and I disagreed on a point where there isn’t a clear factual answer.
Many such questions arise in the design and execution of a scientific project and the interpretation of its results. Reviewers have asked me for changes to things like figure design, or the use of specialized terminology, or how I communicate the strength of confidence in a statistical test result — or even what conclusions I could legitimately draw from the data. I have strong opinions about writing and figure design, and I know all too well they’re not universal; but there may also be ambiguities in choosing methods and statistical tests. Some examples, off the top of my head:
- Using frequentist versus Bayesian methods — Yes, there are specific cases where this choice can change your risks of a false positive; but in many situations, frequentist methods and their Bayesian equivalents will give the same qualitative answer, and which you trust more is down to personal preference.
- Modeling a process via detailed simulation versus a more abstract analytic model — Here again, there are pros and cons to each choice. A simulation can give you granular control of complex details; an analytic model can give you an elegant description of relationships between the variables that are most important. (Many papers present simulations and analytic models of the same process for precisely this reason.)
- Displaying variation in treatment groups — There are many, many options for comparing the central tendency and variation of measurements organized by treatments. Which you choose depends on your sense of what will optimize clarity, emphasize between-group differences, or even specifically indicate when groups would differ significantly in a standard t-test. For the most part, though, there are no truly wrong choices, so long as the resulting figure isn’t actually deceptive.
- Selecting “candidate loci” from a genome-scan for association with a trait or environment — In genome-wide association studies testing thousands or millions of loci, even after you raise the threshold of “significance” to account for multiple testing, you may have many, many more loci associated with the trait or environment of interest than you can investigate individually. You can triangulate with other data or analyses to narrow the list, but how you choose a subset for further study is simply a judgement call.
The whole point of peer review is that there will be questions like this, on which reviewers and I may disagree even if we’re all experts in a narrowly defined field. If an editor feels reviewers’ opinions are better aligned with the consensus of our field, I can either accept that judgement or find a venue where I think I’m more likely to find agreement with my own way of thinking. However this is the real value that I get from peer review, as an author: Even when I’m sure I’m in the right, I may actually be wrong! Peer reviewers are there to tell me when I’m at risk of being wrong.
(Does peer review always accomplish this? Of course not. We’re all fallible humans muddling through a terrifying and unknowable universe. But the goal is that multiple of us are more effective at the muddling than any one of us alone.)
So anyway, Bordignon asked research chemists a question that encompasses both “did peer review force you to say something false in a paper” and “did peer review require you to change a paper in a way you didn’t like but couldn’t defend to an editor” — then foregrounded only the former, much more alarming version of that question in reporting that 22% of her survey participants said “yes.” I think I’ve spent enough text explaining why I think that choice in itself is an error. Why is it a dangerous error?
In short, it’s dangerous because endangering public trust in the scientific process — of which peer review is a critical part — has higher stakes than at any point in my scientific career. The Trump Administration is framing an ongoing assault on research funding and academic freedom as campaign for “gold-standard” science, which is not so much “spin” in the old political sense as an outright lie: placing political apparatchiks in positions to decide what research is funded and how its results are reported. The Secretary of Health and Human Services — who has no business remaining in that role — is using his national platform and institutional power to tell people that science isn’t trustworthy. The Environmental Protection Agency is revising the actual evidence out of its evidence-based understanding that climate change poses a threat to human health.
Work that undermines the credibility of peer review is ammunition for these ascendant campaigns against public health and environmental protection. That’s not to say no one should criticize scientific processes or publishing — there’s plenty of that to be done, and self-critique is how science gets better. But now, more than ever, ensuring such criticisms rest on a rock-solid foundation of evidence is a matter of life and death. Claiming that peer reviewers have made as many as 1 in 5 scientists add errors to their work doesn’t rise to the standard we desperately need at this moment.