What Scientific Consensus Looks Like

Nearly everyone says they believe in science, but relatively few people have much experience with how the scientific process actually plays out. Interest groups on multiple sides of multiple issues often confuse matters by citing “science” that is on their side. Worse, some people undermine the credibility of the entire scientific enterprise by cherry-picking past statements from individual researchers, sometimes out of context, that seem ridiculous today. So, here is what laypeople should understand about how the scientific process really works, when you can claim that “science says…”, and when you should not. Keep this in mind when you are digesting news about recent findings, reading political arguments, and so on.

Rule #1: One paper is not the truth.
I put this first because it’s the most important. The fact that a group of scientists has gone through the trouble of doing the work and writing the paper should tell you that the science is not settled. Nobody today publishes papers about whether the Earth revolves around the sun or vice versa, although that was once a hot question in astronomy. Publishing a paper in a peer-reviewed, scientific journal is fundamentally an attempt to persuade fellow scientists of something – who else reads these journals? Each paper is part of an ongoing dialogue (or shouting match) within the scientific community. What you cannot tell from a paper is how many of the author’s colleagues agree. Is the paper the majority or minority view? Is the community evenly split? Is the paper the last gasp of the side losing the argument?

Peer review is only the first bar we use to determine whether one of these papers should be taken seriously or not, and peer review is not an ironclad guarantee that a paper is “right” in the larger sense of the word. One challenge is that reviewers are busy people and, almost without exception, peer review is something we do anonymously and for free. There are not a lot of incentives to spend too much time on it, although some research communities are trying out new processes. Moreover, there is no way that a single peer reviewer can check, in hours, what it took an entire team of people months or years to accomplish. When I do peer review, I am mostly trying to determine if the paper seems reasonable enough to merit a place in the dialogue. Do the authors explain their methods in enough detail so that someone else could reproduce the work? Do they present enough data to support their interpretation and findings? If their findings are especially surprising, do they make extra efforts to explain and convince?

The news media frequently does stories on recently published single papers, and laypeople often cite individual papers as if a single, peer-reviewed paper were somehow definitive. Sorry, but you don’t have my attention yet – things are more complicated than that – one paper is not truth.

Rule #2: The real peer review happens after the peer review.
Maybe the most important part of the scientific enterprise is the least visible. Individual scientists read and discuss new papers, weigh the evidence across a body of literature, look for patterns, and start to form more definite opinions. If a single paper tells the story of how its authors came to be persuaded of something (and sometimes the “conclusions” of a paper are quite tentative, even for the authors), the real test is when and if their colleagues become convinced.

There are a lot of reasons why a researcher will weight some papers more than others. Maybe the papers that go against the trends of the literature all have small sample sizes. Maybe some methods or measurements are not well-suited to some questions or have been superseded by newer techniques. Subtle explanations can bring coherence to a body of literature that looks contradictory at first glance. The data in a paper might be good even if the authors overlooked an alternative interpretation. Unfortunately, these judgments really do require expertise, so journalists and laypeople often take individual papers too seriously.

Over time, papers that are judged to be more important and persuasive get cited more frequently. Researchers are obsessed with citation count as one indicator of how good or influential their favorite papers have been.

Some of the highest profile journals, places like Science and Nature, have a mission to disseminate rapidly and widely the really new and surprising findings. These are more likely to get news coverage, but the new surprises are often the papers that are most in need of confirmation.

This is also why the scientific process values reproducibility. If a topic is important, other researchers will look into it using the same or alternative methods. If the original researchers were careless or fraudulent, it is very likely to be discovered in time.

Peer-reviewed papers are the building blocks of scientific consensus, but they are not consensus by themselves. Each is subsequently inspected and weighed. A few, sometimes a lot, are disregarded as spurious, and the rest are used to assemble the building we call “consensus”.

Rule #3: When there is consensus, the scientific community will tell you so.
If the critical process of assessing a conflicting body of literature is mostly invisible, it can be hard for outsiders to understand really what is going on. Fortunately, there are a number of trusted organizations specifically tasked with communicating the overall assessments of the scientific community. One leading example is the US National Academy of Sciences, established by Congress in 1863, with the mission to provide “independent, objective advice to the nation on matters related to science and technology.” Other examples would be the Centers for Disease Control, the American Medical Association, and other professional organizations like the American Geophysical Union, etc. Once they issue official statements or reports based on committees convened to assess a question, it can be considered to be the consensus.

Really, the only difficulty is deciding if a particular organization is a reputable authority in a certain area. Sometimes interest groups set up official-sounding organizations to muddy the waters, but it’s usually not hard to weed them out of the conversation.

Some question the integrity of the consensus because aren’t we in it for the research money? Funding is a vital concern of researchers, and it has led to individual instances of bias, but I doubt there has ever been a case where the science establishment declares consensus on a topic simply for the money. There are multiple safeguards. First, university researchers with tenure have the independence and flexibility to change their own research agenda. Only about half of my work is climate change related. If I weren’t sincerely convinced of climate science, I could easily switch to working entirely on traditional air pollution issues. Second, to be honest, in many fields, the sums involved just aren’t that much and go overwhelmingly to pay for the costs of doing research rather than into the pocket of the researcher. Third, besides the transparency and reproducibility checks in the science process, the committee reviews that establish consensus typically include independent scientists, ones that are not getting the research money for a topic but are in a closely related field so they are qualified to review. To get a tainted consensus statement, you would first need a collective failure of ethics by all the researchers receiving money, and then they would need to fool their knowledgeable peers who are not receiving money.

Rule #4: There are always stubborn individuals.
Even after a hard-earned consensus emerges, there will always be some who disagree. These hold outs may be stubbornly holding onto positions they advocated while there was a real scientific debate. Or, more rarely, they may be the eccentric genius that sees what others don’t. Individual scientists are frequently wrong. Official consensus statements of reputable groups of scientists are almost always right. This is especially true for well-studied and controversial topics that have received a lot of scrutiny.

Science is never considered “finished”, and the scientific process allows for dearly held views to be overturned in the face of new evidence. It’s fair for “science” always to remain open to the possibility, however remote, that the consensus is wrong and the stubborn eccentric is right. But it’s foolish for individuals or society as a whole to assume they know better and bet against the consensus. It’s also foolish to put faith in the statements of the lone dissenters or to mistake individual papers for consensus.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s