According to FiveThirtyEight.com’s Christie Aschwanden, “Science Isn’t Broken. It’s just a hell of a lot harder than we give it credit for.” Aschwanden’s article is a remarkably clear and forceful tour of all things wrong with science—poor statistical practice, poor study design, conscious and unconscious manipulation of data, outright falsification of data, plagiarism, fraudulent peer reviews, predatory publishers, fake authors of gibberish articles in mostly fake journals, ingrained psychological biases, and over-enthusiastic journalists breathlessly reporting on each new study as if it is the Truth etched in stone. But despite all that, she concludes that:
Science isn’t broken, nor is it untrustworthy. It’s just more difficult than most of us realize. We can apply more scrutiny to study designs and require more careful statistics and analytic methods, but that’s only a partial solution. To make science more reliable, we need to adjust our expectations of it.
Aschwanden is one of my favorite science journalists, but in this case I feel like she dipped her toes into the abyss and turned back, unwilling to to take that final step into the terrifying unknown of admitting that yes, science is broken—obviously, deeply broken. It is permeated by perverse incentives that reward publishing results over discovering truth, and corporations are hijacking scientific institutions to give their products the stamp of scientific validation. No simple fix in how journalists report about science or to the details of statistical practice or study design will fix it.
The bulk of Aschwanden’s article explains the now familiar topic of p-values and their abuse. P-values, or measures of significance, are meant to be a measure of how surprising a study’s results should be if its hypothesis is false. For example, climate scientists can’t say for sure that surface temperature increases over the last century aren’t due to human causes; all they can tell us is that this pattern of temperature increase, with 9 of the 10 warmest years having occurred since 2000, would be incredibly surprising if humans weren’t responsible. My favorite part of the article is the “Hack Your Way To Scientific Glory” tool, which allows you to bake your own economic study, using a variety of possible parameters to attempt to find a significant correlation between U.S. economic performance and whether Republicans or Democrats are in office. Depending on what choices you make, you can show that either party has a significant impact on economic performance, in either direction. It is the best demonstration I’ve seen of the problems with significance testing.
Aschwanden discusses various potential fixes of, and alternatives to, standard significance testing, as well as many other problems with the current practice of science, such as peer reviewers who rarely check the statistical methodology of the papers they read, because it is time-consuming and there’s no real incentive to do a good job as a peer reviewer. One of the more interesting fixes is a proposal by Brian Nosek to have many teams analyze the same data using different methods, and compare their conclusions—a kind of robustness analysis and meta-study rolled into a single paper. Another is the move towards post-publication peer review (ie. online comments sections).
Nevertheless, I find Aschwanden’s conclusion that “to make science more reliable, we need to adjust our expectations of it” entirely unconvincing. Yes, she’s right that science journalists should be more tempered in their reporting of scientific results. And yes, changes to standard statistical practices and peer review would help. But these things don’t get at the fundamental problem: science (and academia) reward quantity over quality. Two factors have combined to break science: the intense focus on published results for career advancement and the commercialization of research.
As anyone who has been paying any attention to academia knows well, PhD graduates have been outpacing tenured positions for a while now. Since the mid-90s, the number tenured or tenure-track positions has flatlined, while the number of non-tenure track positions has more than doubled and PhD recipients have increased steadily. In the early 90s, there were about 7000 PhDs granted per year in the life sciences, about 6500 in the life sciences, and a similar number in the physical sciences. In 2012 those numbers had expanded to about 12000, 9000, and 8000. More graduates, same number of top-tier positions. The implications are obvious: competition for the few positions that do open up every year is fierce. An absolutely terrifying New Scientist article aimed at postdocs and early-career researchers offers the following pieces of advice:
- “It’s really important to publish, not to put it off,” says Julia S. Austin, who teaches postdocs research writing at the University of Alabama at Birmingham. “Map out where you’re going and when you need to be publishing, and put yourself on that schedule.” You don’t want to let a year or two go by with no progress.
- In astronomy, says Rodger Thompson, a professor at the University of Arizona, postdocs generally have many projects going on at once, so they can churn out between five and 10 papers a year. To make sure you keep up, ask your principal investigator how often they expect you to submit papers when you start a new position.
- “Postdocs really have to have some outstanding publications, usually at least one first-author paper in one of the top-tier journals, to even pass the first round” of interviews for a faculty job, says Stephen Miller.
Although the number of scientific articles published in the U.S. is growing at a modest 0.7% per year according to the NSF, the publication imperative has manifested in many other ways. One of the most talked about effects is the dramatic rise of co-authorship of scientific papers; papers with 50, 100, or even 1000 authors are published regularly. Economist Paula Stephan argues that scientific laboratories have essentially turned into research paper factories: “The incentives are to get bigger and bigger, employing more graduate students and postdocs, which in turn result in more publications, more funding, and more degrees awarded.”1 A more amusing trend is an increase in marketing-speak in paper titles, with words like “novel”, “amazing”, and “unique” becoming significantly more popular in recent years. It’s not only journalists that are overhyping scientific results; scientists are too! As reported in Nature, researchers commented that their results “fit our own observations that in order to get published, you need to emphasize what is special and unique about your study.” Overall the picture we see of scientific publication is that pressures are getting ever more intense as year after year more and more people are chasing after the same few positions.
Along with increasing career pressures on researchers, the last several decades has seen a transformation from publicly funded to industry funded, commercialized research. In the 1960s, between 60 and 70 percent of academic research funding came from the federal government. Industry has been increasing its share of research funding ever since. In the early 80s industry funding passed federal funding as the main source of research funding, and since the late 80s its share has been steadily increasing.2 Patents and other forms of intellectual property are on the rise, both in the U.S. and internationally, with such agreements as TRIPs (trade-related aspects of intellectual property).
These two trends—increasing career pressures and increasing commercialization—combine to subvert scientific practice. Ghost authorship, where industry scientists write studies and academic scientists sign them, is one example of this. The scientist gets a much-needed publication along with funding for further research, while the company gets scientific credibility for what amounts to an advertisement for their product.3 Study after study has demonstrated that industry funded research is overwhelming biased to favor its sponsors.4 Regarding the increased use of words like “novel” and “amazing”, commenter Chris Aldrich observed,
The US Patent Office, in making a determination of whether or not to grant patents, (and even moreso when the underlying documents are published scientific research articles) uses words like “novel” and “innovative” as an indicator of their worthiness. Without a patent, it’s incredibly difficult to protect the intellectual property contained within a published and public work. Other words like “amazing” which are also cited in the paper are geared (in a business sales sense) more toward potential corporate financial investors who may consider purchasing or licensing the resulting work of a research paper.
So is it that, as Aschwanden argued, we just need to adjust our expectations of science? No! All evidence suggests that science is deeply broken. Career pressures are forcing scientists to publish sub-par research and overhype that research while industry eagerly takes advantage of these pressures combined with flatlined public funding to subvert scientific institutions. Mere changes in statistical practice or peer review will be no more effective in fixing these fundamental problems than international agreements with no binding emissions targets will be at slowing global warming.
- Stephan, Paula. How Economics Shapes Science. Harvard University Press, 2012. Print. ↩
- Sent, Esther-Mirjam, and Philip Mirowski. “The Commercialization of Science and the Response of STS.” The Handbook of Science and Technology Studies. Ed. Edward J. Hackett and Society for Social Studies of Science. 3rd ed. Cambridge, Mass.: MIT Press, 2008. ↩
- Stephan, Paula. How Economics Shapes Science. Harvard University Press, 2012. Print. ↩
- Sismondo, Sergio. “Pharmaceutical Company Funding and Its Consequences: a Qualitative Systematic Review.” Contemporary Clinical Trials 29.2 (2008): 109–113. Web. ↩