Twice Is Nice! Double counting evidence in climate model confirmation

Charlotte Werndl (LSE) is speaking at Western University on Monday (the talk will be live-streamed) on evidence and climate change modeling. Having recently read her paper (with co-author Katie Steele) entitled “Climate Models, Calibration, and Confirmation” (CMCC) I thought I would post about it. The paper focuses on the use of evidence in confirming climate models with particular attention paid to double counting, which in this context means using the same evidence for two purposes (more on this use of the term later). I believe the paper is an important one, as it nicely separates concerns about double counting from other, related, confirmatory issues, and I think successfully shows where a form of double counting is legitimate. Still, despite being a casual fan of Bayesianism, I wonder if it gives us what we want in this particular circumstance. I can’t cover all the threads of argument made in the paper, so here I’ll simply discuss what double counting is, why we should worry about it, and how Steele and Werndl (S+W) argue that it could be legitimate in some circumstances.

What’s the worry about double counting? Climate change models typically go through a process called tuning (also sometimes called calibration). Tuning sets the values of parameters in the model that represent highly uncertain processes for which there are few empirical observations. The parameters are treated as “free parameters” that can take on a wide range of values. The values that result in the best fit with observations during tuning are the values chosen for the model. For example, if scientists are interested in global mean surface temperature (GMST), they would vary the parameter values of some uncertain processes until the model’s output of GMST closely matched GMST observations. The model, with these parameter values, would then be used to make climate projections.

The worry is that one way climate models are evaluated is by comparing their results to observations of some historical period; if scientists want to know if a model is adequate for the purpose of predicting GMST, they compare the model output to historical GMST observations. This agreement is supposed to build confidence in (confirm) the model’s ability to simulate the desired quantity. It is typically believed that to gain any confidence in the model at all, the simulation output must be compared to a different set of observations than the one that was used for tuning. After all, the observations used for tuning wouldn’t provide any confidence, because the model was designed to agree with them!

To deal with double counting, CMCC adopts an explicitly Bayesian view of confirmation. The Bayesian view adopted is necessarily contrastive and incremental: a model is confirmed only relative to other models, and the result of confirmation is greater confidence in the model for some particular purpose (not a claim that the model is a correct representation or the truth). Confirmation of one model relative to another can be tracked with the likelihood ratio, which is the probably of the evidence conditional on the first model divided by the probability of the evidence conditional on the second model. If the ratio is >1, the first model is confirmed,

So here is a simple way in which double counting is legitimate on the Bayesian view presented in CMCC. Imagine tuning some model M whose parameters have not yet been set (S+W call this a base model). In order to tune it, we create several different instances of the base-model, all with different parameter values: M1, M2, and so on. We compare the results of each model instance to observations and select the best fitting instance. This is an example of double counting in the following sense: the same data is used to both confirm and tune the model. This is tuning, because we have selected parameter values by comparing outputs to observations, and it is confirmation, because we have gained greater confidence in one instance over all the other instances in light of the data. S+W call this double-counting 1 and it is fairly uncontroversial.

Double-counting 2 seeks to confirm two different (M and L let’s say) base-models, but the situation is much the same. The Bayesian apparatus is more complex, and I’ll leave it to my readers to seek out the details in the paper itself. However, the evaluation still deals with likelihood ratios, it is just that the likelihood ratio needs to take into account all the instances of base-models M and L, as well as our prior probabilities regarding them. The likelihood ratio becomes a weighted sum of the probability of the evidence given each model instance for one base-model over the other. Double-counting 2 is legitimate in two situations 1) the average fit with the data for one base-model’s instances is higher than the other model’s (assuming the priors for each model were equal) and/or 2) the base-models have equivalent fit with the observations, but one model had a higher prior probability (was more plausible). An example of (1) would be that base-model M is tuned to the observations, and on average, its instances are closer to the observations than model L’s. This would result in a greater likelihood for M compared to L, and thus confirm M relative to L. Again, even in this situation tuning “can be regarded as the same process as confirmation in the sense that the evidence is used to do both calibration and confirmation simultaneously” (p618).

Quick Comments

S+W do a great job distinguishing the two kinds of double counting and separating them from other concerns about model tuning and climate projections (this work is done in the second half of the paper not discussed here). They seem right, given the view of confirmation they hold, that confirmation and tuning can be done with the same evidence. After all, double counting S+W’s sense is a sophisticated way of saying that the model that fits the data best is more likely.

A few issues worth thinking about:

1) Double counting here is a bit of a misnomer.…

Review: Cold War Social Science

Cold War Social Science: Knowledge Production, Liberal Democracy, and Human Nature

Mark Solovey & Hamilton Cravens (eds.)
Palgrave Macmillan (2012)

Cold War Social Science is an edited volume that serves as an ideal entrée into the history of social sciences in mid-twentieth-century America, as well as an argument for its subject matter as a distinct subfield in the history of social science.

The volume is divided into three themed sections: Knowledge Production, Liberal Democracy, and Human Nature. As David Engerman’s chapter shows, social scientists were heavily influenced by their experiences during the Second World War: “They abandoned disciplinary questions in favor of policy concerns; they rejected longstanding traditions of solitary work in favor of collective research enterprises; and they worked closely with the national security organs that sponsored their work”(25). Engerman’s and other chapters in the first section detail how wartime and Cold War institutions such as the RAND corporation, along with technological changes and the perceived successes of mathematical economics, led social scientists to focus on quantifiable, theory-driven research projects.…

The Sorry State of Academic eBooks

A couple of weeks ago marketing guru Seth Godin observed that you can tell what an organization values by where they put their best people. Given the current state of academic ebooks, I suspect that most publishers are using unpaid work-study students. I don’t know much about the economics of academic publishing, but this strikes me as a very shortsighted strategy. The market for print editions of academic books is probably both small and almost static—primarily university libraries. But there is a ton of growth potential in e-publishing, not just from academics like me who are too impatient to wait for the library to get a copy of the book we want to read (or for the library to open in the morning), but also from thousands of interested laypeople who would be willing to put the effort into an advanced text if it were convenient and affordable.

In the rest of this post I am going to give a tour of some of the mistakes and annoyances I’ve found in my collection of ebooks. To be fair, some of the books I’ve purchased are pretty good, but for the most part this is a representative sample. I should also note that none of what I say is a slight against the authors of these books. I enjoyed all of them and I think they should be outraged at the poor job their publishers have done in presenting their work to the world.…

Shaky Foundations

I learned from Mark Solovey today that his book, Shaky Foundations, has just been published. Congratulations Mark! Shaky Foundations describes “the politics-patronage-social science nexus in Cold War America”. This makes the last year a big one for Mark, as he was also co-editor of Cold War Social Science, an edited volume about social science during the Cold War. Will Thomas of Etherwave Propaganda has a good review as well as some interesting thoughts on the “definition of social and political ontologies” in relation to the volume.

The Cold War was a transformative period for the social sciences. Progress in game theory, linguistics, economics, operations research, and other disciplines, along with notable successes in World War II, suggested that the social sciences were at last, or nearly, on an equal footing with the physical sciences in terms of objectivity and rigour. At the same time, the complexities of the Cold War led the military to fund social scientific research at an unprecedented scale.

Shaky Foundations has three central arguments. First, the military, the Ford Foundation, and the National Science foundation must be considered as part of a single, though loosely integrated, system for funding the social sciences. Second, these patrons saw the social sciences in instrumental terms—as means of attaining practical ends—and used the physical sciences as a measuring stick for the social sciences. Third, this new patronage system faced several challenges—including ideological challenges from both conservatives and liberals, concerns over the independence of scientific research, status differences between the social and natural sciences, and the difficulty of pursuing value-neutral inquiry—that led to increasing academic and political scrutiny in the 1960s.

I have not yet had a chance to read Mark’s book beyond the introduction, but it looks to be a valuable work for those interested in the history of social science as well as those interested in the relationship between academia and society.


Book Review: The Disappearing Spoon, by Sam Kean

[amtap amazon:asin=0316051640]

Sometimes the right book finds you at the right time, and it shifts your perception just a little, just enough to make a difference. It reminds you of something important you haven’t thought of in a while, or it shows you a new way of looking at and interacting with the world. For me this winter, that book has been The Disappearing Spoon, by Sam Kean. I heard a very fuzzy description of the book at a holiday party, something about the periodic table and political history. As someone eternally interested in chemistry and its impact on society at large, I was intrigued.

The book accompanied me through a whirlwind holiday travel season, and as I read little kernels of story about each of the elements in the periodic table, I found myself unable to stop bringing them up in conversation. As my family pulled foil over Christmas leftovers and discussed my life as a graduate student at the University of Pittsburgh: “Did you know that aluminum used to be more expensive than gold, and that Pittsburgh is where the guy who figured out how to isolate it cheaply set up shop?” As news of the flood in Brisbane hit American televisions: “Did you hear that Australian astronomers used chromium to provide evidence that the fine structure constant may change over time?” As friends argued about how pharmaceutical companies should respond to decline effect and toasted the New Year: “In 1932, Gerhard Domagk tested the first antibacterial drug, Prontosil, on his sick daughter in order to save her arm. If he had gotten caught doing it, he would have been arrested.”…

Science in Democracy, by Mark B. Brown

Mark B. Brown’s Science in Democracy is a must-read for anyone concerned with the interaction between science and politics. It is a tour of political theory — from Machiavelli, to Rousseau, to Dewey, to Latour — as well as an argument for rejecting the traditional “liberal rationalist” view of science and politics, and a guide to facilitating a better relationship between them.

Although I have always been very aware of the connection between science studies and political theory, I have had no more than a vague conception of the theoretical and historical roots of that connection. Brown’s first task, in part one of the book, is to explain liberal rationalism and how it became the dominant view of science and politics. Roughly, liberal rationalism is the idea that there is, and should be, a strong separation between scientists, politicians, and the public. Scientists are expected to be disinterested, objective, and politically neutral. Politicians are supposed to act in the interests of their constituents. The public is supposed to articulate those interests, but not to participate in the operation of either politicians or scientists, lacking the expertise necessary for participating in either sphere. In fact, the very ability to participate in either sphere may, in this view, disqualify one from being considered a proper member of the public at all. Many efforts at public engagement deliberately exclude anyone with knowledge or “preexisting views” relevant to the issues at hand (231-232). Machiavelli plays a dual role for Brown in this respect, both as an early advocate for the kind of public participation in politics that Brown advocates and as the historical originator of a “rhetoric of expertise” that created a strong division between the scientist and the public. Jean-Jacques Rousseau plays a much less ambiguous role, as an avatar of the most extreme vision of liberal rationalism. Rousseau advocated excluding not only the general populace from social or scientific deliberation, but also advocated excluding all but the most exceptionally talented — the Bacons, Newtons, and Descartes — from science or government.…

Review: The Scientific Life by Steven Shapin: A Moral History of a Late Modern Vocation

[amtap amazon:asin=0226750256]

In  The  Scientific  Life,  Steven  Shapin  argues  that  people  and  their virtues matter in late modern science. While scientists struggle to remain objective and impersonal, it is the personal, familiar, and charismatic–the traits once swept aside as vices by the scientifically virtuous–that have come to embody the “truth-speakers” of late modernity. With an enormous and  sometimes  daunting  wealth  of  primary  sources  (from  technical commentaries to his own sociological fieldwork), Steven Shapin breathes life  back  into  these  quotidian  virtues.  The  Scientific  Life  is  as  much a disjointed genealogy of scientific virtue as a reminder that trust still matters at the cutting-edge of scientific “future-making.” Shapin’s mastery of historical narrative is clear; anyone interested in the American scientific persona and how it has transformed in the twentieth century would do well to wade patiently through this thick and rewarding text. But hang up your expectations of historical linearity (and, sometimes, thematic coherence) as you weave through motley professionals, theorists, and critics drawn from over a century of science commentators. Perhaps this work is best described as textured: rich in detail, woven intricately, but hardly smooth to the touch.

Shapin begins by detailing the  transformation from science as calling to science as job in the late nineteenth and early twentieth century (chapters 1-3). During this period, the idea of science as vocation lost its impetus as the fruits of discovery became politically and economically valuable. Robert K. Merton’s sociology exemplified this shift, asserting that neither constitutional nor motivational differences existed between scientists and non-scientists.  The  Mertonian  “moral  equivalence”  of  the  scientist  (i.e. scientists  are  just  ordinary  folk)  eventually  displaced  Weber’s  “man  of science,”  in  whom  moral  authority  once  stemmed  from  a  merging  of curiosity and morality. The “spirited” scientist became the disinterested scientist, in personal convictions and professional identities. Despite the unclear origins of this “moral equivalence” (as Shapin prudently admits), a commitment to the idea persisted in the post-World War II era of “Big Science” and the military-industrial-academic complex.…

Brilliant: The Evolution of Artificial Light, by Jane Brox

[amtap amazon:asin=0547055277]

I was driving to the train station last week, and because of Daylight Savings Time (thanks so much for that, George Vernon Hudson) I’ve been doing so in the dark for the past few weeks. As I drove down one street, something struck me as being terribly wrong. It took me a moment to realize what it was: a lack of streetlights. For some reason, this one street was dark, and it really creeped me out.

The fact that I have become so reliant on artificial light; that its absence strikes me so forcefully, makes me all too typical an example of our modern age. In Brilliant, Jane Brox tries to unpick exactly how I (and everyone else) came to this point. While I found the book unsatisfactory as a history of science, it had several sections that were first-rate cultural history.…

The Poisoner’s Handbook, by Deborah Blum

[amtap amazon:asin=1594202435]

Next time you’re sitting down to watch CSI, CSI:NY, Bones, Navy:NCIS or any of the other dozen or so television shows that strongly feature forensic science, spare a thought for where the science began. When did police forces first start using science in a serious way to catch the bad guys?

That’s the question addressed by Deborah Blum in her delightful new book. Despite the subject matter (a series of grisly murders and accidental poisonings) Blum manages to write a book that is eminently readable, even for the non-ghoulish. Rather than dealing with the entire history of forensic science, she wisely chooses to present a micro-history, looking at the rise of forensic chemistry in New York of the Jazz Age and Great Depression.…

The Immortal Life of Henrietta Lacks, by Rebecca Skloot

[amtap amazon:asin=1400052173]

Many years (and several careers) ago, when I was working in the lab, I often used HeLa cells. They were a standard culture of human cells, used for a variety of purposes. I never gave much thought to from where (which is to say “from whom”) they came. We had a variety of cell lines to choose from, and HeLa were just the standard, go-to choice.

Fortunately, people like Rebecca Skloot are a good deal more curious than I, and she was inspired not only to research the history of these cells (and through that, the history of cell culturing as a science) but also to research the history of the woman from whom all those countless cells have come.

In many ways, tracking down Henrietta Lacks (the original donor) and explaining the story of cell culturing was the easiest part of the story. It is a standard detective tale of tracking down false leads and looking for evidence in unlikely places. And for Skloot, an experienced medical writer for the New York Times Magazine and NPR, this is just another day at the office. But the book goes beyond the scope of a standard history of science/medicine when Skloot decides to track down the descendents of Henrietta Lacks and tell their stories. It becomes an intensely moving, disturbing story of race and class in contemporary America.…