I had that argument yesterday. We’re creating predictive models based on extrapolating based on current trends and get totally tripped up by conditions that have never been previously observed. We’re at a collective cognitive loose end where we need to be able to anticipate what’s going to happen but don’t have any precedent to draw on.
Right. In the literature about computational modeling, we call this kind of thing “structural model error.” It’s an especially nasty kind of uncertainty, because none of the standard tricks that we use to account for other sorts of error in our computational modeling (ensemble modeling, parameter perturbation, multiple model runs, &c.) can really deal with it. There are two big assumptions that underpin how we use computational/numerical models (both in climate science and elsewhere): first, that our models are approximately true, and second that approximately true models will yield approximately true predictions. If we turn out to be mistaken about either of these things, the predictions our models are generating might (in the worst case scenario) end up being worse than useless–they might be actively misleading in the sense that they tell us that very low probability events are nearly certain to happen and very high probability events are extremely unlikely to happen.
The likelihood that we’ve gotten something deeply wrong in our modeling–that there’s some major feedback mechanism, dynamic, pattern, or whatever that we’re simply not accounting for–and thus that we’ve subject to extreme structural model error grows larger and larger the further we get away from the region of a system’s state space in which our models were trained. As we push the climate further and further into unprecedented temperature ranges that aren’t reflected anywhere in our observational data, it becomes more and more probable that we’re going to trigger some really significant feature of the system that will give us a qualitative change in dynamical form, rendering our predictions useless (or worse) until our theoretical understanding catches up. This possibility is something that keeps a lot of climate scientists (and philosophers of science who are engaged with this stuff in a detailed way) up at night. We have no real way of knowing if we’re approaching a region of structural model error until we’re already there, and we don’t really have any way of dealing with the error once we get there (or, at least, no way that anyone has figured out yet). It’s good stuff.
That’s really interesting! Do you have a high-level overview reference that I could use to bash my colleagues over the head with start a conversation in the office about the assumptions we’re making in our modeling work?
“The Myopia of Imperfect Climate Models” and “Laplace’s Demon and the Adventures of His Apprentices” (both by Roman Frigg) are both pretty good overviews of the problem. The former is more of a specific case study with some considerations about general lessons, and the latter discusses the problem by extending the “thought experiment” of Laplace’s Demon. I’ve got a couple of publications on this too, so if you have questions let me know.
It’s great how our models keep failing in the direction of “we underestimated how catastrophic this would be.” That surely bodes well for the future.
I had that argument yesterday. We’re creating predictive models based on extrapolating based on current trends and get totally tripped up by conditions that have never been previously observed. We’re at a collective cognitive loose end where we need to be able to anticipate what’s going to happen but don’t have any precedent to draw on.
Right. In the literature about computational modeling, we call this kind of thing “structural model error.” It’s an especially nasty kind of uncertainty, because none of the standard tricks that we use to account for other sorts of error in our computational modeling (ensemble modeling, parameter perturbation, multiple model runs, &c.) can really deal with it. There are two big assumptions that underpin how we use computational/numerical models (both in climate science and elsewhere): first, that our models are approximately true, and second that approximately true models will yield approximately true predictions. If we turn out to be mistaken about either of these things, the predictions our models are generating might (in the worst case scenario) end up being worse than useless–they might be actively misleading in the sense that they tell us that very low probability events are nearly certain to happen and very high probability events are extremely unlikely to happen.
The likelihood that we’ve gotten something deeply wrong in our modeling–that there’s some major feedback mechanism, dynamic, pattern, or whatever that we’re simply not accounting for–and thus that we’ve subject to extreme structural model error grows larger and larger the further we get away from the region of a system’s state space in which our models were trained. As we push the climate further and further into unprecedented temperature ranges that aren’t reflected anywhere in our observational data, it becomes more and more probable that we’re going to trigger some really significant feature of the system that will give us a qualitative change in dynamical form, rendering our predictions useless (or worse) until our theoretical understanding catches up. This possibility is something that keeps a lot of climate scientists (and philosophers of science who are engaged with this stuff in a detailed way) up at night. We have no real way of knowing if we’re approaching a region of structural model error until we’re already there, and we don’t really have any way of dealing with the error once we get there (or, at least, no way that anyone has figured out yet). It’s good stuff.
That’s really interesting! Do you have a high-level overview reference that I could use to
bash my colleagues over the head withstart a conversation in the office about the assumptions we’re making in our modeling work?“The Myopia of Imperfect Climate Models” and “Laplace’s Demon and the Adventures of His Apprentices” (both by Roman Frigg) are both pretty good overviews of the problem. The former is more of a specific case study with some considerations about general lessons, and the latter discusses the problem by extending the “thought experiment” of Laplace’s Demon. I’ve got a couple of publications on this too, so if you have questions let me know.
Thank you so much!
New unpredicted patterns gives us new data which gives us a better understanding of how the weather works, which gives us better models.