Pandemic predictions

 Pandemic predictions

“The only function of (economic) forecasting is to make astrology look respectable”

JK Galbraith

You can guess where I am going with this. And just so we’re clear, I am only referring to the COVID-19 disease predictions here, not the economic ones. Now, I know some of you may not take too kindly to my comparing infectious disease epidemiologists to astrologers, and some others (hopefully not too many!) may even side with their personal astrologers. But the point I want to make is that though disease prediction models have a rational scientific basis, the large number of known and factors involved, increase the uncertainty to an extent that they become largely unreliable. The other common feature of both these fields (and economics as well) is that wrong predictions have no deterrence value: there is always good enough reason as to why you were wrong in the first place, ranging from “Jupiter is not in a position to shower its grace (sic)” to “..fewer people wore masks” (take your pick). Of course, I am not referring here to the serious academics who do disease modelling; they are only too aware of their fallibility. But many others have emerged from the woodwork during this pandemic. The easy access to data and computational tools has lowered the barrier for entry to a large number of wannabe experts. As a result we have a zillion experts embarrassing themselves (and the rest of us) on the internet. As a species, we really need to learn to recognize the extent of our ignorance!

 

Garbage in

Anyone with some understanding of programming can create a so called “epidemic calculator”. Knowledge of the disease is not a pre-requisite. Take this calculator available freely on the internet, https://gabgoh.github.io/COVID/. The interface allows you to vary things like the population size, and the now famous R0 and see how it affects the number of cases and deaths. Cool! You’d say. True. On the face of it, everything seems fine; it cites all the relevant studies from where it has collected information critical to the model. But on closer inspection, it gives you some ridiculous results that are not compatible with what we know about the disease. For example, if you increase the duration that a patient remains infectious, it paradoxically reduces the number of infectious cases, and the number of deaths! And the number of days in hospital has zero effect on the number of deaths! Now you don’t need a degree in medicine to sense that something is amiss here. The programmer did not respond to my queries.

 

One could argue that models like these are merely a bored graduate student’s hobby horse, and do not affect government policies. But even the models constructed by experts need to be carefully assessed for their underlying assumptions. But, typically, the take-away for policymakers, media and the general public are the headline figures of the number of cases or number of deaths. For example, predictions from a respected university were flagged for their controversial assumptions, and produced unreliable numbers, but nevertheless were used by the US government to devise policy (https://jamanetwork.com/journals/jama/fullarticle/2764824). Another group of well-meaning academics produced a set of predictions for India early on in the pandemic (https://doi.org/10.1101/2020.04.15.20067256). Even their short-term predictions of the number of cases were off by a factor of 3 (best estimate of the predicted number was under 10,000; the actual number was 35,000). And then, there was also the uncertainty involved; the authors suggested that the numbers could be as large as 75,000. It would be naïve to expect that the general public and policymakers, well-intentioned or otherwise, will appreciate how wide the 95% uncertainty intervals around an estimate are!

 

There has also been great interest in predicting the number of cases of COVID-19. I am somewhat baffled by modellers who try to predict case numbers without reference to testing. Case counts are completely determined by the number of tests conducted, and on whom they are conducted (people with symptoms, or passers-by stopping at traffic signals). In the above case, testing in India had increased 35 times during their short prediction window! Predicting deaths or ICU bed occupancy as others have done (https://spiral.imperial.ac.uk:8443/handle/10044/1/77482) is a more reasonable approach. In any case, the uncertainty in the estimates remains a serious problem. But many presumably intelligent folk have glibly ignored this fact. For me, this particular one takes the cake: https://www.flasog.org/static/COVID-19/COVID19PredictionPaper20200426.pdf. If it were not for the tragic irony, it makes for delightful reading. It is seductively titled “When will COVID-19 end? Data-driven prediction”. The highlight is a table that presents the exact date on which the pandemic is predicted to end in each country of the world! Just as a teaser, the world is set to be free of the pandemic on the 6^th of December 2020. Folks in the UK and US, in case you missed it, the pandemic ended for you on the 14^th and 27^th of August!

 

Information and noise

What should you believe then? Predictions are always fraught, much more so in a rapidly evolving pandemic. There are far too many variables that can influence them. And many of these are unknown ones, like the famed “Immunologic dark matter” (https://medium.com/@karlfriston/immunological-dark-matter-b48e20bba9ea). Some very bright folks have tried to account for this by devising devilishly complex modelling techniques (https://wellcomeopenresearch.org/articles/5-89; if you don’t have a degree in advanced math, I wouldn’t bother clicking on this link). For those of us who don’t, I suggest two rules of thumb: First, short-term predictions are less unreliable than longer term ones. Second, there is a lot of uncertainty underlying any prediction; so it pays to look at not just the predicted number, but also at the range above and below it.