No Minister

More Epidemiology Modelling Problems

“You fucked up. You trusted us.”

I was rather hard on Epidemiology Professor Neil Ferguson the other day because of his history of repeated false calls over two decades on various disease pandemics. But I also pointed out that his Imperial College model appears to have fed into models in other countries around the world, and this criticism is starting to mount up:

“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington. 

Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, home to several of the researchers who created the model, and who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”

Not that you’d know any of this from following the MSM, especially here in NZ, or Lefties busily masturbating over pictures of Jacinda, like Chris Trotter, (“SO FAR, SO BLOODY FANTASTIC!”),  Martyn Bradbury (“Thanks to our Government’s wisdom and leadership…”) or Frank Macskasy (“Wonder Woman”).

Amusingly they have all taken swipes at China – without noticing the sickening similarity between their local worship of our government and the standard boilerplate praise lavished on Xi Peng by China’s state media.

By contrast it’s been pleasing to see that none other than the Imperial College is starting to do what scientists are supposed to do: comparing models with reality, although they don’t seem to have got stuck into Ferguson’s to the same degree – yet:

According to a critique by researchers at the London School of Hygiene & Tropical Medicine and Imperial College London,  published this week in Annals of Internal Medicine, the IHME projections are based “on a statistical model with no epidemiologic basis.” 

“Statistical model” refers to putting U.S. data onto the graph of other countries’ Covid-19 deaths over time under the assumption that the U.S. epidemic will mimic that in those countries. But countries’ countermeasures differ significantly.

There are other technical reasons to distrust the IHME model, but the bottom line is that it misinforms national leaders.

This appearance of certainty is seductive when the world is desperate to know what lies ahead,” Britta Jewell of Imperial College and her colleagues wrote in their Annals paper. But the IHME model “rests on the likely incorrect assumption that effects of social distancing policies are the same everywhere.” Because U.S. policies are looser than those elsewhere, largely due to inconsistency between states, U.S. deaths could remain at higher levels longer than they did in China, in particular.

Still, those who live by the sword will likely die by it. Right now those who are in love with government lockdowns and mass house arrest are pointing to the models predictions of mass death early on as justification. Computer models! Run by Experts! But what the scenarios show are ranges so wide that the models could equally screwup in the opposite direction, which is not a comforting thought for governments trying to get out of the mess they have created.

Unfortunately it’s not just the IHME or Imperial College models that have got it so wrong. There’s quite the history of fail in epidemiology:

‘The crisis we face is unparalleled in modern times,” said the World Health Organization’s assistant director, while its director general proclaimed it “likely the greatest peacetime challenge that the United Nations and its agencies have ever faced.” This was based on a CDC computer model projection predicting as many as 1.4 million deaths from just two countries.  

So when did they say this about COVID-19? Trick question: It was actually about the Ebola virus in Liberia and Sierra Leone five years ago, and the ultimate death toll was under 8,000.

Oh dear. The article lists a few others, including bring Ferguson into the picture again.

For AIDS, the Public Health Service announced (without documenting) there would be 450,000 cases by the end of 1993, with 100,000 in that year alone. The media faithfully parroted it. There were 17,325 by the end of that year, with about 5,000 in 1993. SARS (2002-2003) was supposed to kill perhaps “millions,” based on analyses. It killed 744 before disappearing. 

Later, avian flu strain A/H5N1, “even in the best-case scenarios” was to “cause 2 (million) to 7 million deaths” worldwide. A British professor named Neil Ferguson scaled that up to 200 million. It killed 440.

As the article points out, if epidemic models were just haphazardly wrong, we would expect about half the time they would be too low. Instead, they’re almost universally vastly too high. It’s so bad that even the experts in charge of public health in the US have begun to express their doubts:

Then Fauci finally said it. “I’ve spent a lot of time on the models. They don’t tell you anything.”

The fuck? They told our governments to shit themselves and put us in lockdown, a phrase previously preserved for controlling prison riots.

A few days later CDC Director Robert Redfield also turned on the computer crystal balls. “Models are only as good as their assumptions, obviously there are a lot of unknowns about the virus” he said. “A model should never be used to assume that we have a number.”

The fuck? As I said before, the numbers are the whole point. Without numbers you don’t persuade anybody, politician or ordinary citizen, to do anything about this.

As far as I know there have only been a couple of analysis of the NZ models. Both are referred to in this post at Croaking Cassandra. The first looks at cost/benefit in terms of life expectancy gains and losses across the whole NZ population due to the disease and the lockdown.

The second analysis is more appropriate here in that the analysts actually ran the model used by the Otago Covid-19 Research Group (OCRG): it’s in the public domain: You can read the full report here, but the following are the key points to take away from it.

We found that OSRG’s model runs grossly overstated the number of deaths because they made an assumption about the critical tool in the Ministry’s arsenal. It was assumed that there would be no tracing and isolation of cases. This led to an explosion in the number of cases and deaths. 

The reporting of the range of deaths was also inflated by the simple expedient of excluding the model runs that produced low numbers. One of their six scenarios showed just seven deaths over a year.

That of course was not what the public saw from the MSM. They saw the following (Stuff quote):

Up to 14,000 New Zealanders could die if coronavirus spread is uncontrolled, according to new modelling by the University of Otago, Wellington.

But to be fair to the MSM they were not given the R0 ranges either, just the range of 8560 and 14400 deaths, and naturally because they’re the media the emphasised the BIG SCARY NUMBER on their headlines, since that’s all that most people read.

If the OCRG had so little confidence in the 1.5 estimate then they should have replaced it with a more plausible lower estimate, such as a R0 of 2, and then reported that number. Similarly the upper estimate could have been set at a high, but still reasonably possible 3. Instead the public is given a range of between 8560 and 14400 deaths, giving the misleading impression that there is a good deal of certainty around the estimates of high death numbers because the upper and lower bands are relatively close together.

My existence, while grotesque and incomprehensible to you, saves lives!”

Whether our Prime Minister and her Cabinet actually asked any probing questions about all this is not known, but given that she paraphrased “tens of thousands” of New Zealanders dying it’s a good bet that she simply followed the slots and grooves already prepared for her by the experts. And there’s no evidence that anybody else was handed the model to see what they could produce from it. As Tailrisk notes:

When we ran the Covidsim model we found credible paths that could reduce the pace of infections to sustainable levels. Deaths in the range of 50 – 500 over a year are more realistic numbers. 500 deaths is around average for normal seasonal flu [note that Gibson used 870 annual flu deaths].

Credible paths meaning they looked at what other nations had already been doing for two months. But they also did what good modellers are supposed to do; they tested the robustness and sensitivity of the model by altering key variables. One would hope the OCRG did this as well, but perhaps they just didn’t care to think about the life impacts of locking down the economy and simply adopted an autistic view of the disease problem.

Our benchmark model run shows 105 deaths after six months, and 157 after one year. This is broadly consistent with the experiences countries such as South Korea, Hong Kong, Taiwan, which have achieved a good measure of control over their epidemics without the need for harsh lockdowns.

Those are numbers to keep in mind over the winter. They also ran the Te Pūnaha Matatini Model – which produced and publicised even higher death tolls of 80,000 – and found similar problems, although it’s a more sophisticated model.

This is a case of getting out of the model what you put in. In our view, TPM did not use the best available information, and should have either: not released their report until it was updated (and should have told a different story); or released a heavily caveated paper, without any media fanfare.

One month ago the instant reaction to critics of the lockdown was the shroud-waving cry of “You care more for money than old people”, which has natually been very effective in public.

But of course it was never about lives vs. money: it was always about weighing lives vs. lives.

The economy is people’s lives and the next six months are going to be a terrible test of the theories that have paradoxically been pushed for years by groups such as Otago University healthcare experts; that economic decline and increases in poverty translate into more deaths as well as a miserable quality of life for those still living.

We’ll then see how well the macho attitude of “We saved old people’s lives” still stands in front of a weary, scared and miserable public.

See Also:
A spherical cow of uniform density in a frictionless vacuum.


Written by Tom Hunter

April 19, 2020 at 6:00 pm

%d bloggers like this: