The CardBoard
IHME not looking so great - Printable Version

+- The CardBoard (https://thecardboard.org/board)
+-- Forum: Emergency (https://thecardboard.org/board/forum-11.html)
+--- Forum: Covid-19 (https://thecardboard.org/board/forum-12.html)
+--- Thread: IHME not looking so great (/thread-19588.html)

Pages: 1 2

IHME not looking so great - OutsiderFan - 04-30-2020

Those who discounted to value of the IHME modeling appear to have been correct. It was last updated on April 29 according to its website.  

It predicted just 1,266 deaths on April 30. The numbers are in, and the U.S. actually had 2,390 deaths reported. Only off by over 1,000 dead people. We also know deaths are being underreported, strictly based on current mortality rates vs. historical ones. So even the model data isn't very good.

It's really hard to imagine not having at least 100,000 deaths at this point, but IHME predicts "only" 72,000 by August.  We're at over 62,000 on April 30. Hard to imagine just 10,000 in May considering we're adding 10,000+ deaths per week now. Even Spain and Italy are still averaging 300-400 deaths per day, we are way bigger, and I don't believe are implementing as many NPIs as they are. 

Meanwhile, Czech Republic reported 0 deaths on April 29, 0 deaths today, and only 77 cases over the two days. It's almost like mask wearing really helps reduce spread and viral load of those who do become infected, so they don't die from Covid-19.

RE: IHME not looking so great - burger - 04-30-2020

I (and a lot of others) have been saying since March that IHME is useless and should be ignored. 

(04-30-2020, 09:07 AM)OutsiderFan Wrote:  Those who discounted to value of the IHME modeling appear to have been correct. It was last updated on April 29 according to its website.  


Also, it's not a model.

RE: IHME not looking so great - OutsiderFan - 04-30-2020

(04-30-2020, 09:20 AM)burger Wrote:  I (and a lot of others) have been saying since March that IHME is useless and should be ignored. 

(04-30-2020, 09:07 AM)OutsiderFan Wrote:  Those who discounted to value of the IHME modeling appear to have been correct. It was last updated on April 29 according to its website.  


Also, it's not a model.

Yet the Federal Government, and likely the States, are using it as guidance. I actually believe the calls to open things up are in some meaningful measure being driven by the IHME projections. So not only is it incorrect, it might actually be setting us up for a worse result than we would have if it didn't exist.

That begs the question: If not the IHME, what do we have that provides the most accurate future projections? I have not once believed we are having waves. I have always believed there will be a rise to a plateau with NPIs in place, and then when the NPIs are relaxed, the growth will continue from the previous plateau. We have a long, long time before the virus gets through everyone without a vaccine that works.

RE: IHME not looking so great - magnus - 04-30-2020

(04-30-2020, 10:42 AM)OutsiderFan Wrote:  Yet the Federal Government, and likely the States, are using it as guidance. I actually believe the calls to open things up are in some meaningful measure being driven by the IHME projections. So not only is it incorrect, it might actually be setting us up for a worse result than we would have if it didn't exist.

If the States are using IHME projections to reopen, then they're likely not hitting the refresh button because many of the states that are reopening are projected/modeled/whatever to not be ready for another few weeks, given the updated projections as of even a week or two ago.

RE: IHME not looking so great - BostonCard - 04-30-2020

All models are wrong; some models are useful.  If your standard is that the model should predict the exact number of deaths on a given day, then I think you need to change your standards.

The death data is noisy.  For example, it was as low as 1156 deaths on April 26 before jumping up to 2470 on April 28.  That is why if you look at the shaded area, rather than the dotted line, you see that the model is predicting a range of deaths between about 400 and 4500.  Seeing a bit more than 2300 deaths is very much within the plausible range predicted by the model.

Broadly speaking, IHME is probably going to acquit itself pretty well.  It seems likely to me that we will be in the range of 100K - 240K deaths that it originally predicted, and while the first revised number of 60K deaths may have been too optimistic (and I'm still not sold on the most recently revised numbers saying that it will flatten out in the 75K range), this is about the precision one should expect for any sort of modeling exercise.

It certainly looks to be more accurate than the Imperial College model (predicting deaths in the millions) or a prominent economist who first predicted 500 deaths, then revised it to 5000 and then revised again to 50000 (while still falling short).

BC

RE: IHME not looking so great - 2006alum - 04-30-2020

For those who are interested, I strongly recommend you listen to the 538 Politics Model Talk episode in which they interview Dr. Chris Murray, who created the projections.

At around the 16 minute mark, what was very surprising to learn, and what I had never heard before, was that the White House's March 31st announcement that they were estimating 100k-240k deaths in the first wave was NOT based on the IHME estimates. Indeed, the source of that estimate has never been made public, so no one actually knows how the White House got those numbers. 

Here's more on why folks were initially skeptical of that forecast:
[tweet]https://twitter.com/saddamscribe/status/1245936961233096706?s=20[/tweet]
Quote:White House officials have refused to explain how they generated the figure — a death toll bigger than the United States suffered in the Vietnam War or the 9/11 terrorist attacks. They have not provided the underlying data so others can assess its reliability or provided long-term strategies to lower that death count.


Indeed, Murray noted around the 18-minute mark, that confusingly, the White House, after stating the 100k-240k deaths estimate, put a slide of the IHME estimate on the screen in discussing flattening the curve, which has led many in the media to confuse the White House's estimate as being based on the IHME forecast, which it has never been. Moreover, Murray also noted that the 100-240k forecast had no time indicator on the axis, so it was never made clear over what period of time that forecast was estimating the death count.  

FWIW, the initial estimated forecast from the IHME was 35k-150k with the mean of the distribution being 81k, which doesn't seem wildly off base. Murray explained later in the same podcast that the primary error in the initial forecast was due to estimates for hospital demand that were based on CDC data that was wildly overestimating the % of COVID-19 patients who need hospitalization.

After listening to this podcast, my suspicion, sadly, is that the White House cooked up a very high estimated range so that they could later claim to have come in under the estimates. And unless they release the source of their forecast, it's impossible to disprove that.

RE: IHME not looking so great - OutsiderFan - 04-30-2020

I have ZERO doubt the White House used a higher death projection than it thought would happen, so it could declare victory and say "see what a great job we did." Many Governors likely did the same thing.  

One thing you really can't do is say only x will die, and then have 2x or more of projected deaths occur. 

Still, it's pretty remarkable to see that even with the actual numbers not being as high as those initial projections, so many people still support the drastic NPIs. It would appear most Americans do understand the numbers and toll would be much higher without them.

RE: IHME not looking so great - burger - 04-30-2020

(04-30-2020, 10:59 AM)BostonCard Wrote:  All models are wrong; some models are useful.  If your standard is that the model should predict the exact number of deaths on a given day, then I think you need to change your standards.

The death data is noisy.  For example, it was as low as 1156 deaths on April 26 before jumping up to 2470 on April 28.  That is why if you look at the shaded area, rather than the dotted line, you see that the model is predicting a range of deaths between about 400 and 4500.  Seeing a bit more than 2300 deaths is very much within the plausible range predicted by the model.

Broadly speaking, IHME is probably going to acquit itself pretty well.  It seems likely to me that we will be in the range of 100K - 240K deaths that it originally predicted, and while the first revised number of 60K deaths may have been too optimistic (and I'm still not sold on the most recently revised numbers saying that it will flatten out in the 75K range), this is about the precision one should expect for any sort of modeling exercise.

It certainly looks to be more accurate than the Imperial College model (predicting deaths in the millions) or a prominent economist who first predicted 500 deaths, then revised it to 5000 and then revised again to 50000 (while still falling short).

BC

There has been at least one actual study (a preprint--which I can't find now) looking at IHME and finding it quite inaccurate at the state level.  I've seen a lot of twitter posts comparing IHME a month ago with actual values and also finding major discrepancies in both the timing and the size of peaks in deaths.

I believe the Imperial College model was a counterfactual of sorts, showing what might happen with no containment methods.  So it wasn't wrong in any meaningful sense.

There are several other models out there--I haven't spent any time looking at them. I think we're actually at a point where we don't need model guidance anymore.  We know that social distancing reduces the spread of the virus.  I doubt that any model is precise enough to tell us how much infections will increase if you, say, open beaches vs. opening restaurants or other specific policy options.  These will all be experiments of sorts with a lot of unknowns.  Any model "guidance" at this point will just give us false confidence that we know what is going to happen.  We don't.

RE: IHME not looking so great - BostonCard - 04-30-2020

A couple points.  Trump repeatedly minimized the severity of the disease up until the model was presented.  If the administration was guilty of sandbagging, Trump seems to have sabotaged those efforts.  More likely, I could see that the somewhat pessimistic internal assessment might have been aimed squarely at Trump, in an effort to get him to take the pandemic seriously.

Thanks for relaying the info 2006alum.  My vague recollection was that when the estimates were announced, it was quoted as a model "similar to" or "based on" the IMHE estimate.  My guess is that they were prepared internally by experts at the CDC.  I also don't think they were necessarily really sandbagging.  The estimates put forward by the administration were squarely within the mainstream of estimates.  They didn't use the Imperial College predictions, which they easily could have met.  For example, fivethirtyeight had a summary of all the estimates:

[Image: boice-wied.EXPERT-SURVEY.0403-3.png?w=575]

To my view, the White House estimate appears to be right smack in the middle of the prediction models, and overlaps most of them.

(link: https://fivethirtyeight.com/features/best-case-and-worst-case-coronavirus-forecasts-are-very-far-apart/)

BC

RE: IHME not looking so great - needle - 04-30-2020

This site claims to aggregate various covid-19 models. As far as I can tell, it's the project of a researcher at UMass-Amherst.

https://reichlab.io/covid19-forecast-hub/

+++

I found this pre-print from March by researchers working on the IHME model (I beg forgiveness if that's the incorrect term...), but can't swear to its authenticity.

Looking at the state data (very last page), the numbers mostly didn't/haven't matched up with what we know so far.

http://www.healthdata.org/sites/default/files/files/research_articles/2020/COVID-forecasting-03252020_4.pdf

RE: IHME not looking so great - 2006alum - 04-30-2020

(04-30-2020, 03:19 PM)needle Wrote:  I found this pre-print from March by researchers working on the IMHE model (I beg forgiveness if that's the incorrect term...), but can't swear to its authenticity.

Looking at the state data (very last page), the numbers mostly didn't/haven't matched up with what we know so far.

http://www.healthdata.org/sites/default/files/files/research_articles/2020/COVID-forecasting-03252020_4.pdf

Perhaps, but the initial projection still seems relatively in the ballpark: "We estimate 81,114 (95% UI 38,242 to 162,106) deaths in the United States from COVID-19 over the next 4 months." And many of the individual state estimates don't seem too far off.

One thing that's worth mentioning again is that they made their initial projections for hospital usage and ICU usage on the basis of CDC figures that were later shown to be wildly too high, and so all of their early estimates for hospital and ICU usage were far too high. But I think their deaths estimates have held up pretty well, all things considered.

RE: IHME not looking so great - magnus - 05-04-2020

New numbers coming out for the IHME this afternoon.  

This is from a CNN update (I don't know how to link those)

Quote:An influential coronavirus model often cited by the White House said in a press release that it plans to revise its projections to nearly 135,000 Covid-19 deaths in the United States, an increase that one of its researchers tied to relaxed social distancing and increased mobility.

The model, from the Institute for Health Metrics and Evaluation at the University of Washington, previously predicted 72,433 deaths as of Monday morning. A press release from IHME said the full set of new projections will be released later this afternoon.

That's 60K more than their latest estimates.  Remember, this is to the beginning of August which is later than the Wharton study recently mentioned.  Looks like we're going to be getting a lot of bad news projections this week.

RE: IHME not looking so great - OutsiderFan - 05-04-2020

This has been the greatest failure in the history of the United States. And it took only less than 3 months to "accomplish."

We've lost more American lives than the Vietnam War. We've added, is it $4 Trillion to the national debt? We still have 30 million people unemployed, and Main Street business have no money to spend.  It wouldn't be so bad if we shut things down for a couple months and actually saw we meaningfully knocked down the virus and spread vectors by paying this price.  We did neither.  

At least we didn't overwhelm the healthcare system in round 1, so all was not lost, but the return on the cost of our investment and sacrifice is totally and completely unacceptable. Has any administration official been fired, just to show the guy in charge finds it unacceptable?

RE: IHME not looking so great - stupac2 - 05-04-2020

(05-04-2020, 12:32 PM)OutsiderFan Wrote:  Has any administration official been fired, just to show the guy in charge finds it unacceptable?

I'll just leave this here: https://www.nytimes.com/2020/05/01/us/politics/trump-health-department-watchdog.html

RE: IHME not looking so great - OutsiderFan - 05-04-2020

(05-04-2020, 01:16 PM)stupac2 Wrote:  
(05-04-2020, 12:32 PM)OutsiderFan Wrote:  Has any administration official been fired, just to show the guy in charge finds it unacceptable?

I'll just leave this here: https://www.nytimes.com/2020/05/01/us/politics/trump-health-department-watchdog.html

OK, so the people who report the drunk drivers are fired. The drunk drivers are encouraged to continue driving drunk. I must have missed that day of Management 101.

RE: IHME not looking so great - Farm93 - 05-04-2020

The models all have one big problem. 

The subjects learn the projections and then make lifestyle adjustments because of those projections.

When the models were up around 240,000 and many people thought it was 15 going to 0 or just a bad season flu death total (~50,000) they were shocked to action.   As a result more governors issued stay-at-home, more citizens actually stopped.  

The original projections assumed 50% compliance, so as more people were motivated to comply it looked like the USA might land on the low end of projections.   POTUS was more ambitious and declared the virus death tally might be only 60,000.

However, that good news in the model updates motivated many Governors and citizens to ditch safer-at-home and social distancing.  Given many USA citizens will now stop social distancing the models need to be adjusted to higher death totals.

RE: IHME not looking so great - OutsiderFan - 05-04-2020

I like how the new state charts include distancing activity and mobility changes. Also like that it includes testing data.

RE: IHME not looking so great - fullmetal - 05-04-2020

This is one of the reasons why I was okay with higher death count models despite the critique that such models "stoked fear" in the populace.  Hey, at least fear motivates people into corrective action.  Complacency kills.

RE: IHME not looking so great - burger - 05-04-2020

(05-04-2020, 03:09 PM)OutsiderFan Wrote:  I like how the new state charts include distancing activity and mobility changes. Also like that it includes testing data.

The new models are good and bad.

Good: it's a real SEIR (susceptible, exposed, infected, recovered) model, not just a curve fit.  The model confidence intervals behave like projections are supposed to--narrow at first and wider as time goes on.

Bad: A lot of their results seem to flow from the projections in distancing behavior.  And they seem to keep distancing behavior static over time.  So, these results probably don't have much predictive value given that 1) people are already showing reductions in distancing and 2) governors are starting to open things up.

Also bad: They're including temperature effects in the models, which are still very much up for debate, and density effects, which are also very much up for debate.

This is a better model than the original in many ways, but I don't think these predictions are going to hold up.  Too many questionable assumptions.

RE: IHME not looking so great - Goose - 05-04-2020

(05-04-2020, 04:33 PM)burger Wrote:  This is a better model than the original in many ways, but I don't think these predictions are going to hold up.  Too many questionable assumptions.

Then why is it a "better" model? Do you think its predictions are going to be closer i.e. smaller "error" in an RMS sense or absolute error sense?