Confirmation of Reporting Problems - Printable Version
+- The CardBoard (https://thecardboard.org/board)
+-- Forum: Emergency (https://thecardboard.org/board/forum-11.html)
+--- Forum: Covid-19 (https://thecardboard.org/board/forum-12.html)
+--- Thread: Confirmation of Reporting Problems (/thread-20206.html)
+- The CardBoard (https://thecardboard.org/board)
+-- Forum: Emergency (https://thecardboard.org/board/forum-11.html)
+--- Forum: Covid-19 (https://thecardboard.org/board/forum-12.html)
+--- Thread: Confirmation of Reporting Problems (/thread-20206.html)
Pages:
1
2
RE: Confirmation of Reporting Problems - teejers1 - 08-06-2020
(08-06-2020, 11:17 AM)Genuine Realist Wrote:(08-06-2020, 09:41 AM)teejers1 Wrote:Hospitalizations rose, then leveled off. FWIW, the number dropped dramatically yesterday.(08-06-2020, 08:32 AM)Genuine Realist Wrote:(08-05-2020, 07:29 PM)OutsiderFan Wrote: OK, I’ll buy some impact to push down spread could be a result of national retailers requiring masks. However, given the rate of transmission, that doesn’t explain a 20k daily reduction.I don't see why any of this should be a surprise, as the public increasingly comes to understand and accept the norms of social distancing.
So we have SCC’s data fiasco that it says is under reporting cases based on faulty data collection and aggregation methods. Who knows how many counties and states are similarly plagued.
But here’s some evidence of deliberate attempts to ignore the amount of spread:
https://www.beaumontenterprise.com/news/article/Iowa-governor-orders-reduced-testing-at-Dubuque-15423753.php
The AP also reported overall testing capacity has dropped 3.5% over the last two weeks.
But you can’t hide the deaths, which continue to climb.
I suggestd long ago that the discipline of major retailers would be a major factor, and so it may be proving. Not only does such practice reduce places of contagion, but it also sets a powerful example for other modes of socialization.
As to rising deaths, I only follow Santa Clara County. It has been interesting over the past two months that death rates have NOT spiked, even though case counts rose significantly. Hospitalizations rose dramatically at the start, then leveled off at about 180. But they never rose in proportion to the case census.
I appreciate MT's posts (and his avid following of county and state data). But I also find it hard to believe that hospitalization numbers would suffer from whatever glitch is apparently infecting (pun intended) the count on number of tests, positive test rate, etc. Those strike me as completely different sets of numbers (and magnitudes of same) that would be reported differently. Hospitals know how many patients are in beds (acute and ICU), what % of those in those beds are Covid patients, and number of excess beds available. I find it hard to believe that any mix up in testing data would cripple hospitalization numbers.
As to GR's point on SCC hospitalization spiking, it was separately reported that patients from Imperial County and San Quentin prison were sent to hospitals in the Bay Area. The number was never disclosed (nor location of where patients ended up), which is mind-boggling. But the point is that the "spike" in SCC hospitalization could be attributable to these "outside county" numbers. It's astounding to me that this data is not being shared.
But the percentage of those hospitalized attributable to Covid has been in a pretty consistent range, IIRC (5-8%).
Regardless, I think it's worth knowing what number of Covid patients treated in a particular hospital were transferred in from outside the county (at least for abnormal transfers like the Imperial County and San Quentin transfers). Then you may aggregate those to get county information. If you believe that re-opening should be gauged to ability of local community to provide hospital care to those who need it, that kind of information is pretty important.
Of course, there is no indication that the decision-makers in Sacramento or locally are even factoring in that information into their decisions. Why the media does not press them on this is another mind-boggler (at least from here).
RE: Confirmation of Reporting Problems - g1313 - 08-07-2020
(08-05-2020, 08:47 PM)Goose Wrote: We all know that data doesn't "get stuck" in a software system. It is worrisome when a public official says something like that. Either they are trying to dumb it down for the public or they are a pointy-haired boss type repeating what they were told. If the latter, yikes!
I'm sure "getting stuck" is not the technical term, but it is certainly possible that they are having some sort of ETL issue, which data engineers do refer to as "getting stuck."
RE: Confirmation of Reporting Problems - BostonCard - 08-07-2020
Here's a bit more about what happened.
https://deadline.com/2020/08/multiple-errors-by-newsoms-team-caused-massive-blockage-in-california-coronavirus-reporting-system-250000-300000-records-still-delayed-1203007615/
Quote:Ghaly said a July 25 server outage prevented lab records from reaching the state system, which has the appropriately ironic name of CalREDIE. At that time, a workaround was created to allow records to flow in more quickly.
That workaround was supposed to be temporary. But someone somewhere forgot to switch the system back. That mistake caused further data delays and created a backlog. Ghaly said the workaround had since been reversed.
On top of that, the state failed to renew a certification with an intermediary that feeds results from labs like Quest directly into CalREDIE. Because of that bureaucratic error, some test results from Quest were held back between July 31-August 4. Ghaly said the certificate has now been renewed.
BC
RE: Confirmation of Reporting Problems - Goose - 08-07-2020
(08-07-2020, 04:48 PM)g1313 Wrote: I'm sure "getting stuck" is not the technical term, but it is certainly possible that they are having some sort of ETL issue, which data engineers do refer to as "getting stuck."
(08-07-2020, 04:52 PM)BostonCard Wrote: Here's a bit more about what happened.That they had a problem is bad. The fact that that problem wasn't throwing errors like crazy is hard to believe. The idea that they could have such a problem for an hour without somebody knowing about it and reporting it is to me completely unbelievable. This later condition is also completely unacceptable. The data from Quest was being held up because of (presumably a security) certificate that wasn't up to date and nobody noticed?
https://deadline.com/2020/08/multiple-errors-by-newsoms-team-caused-massive-blockage-in-california-coronavirus-reporting-system-250000-300000-records-still-delayed-1203007615/
Quote:Ghaly said a July 25 server outage prevented lab records from reaching the state system, which has the appropriately ironic name of CalREDIE. At that time, a workaround was created to allow records to flow in more quickly.
That workaround was supposed to be temporary. But someone somewhere forgot to switch the system back. That mistake caused further data delays and created a backlog. Ghaly said the workaround had since been reversed.
On top of that, the state failed to renew a certification with an intermediary that feeds results from labs like Quest directly into CalREDIE. Because of that bureaucratic error, some test results from Quest were held back between July 31-August 4. Ghaly said the certificate has now been renewed.
BC
The director of HHS isn't a data base guy. He isn't going to go fix the software. However, his people responded, or rather failed to respond, in a totally incompetent way. He now tells us the system as designed is not capable of handling the task, and that they are going to develop a new software system. He is figuring this out now, after seeing the Italy data volume and the New York data volume. Riiiiiight.
Newsom didn't have the guts to go out there and make the statement. He is right there when things go well and nowhere to be found when they don't. Did I mention that the PUC is supposed to oversee PG&E? Firing the head of HHS right now may not solve anything, but at least it makes a statement. What will probably happen is some intern who works for HHS will be let go and they will say "See, we held someone accountable".
RE: Confirmation of Reporting Problems - CardinalSagehen - 08-07-2020
(08-07-2020, 07:35 PM)Goose Wrote:(08-07-2020, 04:48 PM)g1313 Wrote: I'm sure "getting stuck" is not the technical term, but it is certainly possible that they are having some sort of ETL issue, which data engineers do refer to as "getting stuck."(08-07-2020, 04:52 PM)BostonCard Wrote: Here's a bit more about what happened.That they had a problem is bad. The fact that that problem wasn't throwing errors like crazy is hard to believe. The idea that they could have such a problem for an hour without somebody knowing about it and reporting it is to me completely unbelievable. This later condition is also completely unacceptable. The data from Quest was being held up because of (presumably a security) certificate that wasn't up to date and nobody noticed?
https://deadline.com/2020/08/multiple-errors-by-newsoms-team-caused-massive-blockage-in-california-coronavirus-reporting-system-250000-300000-records-still-delayed-1203007615/
Quote:Ghaly said a July 25 server outage prevented lab records from reaching the state system, which has the appropriately ironic name of CalREDIE. At that time, a workaround was created to allow records to flow in more quickly.
That workaround was supposed to be temporary. But someone somewhere forgot to switch the system back. That mistake caused further data delays and created a backlog. Ghaly said the workaround had since been reversed.
On top of that, the state failed to renew a certification with an intermediary that feeds results from labs like Quest directly into CalREDIE. Because of that bureaucratic error, some test results from Quest were held back between July 31-August 4. Ghaly said the certificate has now been renewed.
BC
The director of HHS isn't a data base guy. He isn't going to go fix the software. However, his people responded, or rather failed to respond, in a totally incompetent way. He now tells us the system as designed is not capable of handling the task, and that they are going to develop a new software system. He is figuring this out now, after seeing the Italy data volume and the New York data volume. Riiiiiight.
Newsom didn't have the guts to go out there and make the statement. He is right there when things go well and nowhere to be found when they don't. Did I mention that the PUC is supposed to oversee PG&E? Firing the head of HHS right now may not solve anything, but at least it makes a statement. What will probably happen is some intern who works for HHS will be let go and they will say "See, we held someone accountable".
I couldn't agree more. I do sympathize a bit with the current owners of these systems, having been exposed to state information systems and the organizations that run them early in my career, back in the mid 90s. These systems are ancient and full of technical debt. Resources and talent are scarce. I could see my way to excusing the fact that a bug occurred and that it did not cause technical alerts or error logs.
But I cannot excuse ANY of the following:
a) Not having cross-checks and compensating controls in place when the workaround went in place on July 25. Everyone should have been on high alert and scrutinizing the data after that.
b) Not questioning the reported data for the last couple weeks, when even casual observers on this board could clearly see that the trends defied reason.
c) Learning of the magnitude of the problem on Monday - which is already ridiculous, given (a) and (b) above - and only having a fix by Saturday or Sunday (if that actually comes through...)
d) Not having just one error, but multiple errors (i.e., expired security certificate). Who knows how many more errors are lurking?
Now we hear from the Mayor of Los Angeles what we probably all wondered:
Quote:“We don’t believe that it’s something that just happened in the last two or three weeks,” he said. “The state has told us that these reporting problems may have been throughout.”
I don't see how we can take the case count metrics seriously going forward - the system is completely FUBAR'ed, and aside from that, the actual testing delays are horrible, too. If Governor Newsom truly believes that "data is foundational", he should drop this already flawed metric from the watch-list criteria. Hospitalizations is probably the closest thing we have to a reliable leading indicator.
I'd also be on-board with frequent, statistically valid samples of public employees in each county as a better indicator of prevalence than what we have now. If you just take a small number of people, you can return the results much more quickly. It really doesn't help anyone to receive results that are more than a few days old. But it would help us to know in near real-time where the virus is surging and waning.
RE: Confirmation of Reporting Problems - OutsiderFan - 08-08-2020
How many other states are misreporting data? All of them? We know not everyone who gets infected is tested, so have only guesses as to how many in fact are. I agree the case data isn't that important, but if that is the method chosen to guide response, then it is problematic to have it be incorrect.
Still, it has baffled me since March why we don't to random population testing and instead require an asinine list of prerequisites before you can get tested. Unless, of course, you play pro sports.
We could be doing faster, cheaper less sensitive tests en masse. We could be doing random population sampling with high sensitivity, like is done to detect flu. Instead, we choose the third, dumbest, most expensive, and useless option available, high sensitivity, high cost, low volume testing that takes too long for results to matter. SMH.
RE: Confirmation of Reporting Problems - M T - 08-08-2020
By random testing, I presume you mean where the state forces a random set of people to take the test when they don't desire to and don't medically need to. I think I understand why the states don't do that.
-----
If all the numbers backed up for 2 weeks are going to be dumped into the states, put on your seat belts. This will be a bumpy ride.
I don't see how the state can deal with this for their monitoring list unless they keep the list frozen for 2 more weeks. They could switch to cases by date of sample, or average this deluge over 14 days (ie, increase each of the last 14 days by 1/14 of this bump.)
-----
SCC has marked all case data since July 16 as impacted by the data problems. That's a bit earlier than I noticed, but it could be. As of early Saturday, Aug 8, I see no signs that the numbers are flowing back to SCC properly yet. (That is, (Aug 6) Thursday's data (reported by SCC Friday) don't show the flow.) If the Qwest data was the issue and it was cleared before August 5 (as reported), then those numbers should be back.
The claim that the problems stemmed from a July 25th server outage seems possible. That means that by the end of July 24th, they had only processed a bit over half of the samples collected on July 20th for SCC. I suppose that is possible. The Cases by Date (of sample) dataset I downloaded on July 25 (meaning, data as of July 24; last date in data is July 23) showed this many cases (with 1 week earlier numbers in parentheses, and % of that week-earlier number processed this week)
July 16: 275 (July 9: 275, 100%)
July 17: 158 (234, 68%)
July 18: 62 (171, 36%)
July 19: 40 (118, 34%)
July 20: 57 (174, 33%)
July 21: 57 (319, 18%)
July 22: 23 (305, 8%)
July 23: 8 (275, 3%)
RE: Confirmation of Reporting Problems - Goose - 08-08-2020
(08-07-2020, 10:40 PM)CardinalSagehen Wrote: I couldn't agree more. I do sympathize a bit with the current owners of these systems, having been exposed to state information systems and the organizations that run them early in my career, back in the mid 90s. These systems are ancient and full of technical debt. Resources and talent are scarce. I could see my way to excusing the fact that a bug occurred and that it did not cause technical alerts or error logs.I myself have been "exposed" to DOD systems that are even older. However, CalREDIE isn't that ancient. It was brought online on January 4, 2010.
https://www.cdph.ca.gov/Programs/CID/DCDC/Pages/About-CalREDIE%E2%80%93System-Overview.aspx
At least it (probably) wasn't written in COBOL. My experience is that anything involving personal data has all kinds of audit trails as to who entered what and when. I still can't believe that nobody at CalREDIE knew that data was being rejected. It may be that for whatever reason that fact didn't flow up the chain, or something similar. I really hope we find out what actually happened, but I doubt we will.
Quote:But I cannot excuse ANY of the following:Totally agree. The level of incompetence is staggering, especially given that they have be at this for months.
a) Not having cross-checks and compensating controls in place when the workaround went in place on July 25. Everyone should have been on high alert and scrutinizing the data after that.
b) Not questioning the reported data for the last couple weeks, when even casual observers on this board could clearly see that the trends defied reason.
c) Learning of the magnitude of the problem on Monday - which is already ridiculous, given (a) and (b) above - and only having a fix by Saturday or Sunday (if that actually comes through...)
d) Not having just one error, but multiple errors (i.e., expired security certificate). Who knows how many more errors are lurking?
Quote:I'd also be on-board with frequent, statistically valid samples of public employees in each county as a better indicator of prevalence than what we have now. If you just take a small number of people, you can return the results much more quickly. It really doesn't help anyone to receive results that are more than a few days old. But it would help us to know in near real-time where the virus is surging and waning.Not going to happen, for good reasons. First, public employees aren't "typical". They don't represent the public at large or anything like it. Second, in Santa Clara County the prevalence of the disease is minuscule (relatively). Even at 200 positive tests per day, the number of people who would test positive in a sample of 1000 people is about 1.4(assuming a 14 day positive interval). If you get 0 in your 1000 tests, what does that tell you? If you get 2, what does that tell you? Almost nothing. Testing a smaller number of people will most of the time result in 0 or 1 positive. 1000 additional tests is not a drop in the bucket either, given our testing issues. On top of that, the minuscule number of cases are not randomly distributed. They exist in "pockets" of infection. Finding those pockets in a county of 2 million people is difficult/impossible. That is what contact tracing should do, but apparently we can't do that either.
RE: Confirmation of Reporting Problems - M T - 08-10-2020
To reiterate from the Deadline article posted above:
"On Friday, California’s top health official was sent out by his boss to face the music. ...
"Dr. Mark Ghaly, director of the state’s Health and Human Services Department, ...admitted that multiple errors ... had caused a backlog of 250,000-300,000 records in its case data reporting system.
"The state’s backlog of 250,000-300,000 records will be resolved in next 24-48 hours, he said. The test results will be added to the case totals for the days on which they would have originally reported."
"The director said that the errors only impacted daily case numbers..."
"“I became aware of the magnitude of the data backlog in the late afternoon on Monday [August 3],” said Ghaly"
48+ hours after the press conference, I see no sign of changes in numbers (testing, cases) published on Sunday by Santa Clara County. I don't see that the last few days numbers are showing signs that the problems have been fixed, and certainly no addition of purportedly 300K testing records (which would mean about 24K extra cases).
Moreover, I wonder whether 300K test records is accurate. Santa Clara county (<2M pop) reported 23K fewer tests in the first week of missing data, compared to the previous week, a drop of a bit more than 50%. The State of California is 20x the size of SCC. If SCC's testing levels & loss of results were representative of the state, that is about 460K missing tests per week for the state. There have been 3 weeks where data apparently has been missing. That would be 1.4M tests. If positive testing rates were about 8%, that's 112K cases. I actually expect this to be an overestimate of missing results because L.A. County's drop off seemed to be less, and the 2nd week of missing data had about 75% of the tests reported 2 weeks earlier. (But, I certainly hope the 250K-300K quoted wasn't number of positive results, but he did say it was "case data". If it is, the problem has been going on longer.)
Possibly, the backlog is being held back until Monday so the governor can announce it (and possibly replacements for the people in charge).
(I first reported the drop in testing & cases in SCC on the morning of July 31. The head of the Calif. HHS only heard about it on Aug. 3 ???)
I compared the numbers in the state's reporting of daily cases for SCC (last updated 8/8) with the numbers as originally announced by SCC as new daily cases. These numbers apparently are extracted from the real database at a different time of day, because they are +/- 30% from the originally announced numbers (and, typically, about 5% of each day's numbers are subsequently removed ("reclassified")). The state's numbers for the period in question track what SCC has previously announced. There have been no data added.
California's reporting of the number of tests for the state per week (Sunday to Saturday) and the number of new cases that week.
I'm also reporting the new cases/tests. Because of repeated tests, I don't know that these percentages match the "% positive" numbers.
I do not know if the "number of tests" means the number of tests for which results are known.
Week Tests New cases New cases/Tests
6/14-6/20 543,504 22,372 4.12%
6/21 650,006 34,115 5.25%
6/28 731,661 46,733 6.39%
7/05 751,012 49,120 6.54%
7/12 870,173 55,530 6.38%
7/19 882,040 62,121 7.04%
7/26-8/1 888,118 48,612 5.47%
There were hefty increases in testing numbers in June but the growth of these numbers dramatically slowed in mid July. (July 23 is the data of the document detailing priorities in testing.)
Can we believe the 5.47%? I don't know. We aren't told the nature of the missing cases. If they are randomly dropped, then yes. But, if they are (for instance), cases that were reported late, that's a potential bias (perhaps for more or fewer positives, I don't know).
I hope we find out that the decrease in cases/test is real. That would be good news.
UPDATE: (Mon 05:30) The state's County variance info page now says:
"Note: Recent issues found in the state's electronic laboratory reporting system that contributed to delays have been fixed.
Today's data includes some cases from prior days, and we continue to work through the backlog."
RE: Confirmation of Reporting Problems - M T - 08-10-2020
The logjam has broken and old tests are being reported. So, far just maybe 10% or so overall.
751 "new" cases reported today for Santa Clara County. The biggest daily increase in tests by date of sample before 8/4 was for 7/21 with just 369 additional tests reported. That day is still just at 4111 tests when the week before it was 6500 tests. (7/22 stands at 4243 tests when the week before was 9303.)
The days around 7/29 had negligible increases (50ish) and stand around 6500 +/- 250 tests. Around 75% of the week before this started.
The newly reported cases today were
Week (total tests) New tests New positive New % positive (Total % positive)
7/12-18: (45,895) 27 12 55.6% (6.0%)
7/19-25: (23,762) 1033 82 7.9% (7.2%)
7/26-8/1: (34,185) 312 93 29.8% (6.3%)
8/2-8/8: (26,299) 7876 691 8.8% (3.8%) (Incomplete as data is still coming in)
So, the initial wave of added tests have a higher %-positive, so it may be that the recent %-positive numbers were misleadingly low.
You may remember that I was concerned by the decrease in testing and the corresponding decrease in cases being found. There were two (or more) overlapping reasons for the decrease in testing. One was this failure to process tests and their results. The other was the change to stop processing (and stop testing) individuals without symptoms unless they claimed they were a close contact.
Any reason that restricted the detection of COVID-19 cases is a concern, whether it was a server error, lack of supplies, or government edict.
[By the way, I reported to the CDPH an error in their Data Definition for the Testing Data, and suggested a clarification to indicate which of 3 different values they may be reporting. I got back a form letter that included "That's a great question," when there was no question asked. Form letters that have gratuitous remarks like that really push my buttons.]
RE: Confirmation of Reporting Problems - Goose - 08-11-2020
Well, it happens to the best of us. The security certificate for the site that hosts the Cardboard has expired as of today. Fortunately, no testing data was harmed.
RE: Confirmation of Reporting Problems - BostonCard - 08-11-2020
(08-11-2020, 05:44 PM)Goose Wrote: Well, it happens to the best of us. The security certificate for the site that hosts the Cardboard has expired as of today. Fortunately, no testing data was harmed.
Yeah, now we're dealing with a 7-day backlog of posts.
BC
RE: Confirmation of Reporting Problems - CardinalSagehen - 08-11-2020
(08-11-2020, 05:44 PM)Goose Wrote: Well, it happens to the best of us. The security certificate for the site that hosts the Cardboard has expired as of today. Fortunately, no testing data was harmed.
Your response time was what - an hour or two? Days faster than HHS! You’re an unpaid volunteer, and this isn’t even life or death!
RE: Confirmation of Reporting Problems - Goose - 08-11-2020
(08-11-2020, 07:39 PM)CardinalSagehen Wrote:Better still, I am not even a mod here. I just got the chance to take a free shot, so I did =).(08-11-2020, 05:44 PM)Goose Wrote: Well, it happens to the best of us. The security certificate for the site that hosts the Cardboard has expired as of today. Fortunately, no testing data was harmed.
Your response time was what - an hour or two? Days faster than HHS! You’re an unpaid volunteer, and this isn’t even life or death!