[CardinalSagehen]

I'm sure "getting stuck" is not the technical term, but it is certainly possible that they are having some sort of ETL issue, which data engineers do refer to as "getting stuck."

Here's a bit more about what happened.

https://deadline.com/2020/08/multiple-er...203007615/

Ghaly said a July 25 server outage prevented lab records from reaching the state system, which has the appropriately ironic name of CalREDIE. At that time, a workaround was created to allow records to flow in more quickly.

That workaround was supposed to be temporary. But someone somewhere forgot to switch the system back. That mistake caused further data delays and created a backlog. Ghaly said the workaround had since been reversed.

On top of that, the state failed to renew a certification with an intermediary that feeds results from labs like Quest directly into CalREDIE. Because of that bureaucratic error, some test results from Quest were held back between July 31-August 4. Ghaly said the certificate has now been renewed.

BC

That they had a problem is bad. The fact that that problem wasn't throwing errors like crazy is hard to believe. The idea that they could have such a problem for an hour without somebody knowing about it and reporting it is to me completely unbelievable. This later condition is also completely unacceptable. The data from Quest was being held up because of (presumably a security) certificate that wasn't up to date and nobody noticed?
The director of HHS isn't a data base guy. He isn't going to go fix the software. However, his people responded, or rather failed to respond, in a totally incompetent way. He now tells us the system as designed is not capable of handling the task, and that they are going to develop a new software system. He is figuring this out now, after seeing the Italy data volume and the New York data volume. Riiiiiight.

Newsom didn't have the guts to go out there and make the statement. He is right there when things go well and nowhere to be found when they don't. Did I mention that the PUC is supposed to oversee PG&E? Firing the head of HHS right now may not solve anything, but at least it makes a statement. What will probably happen is some intern who works for HHS will be let go and they will say "See, we held someone accountable".

I couldn't agree more.  I do sympathize a bit with the current owners of these systems, having been exposed to state information systems and the organizations that run them early in my career, back in the mid 90s.  These systems are ancient and full of technical debt.  Resources and talent are scarce.  I could see my way to excusing the fact that a bug occurred and that it did not cause technical alerts or error logs.

But I cannot excuse ANY of the following:
a) Not having cross-checks and compensating controls in place when the workaround went in place on July 25. Everyone should have been on high alert and scrutinizing the data after that.
b) Not questioning the reported data for the last couple weeks, when even casual observers on this board could clearly see that the trends defied reason.
c) Learning of the magnitude of the problem on Monday - which is already ridiculous, given (a) and (b) above - and only having a fix by Saturday or Sunday (if that actually comes through...)
d) Not having just one error, but multiple errors (i.e., expired security certificate).  Who knows how many more errors are lurking?  

Now we hear from the Mayor of Los Angeles what we probably all wondered:

“We don’t believe that it’s something that just happened in the last two or three weeks,” he said. “The state has told us that these reporting problems may have been throughout.”

I don't see how we can take the case count metrics seriously going forward - the system is completely FUBAR'ed, and aside from that, the actual testing delays are horrible, too.  If Governor Newsom truly believes that "data is foundational", he should drop this already flawed metric from the watch-list criteria.  Hospitalizations is probably the closest thing we have to a reliable leading indicator. 

I'd also be on-board with frequent, statistically valid samples of public employees in each county as a better indicator of prevalence than what we have now.  If you just take a small number of people, you can return the results much more quickly. It really doesn't help anyone to receive results that are more than a few days old. But it would help us to know in near real-time where the virus is surging and waning.

[CardinalSagehen]

I'm sure "getting stuck" is not the technical term, but it is certainly possible that they are having some sort of ETL issue, which data engineers do refer to as "getting stuck."

Here's a bit more about what happened.

https://deadline.com/2020/08/multiple-er...203007615/

Ghaly said a July 25 server outage prevented lab records from reaching the state system, which has the appropriately ironic name of CalREDIE. At that time, a workaround was created to allow records to flow in more quickly.

That workaround was supposed to be temporary. But someone somewhere forgot to switch the system back. That mistake caused further data delays and created a backlog. Ghaly said the workaround had since been reversed.

On top of that, the state failed to renew a certification with an intermediary that feeds results from labs like Quest directly into CalREDIE. Because of that bureaucratic error, some test results from Quest were held back between July 31-August 4. Ghaly said the certificate has now been renewed.

BC

That they had a problem is bad. The fact that that problem wasn't throwing errors like crazy is hard to believe. The idea that they could have such a problem for an hour without somebody knowing about it and reporting it is to me completely unbelievable. This later condition is also completely unacceptable. The data from Quest was being held up because of (presumably a security) certificate that wasn't up to date and nobody noticed?
The director of HHS isn't a data base guy. He isn't going to go fix the software. However, his people responded, or rather failed to respond, in a totally incompetent way. He now tells us the system as designed is not capable of handling the task, and that they are going to develop a new software system. He is figuring this out now, after seeing the Italy data volume and the New York data volume. Riiiiiight.

Newsom didn't have the guts to go out there and make the statement. He is right there when things go well and nowhere to be found when they don't. Did I mention that the PUC is supposed to oversee PG&E? Firing the head of HHS right now may not solve anything, but at least it makes a statement. What will probably happen is some intern who works for HHS will be let go and they will say "See, we held someone accountable".

I couldn't agree more.  I do sympathize a bit with the current owners of these systems, having been exposed to state information systems and the organizations that run them early in my career, back in the mid 90s.  These systems are ancient and full of technical debt.  Resources and talent are scarce.  I could see my way to excusing the fact that a bug occurred and that it did not cause technical alerts or error logs.

But I cannot excuse ANY of the following:
a) Not having cross-checks and compensating controls in place when the workaround went in place on July 25. Everyone should have been on high alert and scrutinizing the data after that.
b) Not questioning the reported data for the last couple weeks, when even casual observers on this board could clearly see that the trends defied reason.
c) Learning of the magnitude of the problem on Monday - which is already ridiculous, given (a) and (b) above - and only having a fix by Saturday or Sunday (if that actually comes through...)
d) Not having just one error, but multiple errors (i.e., expired security certificate).  Who knows how many more errors are lurking?  

Now we hear from the Mayor of Los Angeles what we probably all wondered:

“We don’t believe that it’s something that just happened in the last two or three weeks,” he said. “The state has told us that these reporting problems may have been throughout.”

I don't see how we can take the case count metrics seriously going forward - the system is completely FUBAR'ed, and aside from that, the actual testing delays are horrible, too.  If Governor Newsom truly believes that "data is foundational", he should drop this already flawed metric from the watch-list criteria.  Hospitalizations is probably the closest thing we have to a reliable leading indicator. 

I'd also be on-board with frequent, statistically valid samples of public employees in each county as a better indicator of prevalence than what we have now.  If you just take a small number of people, you can return the results much more quickly. It really doesn't help anyone to receive results that are more than a few days old. But it would help us to know in near real-time where the virus is surging and waning.