Stanford study estimates 48,000+ infected in Santa Clara County - Printable Version
+- The CardBoard (https://thecardboard.org/board)
+-- Forum: Emergency (https://thecardboard.org/board/forum-11.html)
+--- Forum: Covid-19 (https://thecardboard.org/board/forum-12.html)
+--- Thread: Stanford study estimates 48,000+ infected in Santa Clara County (/thread-19445.html)
+- The CardBoard (https://thecardboard.org/board)
+-- Forum: Emergency (https://thecardboard.org/board/forum-11.html)
+--- Forum: Covid-19 (https://thecardboard.org/board/forum-12.html)
+--- Thread: Stanford study estimates 48,000+ infected in Santa Clara County (/thread-19445.html)
RE: Stanford study estimates 48,000+ infected in Santa Clara County - OutsiderFan - 04-26-2020
Valuable resources and time are being wasted on these antibody tests. Not only are they not accurate, we don't even know yet if there is such thing as immunity after infection.
The only thing that makes sense is ramping up the RNA detection tests that identify current infections, making them yield results ASAP, and to stop farting around with testing only suspected positive people. We need to use the testing capacity we do have to do random population sampling so we can build models around what we project is happening in the entire population, infected to not. The more localized the better.
RE: Stanford study estimates 48,000+ infected in Santa Clara County - dabigv13 - 04-26-2020
I agree with BC that we need to be testing these antibodies with much larger samples of negatives. They are widely available so should be feasible.
I disagree with OF that these are pointless. One of the tricky things is that early in an epidemic is when you need the very best antibodies. In a later stage when many people have the disease, a test with 97% specificity is still a very useful one. And there's great reason to believe you would have some degree of immunity for some indeterminate time, though the exact nature of this immunity is still unknown.
RE: Stanford study estimates 48,000+ infected in Santa Clara County - burger - 04-26-2020
(04-26-2020, 09:14 AM)dabigv13 Wrote: Important work being done to evaluate some of the antibody tests on the market by the below group.
https://covidtestingproject.org/
They've tested 10 different antibody tests so far, including the one imported by Premier, which was the test used in the Santa Clara and LA studies. In testing the specificity of the test, they checked 108 samples from 2018, before Covid19 existed, and found 3 positives, for a false positive rate of 2.8%, much higher than the 0.5% the Santa Clara paper indicates (2 positives out of 371, though this was by the manufacturer; they also validated with 30 samples at Stanford and had no false positives).
So essentially their entire findings could be explained by the false positives of their test, and their study doesn't meaningfully contribute to our knowledge of covid prevalence in SCC (or LA), though a prevalence of 1-4% remains plausible.
There's a good summary of this project here: https://www.nytimes.com/2020/04/24/health/coronavirus-antibody-tests.html
They condensed years of work into a few weeks. Pretty amazing effort.
RE: Stanford study estimates 48,000+ infected in Santa Clara County - Goose - 04-26-2020
(04-26-2020, 09:42 AM)OutsiderFan Wrote: The only thing that makes sense is ramping up the RNA detection tests that identify current infections, making them yield results ASAP, and to stop farting around with testing only suspected positive people. We need to use the testing capacity we do have to do random population sampling so we can build models around what we project is happening in the entire population, infected to not. The more localized the better.OF, with respect, I think you are wrong. First of all, when you say "The more localized the better", what do you mean? If you test some locales different than others, it isn't random. Second, the worst hit county (percentage wise) in the bay area is San Francisco. They have a current measured infection rate of 0.15% Let's assume that is low by a factor of 10. If you test the entire 883,395 people in San Francisco instantaneously and simultaneously you would expect to find about 13,000 infected people. But, even under the most optimistic scenario, you can't do 883,395 people in an instant. Look at Iceland, the country who has done the most "random" testing. They have been testing their population of 364,000 since January and got through 10% of it on April 19th. Since most of the "mild" cases of the disease come and go in three weeks or so, several generations of the disease would have come and gone while they tested 10% of the population. Some of that testing is "randomly" selected, but most is not. The incidence of this disease is small enough that gathering co-temporal statistics about it by random sampling is basically impossible, and what you do find will of necessity be retrospective.
The contact tracing and tracking approach OTOH involves deploying your testing capability among the people that are most likely to benefit from it. Those are people with a significantly higher probability of disease. Surveillance testing is also targeted at "suspicious" or probable outbreaks. The data derived from these tests will be timely enough to take targeted steps (quarantine) that will (hopefully) control the disease. That is a big reason this approach is the goal of just about everyone. Korea, HK and Taiwan are making it work pretty well. Singapore is have problems right now but still hasn't been forced to completely "shut down" like we have.
To be clear, I understand you aren't recommending testing everybody. However, to "know" the actual infection rate is 0.16% instead of 0.15% with any degree of confidence, you are going to have test a whole lot of people, 99.8% who will be negative. If it takes you three weeks to do so, the infection rate at the end of three weeks could now be a lot higher or lower than what it was when you started because it changes (in theory) exponentially. So what have you actually accomplished?
RE: Stanford study estimates 48,000+ infected in Santa Clara County - OutsiderFan - 04-26-2020
I'd rather be wrong and learn from my mistakes than never test my ideas, so I don't mind the criticism or take any umbrage to it.
When I say localized sampling, here's what I was thinking. Say each county gets enough tests to sample 1% of its population every month. They each develop a random sampling method and then when results return, the survey data and doctor-reported data can be given to the epidemiologists and statistical experts to map expected outbreak levels, not past outbreak levels as is done now.
Maybe I'm wrong, but I just don't see the value in identifying cases presenting symptoms when the real issue is stopping the spread among those without symptoms.
On a related note, if the saliva test can be produced to give immediate results and be scaled, it would be a game changer:
[tweet]https://twitter.com/awyllie13/status/1252995984189034497[/tweet]
RE: Stanford study estimates 48,000+ infected in Santa Clara County - Goose - 04-26-2020
(04-26-2020, 12:47 PM)OutsiderFan Wrote: I'd rather be wrong and learn from my mistakes than never test my ideas, so I don't mind the criticism or take any umbrage to it.
That is such a rational and balanced attitude, are you sure you belong on a sports board :-)? The good thing here is that we can do "thought experiments" here and it costs nothing.
Quote:When I say localized sampling, here's what I was thinking. Say each county gets enough tests to sample 1% of its population every month. They each develop a random sampling method and then when results return, the survey data and doctor-reported data can be given to the epidemiologists and statistical experts to map expected outbreak levels, not past outbreak levels as is done now.The identification of cases presenting symptoms is most important for several reasons. First, as can be seen by the negative rate of the testing being done under the current rules, 89% of the people presenting a reason to be tested are not at present shedding enough virus to be detected. We can say they don't have active disease. They might tomorrow, but for now they are not contagious and can just go home.
Maybe I'm wrong, but I just don't see the value in identifying cases presenting symptoms when the real issue is stopping the spread among those without symptoms.
The cases that do test positive can spread the disease and must be quarantined. Furthermore, it is certain they were infected by someone else and possible they have in turn infected others. Their contacts need to be interviewed for sure and tested if indicated. These contacts have a greatly increased probability of having the disease over the general public. If the contacts are tested, some presently asymptomatic (or very minor symptomatic) spreaders will be detected. These asymptomatic spreaders will therefore be quarantined, and their contacts will be examined. Using this approach, even multi-level asymptomatic spreaders will be detected (although not all of them). Further, contacts that test negative will also be quarantined, because it is probable that some of them will become ill later.
It is certainly true with this virus that people often are contagious for several days before they show symptoms. It is undoubtedly true that some number of people get the virus, become contagious, and recover from the virus without exhibiting noticeable symptoms. How common this is is being hotly debated at the moment. Multipliers between 2 and 10 have be suggested, but we just don't know. However, some of the people that these asymptomatic persons (to what extent they exist) have infected WILL be symptomatic, the asymptomatic persons will be a contact and therefore will be quarantined. They also will get tested. If they don't test positive, they are over the disease and whatever damage they did is in the past. We won't find the all this way, but we will find a lot of them.
Obviously, earlier detection of asymptomatic spreaders would be better if possible. One could look at "random" testing to do that. Everybody on this board is undoubtedly aware of the issues regarding true "randomness", but let's put those aside for a moment. If we wanted to test 1% of SF County, we would need 833,000 * 0.01 = 8,330 tests. To make it meaningful, we would like to get all the random samples in a day. I don't think you could do that. In a week, probably you could. SF county has done a total of 15,110 tests since the outbreak started, so doing these 8,330 tests is a big leap of faith, but let's say you could do it, and let's say that we can ignore the change in parameters over the week it took to take the samples. In the 8,330 tests we would expect to find about 0.15% symptomatic positives, or 12.5 people, on average. These people would get found anyway because they will report with symptoms. IF the asymptomatic/mild symptom level is 10 times the symptomatic value, we will have 125 such people, on average. Sounds like a win, until you realize there are 12,500 such people in the population, you have found 1% of them, and you can't do this again for 30 days. That is probably 1.5 to 2 "cycles" of the disease.
Can we use these numbers to make a "model". Eventually, possibly. One very serious problem is we know nothing about the probability distribution of our measurements. We are also not sampling a stationary process. Each month, the "state of the disease" will be different. It is difficult for me to envision how these numbers could be used in the short term. It is also not obvious that what happens in one county is directly transferable to another county. One obvious problem with the "thought experiment" outlined above is that I have assumed there is a fixed ratio between symptomatic and asymptomatic people who test positive. There may not be. It also could be a variable. In any case, if there is a fixed correlation between the two numbers, measuring symptomatic cases as we do now would allow us to infer the number of asymptomatic ones. Random sampling would eventually tell us how well we could do this, but it will take much more data than we can get in the short term.
My main thought is that using the symptomatic patients to lead us to the asymptomatic spreaders will probably identify more such people faster than random testing. Neither "net" is prefect, but the contact tracing approach will let us expend our tests in a more "target rich" environment.
Thought welcome!
RE: Stanford study estimates 48,000+ infected in Santa Clara County - M T - 04-26-2020
(04-26-2020, 09:14 AM)dabigv13 Wrote: ...much higher than the 0.5% the Santa Clara paper indicates (2 positives out of 371, though this was by the manufacturer; they also validated with 30 samples at Stanford and had no false positives).
I will repeat what I said earlier... one of the failures of the that paper and its "2 positives out of 371" is that they screwed that up too. The paper is a little convoluted. It appears they claim the manufacturer indicates 2 positives out of 371 for IgM and (unstated, other than "100%") 0 positives for IgG. The manufacturer's site (dated 2/27/2020, and as archived on April 22) indicates 3 positives out of 371 for IgM and 2 positives out of 371 for IgG. The study's definition of positive was that either antibody was found ("The total number of positive cases by either IgG or IgM in our unadjusted sample was 50"). The paper didn't take that into account (using 100% for IgM).
Updating with CovidTestingProject.org numbers
IgM: Mfgr: 3/371 Stanford 0/30 CTP.org 2/108 Total: 5/509 99.018%
IgG: Mfgr: 2/371 Stanford 0/30 CTP.org 1/108 Total 3/509 99.411%
If you consider the two tests as independent, that suggests a FP rate of 1.57%.
On 3330 tests * 1.57% FP rate = 52 false positives.
Bingo! Practically right on what they measured.
One of the problems of proving what you expected is that you likely will revisit any calculation that isn't what you expect but not necessarily all the ones that give the "right" answer. Clearly they didn't review this calculation. But why would they, since it gives the number they expect?
Just like their testing on only 30 samples on their own. If they had expected a 0.5% FP rate, 30 seems too low to measure it as anything but 0. (Indeed, I wonder if they had gotten 1 false positive in their 30, maybe that would have messed up their numbers enough that they would have tested more.)
RE: Stanford study estimates 48,000+ infected in Santa Clara County - BostonCard - 04-26-2020
(04-26-2020, 04:21 PM)M T Wrote:(04-26-2020, 09:14 AM)dabigv13 Wrote: ...much higher than the 0.5% the Santa Clara paper indicates (2 positives out of 371, though this was by the manufacturer; they also validated with 30 samples at Stanford and had no false positives).
I will repeat what I said earlier... one of the failures of the that paper and its "2 positives out of 371" is that they screwed that up too. The paper is a little convoluted. It appears they claim the manufacturer indicates 2 positives out of 371 for IgM and (unstated, other than "100%") 0 positives for IgG. The manufacturer's site (dated 2/27/2020, and as archived on April 22) indicates 3 positives out of 371 for IgM and 2 positives out of 371 for IgG. The study's definition of positive was that either antibody was found ("The total number of positive cases by either IgG or IgM in our unadjusted sample was 50"). The paper didn't take that into account (using 100% for IgM).
Updating with CovidTestingProject.org numbers
IgM: Mfgr: 3/371 Stanford 0/30 CTP.org 2/108 Total: 5/509 99.018%
IgG: Mfgr: 2/371 Stanford 0/30 CTP.org 1/108 Total 3/509 99.411%
If you consider the two tests as independent, that suggests a FP rate of 1.57%.
On 3330 tests * 1.57% FP rate = 52 false positives.
Bingo! Practically right on what they measured.
One of the problems of proving what you expected is that you likely will revisit any calculation that isn't what you expect but not necessarily all the ones that give the "right" answer. Clearly they didn't review this calculation. But why would they, since it gives the number they expect?
Just like their testing on only 30 samples on their own. If they had expected a 0.5% FP rate, 30 seems too low to measure it as anything but 0. (Indeed, I wonder if they had gotten 1 false positive in their 30, maybe that would have messed up their numbers enough that they would have tested more.)
You can't consider the two tests independent, though.
BC
RE: Stanford study estimates 48,000+ infected in Santa Clara County - BostonCard - 04-26-2020
One add... there are two main types of serology testing, and they could be used in a complementary fashion to answer the question.
Most of the serology testing that we've seen is an ELISA: ELISA, or enzyme-linked immunosorbent assay, is a screening test used to detect the presence and concentration of specific antibodies that bind to a viral protein. If you need something more specific, then you can do a microneutralization assay which is a highly specific confirmatory test used to measure neutralizing antibodies, or antibodies that can neutralize virus. This method is considered a gold standard for detection of specific antibodies in serum samples. However, compared with the ELISA, the microneutralization assay is labor-intensive and time-consuming, requiring at least 5 days before results are available.
So if you really wanted answers (especially if you are trying to determine whether people can go to work or if you are not sure if your prevalence study is accounted for by false positives) is to screen everyone via ELISA, and then confirm positive tests via the microneutralization assay.
BC
RE: Stanford study estimates 48,000+ infected in Santa Clara County - M T - 04-26-2020
(04-26-2020, 05:14 PM)BostonCard Wrote: You can't consider the two tests independent, though.
Thanks for the correction. I will happily defer to someone who can produce the appropriate statistics.
In the 2+1 false positives at CovidTestingProject.org were on different samples, so Stanford would have judged 3 positives.
For the manufacturer's samples, Stanford would have judged 3 to 5 samples positive, since we don't know whether those 3 IgM + 2 IgG samples were on 3, 4, or 5 different samples.
Minimum FP rate: 3/371 + 0/30 + 3/108 = 1.18% (3330 + 1.18% = 39.3)
Maximum FP rate: 5/371 + 0/30 + 3/108 = 1.57% (3330 + 1.57% = 52)
(Note that I don't know whether Stanford had access to CovidTestingProject.org. They presumably did have access to the 3 to 5 false positives by the manufacturer, but used 2.)
I wonder why Stanford didn't reveal the number of results that were positive for IgG only, IgM only, and IgG + IgM.
Indeed, with only 50 positives, and since we know they knew how to contact those with positive results, I wish they had run an independent set of the virus antibody tests on these individuals. Didn't Stanford claim to be creating a gold standard antibody test? They could clear this up quickly if they showed, say, 45 of their 50 positive individuals were positive on each of multiple antibody tests.
If the manufacturer were to indicate that its 5 positives were all on different samples (not that Stanford necessarily would have known these numbers), can the Stanford study actually be showing that there is a near 0% incidence of COVID-19 in SCC? (modulo the comments about self-selection, etc.) Presumably, it would show some upper bound on the infection rate rather than just showing 0%.
I wonder if Stanford warned these individuals of the (low, medium, high) likelihood that their positive test could be a false positive. I expect they warned that having antibodies didn't necessarily mean immunity. I also kinda wonder why someone who tested positive hasn't identified themselves to the media and talked about the process and what it means to them.
RE: Stanford study estimates 48,000+ infected in Santa Clara County - 2006alum - 04-30-2020
They've quietly updated the Santa Clara preprint. Unsurprisingly, the revised estimates are much less headline-grabbing:
[tweet]https://twitter.com/spence_jeffrey_/status/1256008621017554944?s=20[/tweet]
Strangely, the updated preprint still does not address the fact that there were "rogue" recruitment methods that may have influenced the participation pool beyond the parameters envisioned.
RE: Stanford study estimates 48,000+ infected in Santa Clara County - burger - 04-30-2020
(04-30-2020, 06:01 PM)2006alum Wrote: They've quietly updated the Santa Clara preprint. Unsurprisingly, the revised estimates are much less headline-grabbing:
[tweet]https://twitter.com/spence_jeffrey_/status/1256008621017554944?s=20[/tweet]
Strangely, the updated preprint still does not address the fact that there were "rogue" recruitment methods that may have influenced the participation pool beyond the parameters envisioned.
Of course they didn't mention the recruitment--admitting that the sampling was biased would be tantamount to admitting that the results are crap, which is the case.
I haven't looked at the new preprint, but that twitter thread strongly implies that, ignoring the sampling issue, their confidence intervals are too conservative. Without going into details, that sort of mistake is common in this type of analysis.