[BostonCard]

The authors of the study would hardly be the first ones to have a pet theory that their research supported.  In fact, almost all researchers will go into a research project guided by the belief that their study will show something hitherto unknown or incorrect.

Listen, I think the numbers reported are likely to be an overestimate of the true prevalence of prior exposure to COVID-19, just like the reported number of positive tests is an underestimate of the number of true positives.  You would never hang your hat on a single number, but over time as more and better studies are conducted, the data will converge on a better estimate.

I do agree that the authors probably should be a little more circumspect about their findings in the lay press, especially this early in the process.  But asking for a retraction of a pre-print paper just because it has some flaws is just not a standard that science does or should follow.

As to the bad faith argument, I'm with Goose here.  I think they just got caught up in the excitement of their own findings.  Ioannidis should know better, but even the greats are human.

BC

---

I just heard someone say the study has been retracted. Can’t find confirmation.

No evidence that it has been retracted from the preprint server:

https://www.medrxiv.org/content/10.1101/...20062463v1

BC

---

4) I don't think you understand how peer review is supposed to work.  

I am well-acquainted with the peer review process.

BC

[BostonCard]

The authors of the study would hardly be the first ones to have a pet theory that their research supported.  In fact, almost all researchers will go into a research project guided by the belief that their study will show something hitherto unknown or incorrect.

Listen, I think the numbers reported are likely to be an overestimate of the true prevalence of prior exposure to COVID-19, just like the reported number of positive tests is an underestimate of the number of true positives.  You would never hang your hat on a single number, but over time as more and better studies are conducted, the data will converge on a better estimate.

I do agree that the authors probably should be a little more circumspect about their findings in the lay press, especially this early in the process.  But asking for a retraction of a pre-print paper just because it has some flaws is just not a standard that science does or should follow.

As to the bad faith argument, I'm with Goose here.  I think they just got caught up in the excitement of their own findings.  Ioannidis should know better, but even the greats are human.

BC

---

I just heard someone say the study has been retracted. Can’t find confirmation.

No evidence that it has been retracted from the preprint server:

https://www.medrxiv.org/content/10.1101/...20062463v1

BC

---

4) I don't think you understand how peer review is supposed to work.  

I am well-acquainted with the peer review process.

BC

[burger]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

[burger]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

[stupac2]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

I think that BC generally wants to assume the best of people. I happen to agree with you that this is embarrassing, reflects very negatively on Stanford, and rises to the point where some form of professional misconduct investigation should happen, but it's possible to look at this and see acts of good faith, if stunning incompetence.

[stupac2]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

I think that BC generally wants to assume the best of people. I happen to agree with you that this is embarrassing, reflects very negatively on Stanford, and rises to the point where some form of professional misconduct investigation should happen, but it's possible to look at this and see acts of good faith, if stunning incompetence.

[Goose]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

Why the trash bin? Don't you believe in actually burning books you don't like? Sorry to be so aggressive here, but I must admit this approach frosts me more than a bit. I understand the criticism of the study. That is valid and good. Let everyone decide for themselves. However, ascribing evil motives and trying to remove the study from view by throwing it in the trash is not acceptable in a free society. Crap is published all the time. However, what is "crap" is often in the eye of the beholder. Criticize, debunk, question all you want. Suppress? Very different issue.

[Goose]

This isn't a "caught up in the excitement" problem.  This is junk science.  Biased sampling followed by improper analysis of already bad data.  This belongs in the trash bin.  I don't even understand why we're arguing about this crap.

Why the trash bin? Don't you believe in actually burning books you don't like? Sorry to be so aggressive here, but I must admit this approach frosts me more than a bit. I understand the criticism of the study. That is valid and good. Let everyone decide for themselves. However, ascribing evil motives and trying to remove the study from view by throwing it in the trash is not acceptable in a free society. Crap is published all the time. However, what is "crap" is often in the eye of the beholder. Criticize, debunk, question all you want. Suppress? Very different issue.

[BostonCard]

Also, it is possible that the concerns about selection bias and the test specificity are overblown, and, in fact, the numbers that are quoted in the article are broadly right.  For the record, I don't think so, but it is possible.

Take the concern about sensitivity.  Yes, based on the confidence interval of the sensitivity it is possible that all the positives are false positives.  But based on the point estimate of the sensitivity, that is not the case.  So, bottom line, I think the confidence interval that the paper cites is too narrow, but their point estimate is unaffected.  I believe that will need to be fixed during peer review and when this paper comes out, I'd say it is likely that it will feature broader confidence intervals that propagate the sensitivity uncertainty better.

The concern about selection bias is legitimate, but they acknowledge it in the paper.  Unless you can come up with a way to measure and adjust for it, the best they can do is what they have done; list it as a limitation.  You might speculate on its directionality, but there is no way to be sure of its magnitude.

Yesterday, I posted a link to an article about hydroxychloroquine that has also been making the rounds on news outlets.  Unfortunately, it showed that hydroxychloroquine doesn't do anything for patients with COVID-19.  It was a retrospective study, and suffers from the same bias (patients with more severe disease are more likely to be treated, a phenomenon known as confounding by indication).  I noted the bias when I pointed the paper and said I'd wait until higher quality papers would come out to make up my mind for certain. However, I also pointed out that it is unlikely that HCQ is a "game changer" because we would have seen it.  So, even though there is likely bias in the paper, I wouldn't demand its retraction.

Same here.  I will wait until true random sampling (or saturation testing) is performed before I make up my mind for certain on the prevalence and IFR.  But the study still moves the needle, and it is reasonable to think that the prevalence might be higher and the CFR might be lower than previously reported.

BC

[BostonCard]

Also, it is possible that the concerns about selection bias and the test specificity are overblown, and, in fact, the numbers that are quoted in the article are broadly right.  For the record, I don't think so, but it is possible.

Take the concern about sensitivity.  Yes, based on the confidence interval of the sensitivity it is possible that all the positives are false positives.  But based on the point estimate of the sensitivity, that is not the case.  So, bottom line, I think the confidence interval that the paper cites is too narrow, but their point estimate is unaffected.  I believe that will need to be fixed during peer review and when this paper comes out, I'd say it is likely that it will feature broader confidence intervals that propagate the sensitivity uncertainty better.

The concern about selection bias is legitimate, but they acknowledge it in the paper.  Unless you can come up with a way to measure and adjust for it, the best they can do is what they have done; list it as a limitation.  You might speculate on its directionality, but there is no way to be sure of its magnitude.

Yesterday, I posted a link to an article about hydroxychloroquine that has also been making the rounds on news outlets.  Unfortunately, it showed that hydroxychloroquine doesn't do anything for patients with COVID-19.  It was a retrospective study, and suffers from the same bias (patients with more severe disease are more likely to be treated, a phenomenon known as confounding by indication).  I noted the bias when I pointed the paper and said I'd wait until higher quality papers would come out to make up my mind for certain. However, I also pointed out that it is unlikely that HCQ is a "game changer" because we would have seen it.  So, even though there is likely bias in the paper, I wouldn't demand its retraction.

Same here.  I will wait until true random sampling (or saturation testing) is performed before I make up my mind for certain on the prevalence and IFR.  But the study still moves the needle, and it is reasonable to think that the prevalence might be higher and the CFR might be lower than previously reported.

BC

[Goose]

The only answer I can reach is that they are acting in bad faith. I'm still waiting for evidence to the contrary.

I have proposed an alternative. Stupidity. Incompetence. There more evidence of that there is evidence of bad faith. You provide evidence of error. You provide no evidence of motivation, i.e. bad faith. Asking others to meet a standard of proof you have not met makes no sense and is erecting a straw man. Most probably, the authors do not substantially agree with your analysis. That may make them wrong. It does not make them operating in "bad faith".

[Goose]

The only answer I can reach is that they are acting in bad faith. I'm still waiting for evidence to the contrary.

I have proposed an alternative. Stupidity. Incompetence. There more evidence of that there is evidence of bad faith. You provide evidence of error. You provide no evidence of motivation, i.e. bad faith. Asking others to meet a standard of proof you have not met makes no sense and is erecting a straw man. Most probably, the authors do not substantially agree with your analysis. That may make them wrong. It does not make them operating in "bad faith".

[dabigv13]

https://www.wired.com/story/new-covid-19...hey-right/

Learned some things from this article. Apparently one of the coauthors of the study is just some VC investor asshole who just wrote a WSJ oped about how the virus isn't so deadly. Sure some of the papers I've been on have had some dubious coauthors but this is beyond the pale. What figure did he contribute to in the paper?

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

Unclear if they are the same company as Hangzhou Alltest Biotech. They were the producers of the worst performing commercial test from the only data comparing commercial tests out there. Specificity in the 80s, woof.
https://www.medrxiv.org/content/10.1101/...l.pdf+html

[dabigv13]

https://www.wired.com/story/new-covid-19...hey-right/

Learned some things from this article. Apparently one of the coauthors of the study is just some VC investor asshole who just wrote a WSJ oped about how the virus isn't so deadly. Sure some of the papers I've been on have had some dubious coauthors but this is beyond the pale. What figure did he contribute to in the paper?

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

Unclear if they are the same company as Hangzhou Alltest Biotech. They were the producers of the worst performing commercial test from the only data comparing commercial tests out there. Specificity in the 80s, woof.
https://www.medrxiv.org/content/10.1101/...l.pdf+html

[2006alum]

https://www.wired.com/story/new-covid-19...hey-right/

Learned some things from this article. Apparently one of the coauthors of the study is just some VC investor asshole who just wrote a WSJ oped about how the virus isn't so deadly. Sure some of the papers I've been on have had some dubious coauthors but this is beyond the pale. What figure did he contribute to in the paper?

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

This is the bad faith I'm talking about. Three co-authors went on record a month ago in two different nationally-circulated op-eds saying the virus was overblown, it was no worse than the flu, and that social distancing was the wrong move. Were this study to confirm their prior, data-light hot take, it would burnish their reputation. They got out ahead of the science with a hot take, and then coincidentally, posted a study that purports to support their hot take even though there's a strong possibility it does no such thing, notwithstanding their topline takeaway that this is no worse than the flu. 

Speaking of bad faith, did anyone follow how (Law) Professor Richard Epstein -- he of the infamous "this will only cause 500 deaths" op-ed that was later revised to 5,000 deaths due to a "factoring error" -- was caught surreptitiously revising his entire op-ed to look like he'd originally claimed 5,000 (instead of 500) deaths and updated to 50,000 (instead of 5,000)? He got called out on it thanks to a careful reader and the waybackmachine:
[tweet]https://twitter.com/JohnPMacke/status/1251716101819584513?s=20[/tweet]

What do both have in common? They put their reputation on the line with a contrarian hot take that is proving every day to be less and less likely, and now they are pursuing questionable means to try to revive their position. I really hate to say this as someone who went to Stanford and cares a lot about academia, but "experts" like Epstein and Ioannidis are what cause non-experts to stop taking experts seriously. They don't speak with caution or caveats, but with bold, provocative flair bordering on hubris, and when they seem to be in the wrong, they double down rather than admit the strong possibility of their error.

And to be clear, my position was never that they should retract it. My position is that they should revise their topline summary to read, "could be widespread undetected community transmission, could be almost none." But that finding is clearly not the one they were motivated to find.

[2006alum]

https://www.wired.com/story/new-covid-19...hey-right/

Learned some things from this article. Apparently one of the coauthors of the study is just some VC investor asshole who just wrote a WSJ oped about how the virus isn't so deadly. Sure some of the papers I've been on have had some dubious coauthors but this is beyond the pale. What figure did he contribute to in the paper?

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

This is the bad faith I'm talking about. Three co-authors went on record a month ago in two different nationally-circulated op-eds saying the virus was overblown, it was no worse than the flu, and that social distancing was the wrong move. Were this study to confirm their prior, data-light hot take, it would burnish their reputation. They got out ahead of the science with a hot take, and then coincidentally, posted a study that purports to support their hot take even though there's a strong possibility it does no such thing, notwithstanding their topline takeaway that this is no worse than the flu. 

Speaking of bad faith, did anyone follow how (Law) Professor Richard Epstein -- he of the infamous "this will only cause 500 deaths" op-ed that was later revised to 5,000 deaths due to a "factoring error" -- was caught surreptitiously revising his entire op-ed to look like he'd originally claimed 5,000 (instead of 500) deaths and updated to 50,000 (instead of 5,000)? He got called out on it thanks to a careful reader and the waybackmachine:
[tweet]https://twitter.com/JohnPMacke/status/1251716101819584513?s=20[/tweet]

What do both have in common? They put their reputation on the line with a contrarian hot take that is proving every day to be less and less likely, and now they are pursuing questionable means to try to revive their position. I really hate to say this as someone who went to Stanford and cares a lot about academia, but "experts" like Epstein and Ioannidis are what cause non-experts to stop taking experts seriously. They don't speak with caution or caveats, but with bold, provocative flair bordering on hubris, and when they seem to be in the wrong, they double down rather than admit the strong possibility of their error.

And to be clear, my position was never that they should retract it. My position is that they should revise their topline summary to read, "could be widespread undetected community transmission, could be almost none." But that finding is clearly not the one they were motivated to find.

[pefloresjr]

Beyond the weaknesses of this study, some of the negative reaction to this is certainly due to the way it has been misused and promoted by the authors.  However, it looks to me like many are reacting more harshly because this is a STANFORD study.  Most here have high expectations for science out of Stanford and most of us have some measure of personal pride tied to Stanford excellence.  That factor doesn't really change the discussion regarding the actions the authors should take.  It does, however, inform some of the emotions expressed about it on this board.  

As for the study, I am not involved in peer review, but see some of it from my wife's work.  The language used by some scientists regarding this study struck me as particularly strong.  The next releases may quell some of that criticism, or not.  I am in the wait and see crowd but I don't think those who think the author's should "do the right thing" by admitting the weaknesses and even withdrawing it until they do better work is not akin to book burning or censorship.  It is asking that Stanford studies meet a high standard.  It is not unusual for papers from legitimate institutions to be withdrawn so that they can be further analyzed or so that work can be refined to meet pre-publication questions.  That is not censorship, it is part of the peer review process.  Again, the choice is with the authors and their department heads.  

Cheers,
Pete F.

[pefloresjr]

Beyond the weaknesses of this study, some of the negative reaction to this is certainly due to the way it has been misused and promoted by the authors.  However, it looks to me like many are reacting more harshly because this is a STANFORD study.  Most here have high expectations for science out of Stanford and most of us have some measure of personal pride tied to Stanford excellence.  That factor doesn't really change the discussion regarding the actions the authors should take.  It does, however, inform some of the emotions expressed about it on this board.  

As for the study, I am not involved in peer review, but see some of it from my wife's work.  The language used by some scientists regarding this study struck me as particularly strong.  The next releases may quell some of that criticism, or not.  I am in the wait and see crowd but I don't think those who think the author's should "do the right thing" by admitting the weaknesses and even withdrawing it until they do better work is not akin to book burning or censorship.  It is asking that Stanford studies meet a high standard.  It is not unusual for papers from legitimate institutions to be withdrawn so that they can be further analyzed or so that work can be refined to meet pre-publication questions.  That is not censorship, it is part of the peer review process.  Again, the choice is with the authors and their department heads.  

Cheers,
Pete F.

[M T]

Unclear if they are the same company as Hangzhou Alltest Biotech. They were the producers of the worst performing commercial test from the only data comparing commercial tests out there. Specificity in the 80s, woof.

These companies are NOT the same:
Alltest
Biotest  / Premier Biotech.   The manufacturer's numbers for the test can be found here (which were used in the study).

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

As of April 1, manufacturers were no longer allowed to export COVID-19 tests unless they have been approved by the China equivalent of the FDA. This test has not yet had that approval, so it can no longer be exported.  Shipments were allowed prior to that date.  The use of the word "banned" suggests something nefarious, but I don't think that's the way to portray this.  (An NBC report mis-characterized the situation, per Biotest Biotech)

[M T]

Unclear if they are the same company as Hangzhou Alltest Biotech. They were the producers of the worst performing commercial test from the only data comparing commercial tests out there. Specificity in the 80s, woof.

These companies are NOT the same:
Alltest
Biotest  / Premier Biotech.   The manufacturer's numbers for the test can be found here (which were used in the study).

Also of note, the actual manufacturer of the serology tests is a Chinese company that is actually now banned from exporting tests to the US. Hangzhou Biotest Biotech. Sounds like a reputable entity......

As of April 1, manufacturers were no longer allowed to export COVID-19 tests unless they have been approved by the China equivalent of the FDA. This test has not yet had that approval, so it can no longer be exported.  Shipments were allowed prior to that date.  The use of the word "banned" suggests something nefarious, but I don't think that's the way to portray this.  (An NBC report mis-characterized the situation, per Biotest Biotech)

[M T]

In the section of the Stanford report on the statistical measurements of the accuracy of the test, they mention "Among 371 pre-COVID samples, 369 were negative."  But that's only for IgM.   For IgG, per the manufacturer, it is 368 were negative (3 positive).  I don't see that number (99.2%) being used in their calculations.

It appears to me that the study used the wrong specificity on the IgG part of the test.  They state "Similarly, our estimates of specificity are 99.5%(95 CI 98.1-99.9%)and 100%(95 CI 90.5-100%)."  I believe they mean that the specificity of the IgM test is 99.5% and the specificity of the IgG test is 100%.  However, the manufacturer  indicates that the IgG gave 3 false positives in 371 tests with IgG.   So I believe their methodology should use either the minimum 99.2% or (75+368)/(75+371) = 99.3% for the IgG specificity.

I'll leave it to the statisticians to calculate the overall specificity (and the confidence intervals) when either a positive IgG or IgM is counted as a positive.  I'm confident it is below the 99.5% used in the report.  (I think this only drops it to 99.4%, or about 20 false positives in 3330 tests, leaving a raw incidence of 0.1% in this population.)

[M T]

In the section of the Stanford report on the statistical measurements of the accuracy of the test, they mention "Among 371 pre-COVID samples, 369 were negative."  But that's only for IgM.   For IgG, per the manufacturer, it is 368 were negative (3 positive).  I don't see that number (99.2%) being used in their calculations.

It appears to me that the study used the wrong specificity on the IgG part of the test.  They state "Similarly, our estimates of specificity are 99.5%(95 CI 98.1-99.9%)and 100%(95 CI 90.5-100%)."  I believe they mean that the specificity of the IgM test is 99.5% and the specificity of the IgG test is 100%.  However, the manufacturer  indicates that the IgG gave 3 false positives in 371 tests with IgG.   So I believe their methodology should use either the minimum 99.2% or (75+368)/(75+371) = 99.3% for the IgG specificity.

I'll leave it to the statisticians to calculate the overall specificity (and the confidence intervals) when either a positive IgG or IgM is counted as a positive.  I'm confident it is below the 99.5% used in the report.  (I think this only drops it to 99.4%, or about 20 false positives in 3330 tests, leaving a raw incidence of 0.1% in this population.)

[OutsiderFan]

What are the odds Stanford's brass demands an investigation into what happened?

I understand the "book burning" angst, and the Stanford bias (not favoring, but holding it to the highest standards not met in this case). At the same time, there is a real possibility of purposeful fraud here. I'm not saying there was, but given all the factors involved that we know about, transparency is the best disinfectant. Maybe it was innocent mistakes.  Maybe there was something nefarious.  All I know is something is not right with this situation and it needs to be cleared up, to protect the brand.

Personally, I don't knock Stanford for the Theranos debacle, but it certainly doesn't help in terms of benefits of doubts. It's time to remove all doubts and get to the absolute truth.

[OutsiderFan]

What are the odds Stanford's brass demands an investigation into what happened?

I understand the "book burning" angst, and the Stanford bias (not favoring, but holding it to the highest standards not met in this case). At the same time, there is a real possibility of purposeful fraud here. I'm not saying there was, but given all the factors involved that we know about, transparency is the best disinfectant. Maybe it was innocent mistakes.  Maybe there was something nefarious.  All I know is something is not right with this situation and it needs to be cleared up, to protect the brand.

Personally, I don't knock Stanford for the Theranos debacle, but it certainly doesn't help in terms of benefits of doubts. It's time to remove all doubts and get to the absolute truth.

[Mick]

What are the odds Stanford's brass demands an investigation into what happened?

I understand the "book burning" angst, and the Stanford bias (not favoring, but holding it to the highest standards not met in this case). At the same time, there is a real possibility of purposeful fraud here. I'm not saying there was, but given all the factors involved that we know about, transparency is the best disinfectant. Maybe it was innocent mistakes.  Maybe there was something nefarious.  All I know is something is not right with this situation and it needs to be cleared up, to protect the brand.

Personally, I don't knock Stanford for the Theranos debacle, but it certainly doesn't help in terms of benefits of doubts. It's time to remove all doubts and get to the absolute truth.

To your point, I wonder how hard Dr. Gardner attempted to shed light on Theranos and what Holmes was trying to do.  She makes a lot of noise in retrospect, but did she do as much to warn the public as Theranos was ramping up?  She made it clear to 19 year old Holmes that what she was attempting was impossible.  Now it just sounds like Monday-morning quarterbacking.

https://www.mercurynews.com/2019/06/03/s...anos-saga/

[Mick]

What are the odds Stanford's brass demands an investigation into what happened?

I understand the "book burning" angst, and the Stanford bias (not favoring, but holding it to the highest standards not met in this case). At the same time, there is a real possibility of purposeful fraud here. I'm not saying there was, but given all the factors involved that we know about, transparency is the best disinfectant. Maybe it was innocent mistakes.  Maybe there was something nefarious.  All I know is something is not right with this situation and it needs to be cleared up, to protect the brand.

Personally, I don't knock Stanford for the Theranos debacle, but it certainly doesn't help in terms of benefits of doubts. It's time to remove all doubts and get to the absolute truth.

To your point, I wonder how hard Dr. Gardner attempted to shed light on Theranos and what Holmes was trying to do.  She makes a lot of noise in retrospect, but did she do as much to warn the public as Theranos was ramping up?  She made it clear to 19 year old Holmes that what she was attempting was impossible.  Now it just sounds like Monday-morning quarterbacking.

https://www.mercurynews.com/2019/06/03/s...anos-saga/

[burger]

Also, it is possible that the concerns about selection bias and the test specificity are overblown, and, in fact, the numbers that are quoted in the article are broadly right.  For the record, I don't think so, but it is possible.

Take the concern about sensitivity.  Yes, based on the confidence interval of the sensitivity it is possible that all the positives are false positives.  But based on the point estimate of the sensitivity, that is not the case.  So, bottom line, I think the confidence interval that the paper cites is too narrow, but their point estimate is unaffected.  I believe that will need to be fixed during peer review and when this paper comes out, I'd say it is likely that it will feature broader confidence intervals that propagate the sensitivity uncertainty better.

The concern about selection bias is legitimate, but they acknowledge it in the paper.  Unless you can come up with a way to measure and adjust for it, the best they can do is what they have done; list it as a limitation.  You might speculate on its directionality, but there is no way to be sure of its magnitude.

Yesterday, I posted a link to an article about hydroxychloroquine that has also been making the rounds on news outlets.  Unfortunately, it showed that hydroxychloroquine doesn't do anything for patients with COVID-19.  It was a retrospective study, and suffers from the same bias (patients with more severe disease are more likely to be treated, a phenomenon known as confounding by indication).  I noted the bias when I pointed the paper and said I'd wait until higher quality papers would come out to make up my mind for certain. However, I also pointed out that it is unlikely that HCQ is a "game changer" because we would have seen it.  So, even though there is likely bias in the paper, I wouldn't demand its retraction.

Same here.  I will wait until true random sampling (or saturation testing) is performed before I make up my mind for certain on the prevalence and IFR.  But the study still moves the needle, and it is reasonable to think that the prevalence might be higher and the CFR might be lower than previously reported.

BC

Much of my job involves designing sampling schemes and analyzing the resulting data.  The whole point of these sampling regimes is to be able to make inferences about the population.  If the sampling is biased--unrepresentative of the population--the resulting estimates are essentially useless.  Without accessory information about the nature of the bias, you have zero basis for estimating population means.

For this study, because there was self-selection in the study participants (we have evidence for this from facebook and nextdoor threads), we have no way of knowing how representative the 3000+ people sampled were of the county.  But with only 50 positives, it would not take many 'excess' positives (beyond what you would get if you sampled everyone or did a truly random sample) to make the resulting estimate wrong.

With that as a given, we simply have no idea how unrepresentative the sample was . Was there one excess positive?  Five?  Twenty?  No one knows.  And no one can know with this dataset.  You could apply a correction using accessory information from the study participants like history of symptoms, exposure to ill people, and desire to be tested, and you could correct using the same information gathered at random in the population.  But because that was not done, we're back to an unrepresentative sample and no ability to make inferences about the population.

Like I said earlier, this isn't just a mild methodological dispute.  It's a fundamental error.  I don't ascribe any bad intentions to the authors of the paper.  Maybe none of them have ever analyzed a sample survey.  Or maybe they just didn't consider biased responses a possibility.  But this error makes all of their conclusions invalid.  This study should not move the needle any more than my making up fake data and writing a paper with an IFR of 3% should.  We need to be guided by good data and good analyses.  Period.

[burger]

Also, it is possible that the concerns about selection bias and the test specificity are overblown, and, in fact, the numbers that are quoted in the article are broadly right.  For the record, I don't think so, but it is possible.

Take the concern about sensitivity.  Yes, based on the confidence interval of the sensitivity it is possible that all the positives are false positives.  But based on the point estimate of the sensitivity, that is not the case.  So, bottom line, I think the confidence interval that the paper cites is too narrow, but their point estimate is unaffected.  I believe that will need to be fixed during peer review and when this paper comes out, I'd say it is likely that it will feature broader confidence intervals that propagate the sensitivity uncertainty better.

The concern about selection bias is legitimate, but they acknowledge it in the paper.  Unless you can come up with a way to measure and adjust for it, the best they can do is what they have done; list it as a limitation.  You might speculate on its directionality, but there is no way to be sure of its magnitude.

Yesterday, I posted a link to an article about hydroxychloroquine that has also been making the rounds on news outlets.  Unfortunately, it showed that hydroxychloroquine doesn't do anything for patients with COVID-19.  It was a retrospective study, and suffers from the same bias (patients with more severe disease are more likely to be treated, a phenomenon known as confounding by indication).  I noted the bias when I pointed the paper and said I'd wait until higher quality papers would come out to make up my mind for certain. However, I also pointed out that it is unlikely that HCQ is a "game changer" because we would have seen it.  So, even though there is likely bias in the paper, I wouldn't demand its retraction.

Same here.  I will wait until true random sampling (or saturation testing) is performed before I make up my mind for certain on the prevalence and IFR.  But the study still moves the needle, and it is reasonable to think that the prevalence might be higher and the CFR might be lower than previously reported.

BC

Much of my job involves designing sampling schemes and analyzing the resulting data.  The whole point of these sampling regimes is to be able to make inferences about the population.  If the sampling is biased--unrepresentative of the population--the resulting estimates are essentially useless.  Without accessory information about the nature of the bias, you have zero basis for estimating population means.

For this study, because there was self-selection in the study participants (we have evidence for this from facebook and nextdoor threads), we have no way of knowing how representative the 3000+ people sampled were of the county.  But with only 50 positives, it would not take many 'excess' positives (beyond what you would get if you sampled everyone or did a truly random sample) to make the resulting estimate wrong.

With that as a given, we simply have no idea how unrepresentative the sample was . Was there one excess positive?  Five?  Twenty?  No one knows.  And no one can know with this dataset.  You could apply a correction using accessory information from the study participants like history of symptoms, exposure to ill people, and desire to be tested, and you could correct using the same information gathered at random in the population.  But because that was not done, we're back to an unrepresentative sample and no ability to make inferences about the population.

Like I said earlier, this isn't just a mild methodological dispute.  It's a fundamental error.  I don't ascribe any bad intentions to the authors of the paper.  Maybe none of them have ever analyzed a sample survey.  Or maybe they just didn't consider biased responses a possibility.  But this error makes all of their conclusions invalid.  This study should not move the needle any more than my making up fake data and writing a paper with an IFR of 3% should.  We need to be guided by good data and good analyses.  Period.

[2006alum]

Ahem -- for those who were skeptical of my acting in bad faith accusation:

[tweet]https://twitter.com/CT_Bergstrom/status/1253752435467710464?s=20[/tweet]

A Stanford University professor’s wife invited parents in a wealthy enclave of Northern California to sign up for her husband’s coronavirus antibody study this month, falsely claiming that an “FDA approved” test would tell them if they had immunity and could “return to work without fear,” according to an email obtained by BuzzFeed News.

The email, sent to a listserv for Ardis G. Egan Junior High School in the city of Los Altos on Friday, April 2, advertised a study set to begin that weekend. With the subject line “COVID-19 antibody testing - FREE,” the email described how participants could gain “peace of mind” and “know if you are immune.” The results would help researchers calculate the virus’s spread throughout the surrounding county of Santa Clara, according to the message sent by Catherine Su, a radiation oncologist married to Jay Bhattacharya, the Stanford professor of medicine leading the study.

Weeks later, early results from Bhattacharya’s team would conclude that, based on the tests, the area had 50 to 85 times more infections than reported cases. That finding, along with their claim that the coronavirus would therefore have a lower fatality rate than previously thought, made national headlines. But almost as immediately, the study came under fire from scientists, who said it was based on a heavily flawed data analysis that ignored questions about the antibody test’s false positive rate — as well as a problematic Facebook recruitment strategy.

The email reveals that the researchers did not disclose another way participants were recruited that could have further skewed the results. In addition to targeting a specific demographic of parents in a wealthy part of Silicon Valley — making it even less likely that the participants represented a random sample — the email falsely claimed that the study’s antibody test was FDA approved, and was worded in a way that might have disproportionately attracted people who had previously been sick. It also misrepresented what participants could learn about their health from the testing.

Bhattacharya’s explanation is that he didn't know about the email and didn't approve it. Sure, so she invented all of this out of thin air with no knowledge or guidance from her husband? How did she even have enough knowledge about the study to be able to do this kind of promotion? And as an MD wife of a professor of medicine, surely she knows that making false claims about FDA approval and using unapproved means of solicitation for the study could compromise its integrity. Sure smells fishy to me.

[2006alum]

Ahem -- for those who were skeptical of my acting in bad faith accusation:

[tweet]https://twitter.com/CT_Bergstrom/status/1253752435467710464?s=20[/tweet]

A Stanford University professor’s wife invited parents in a wealthy enclave of Northern California to sign up for her husband’s coronavirus antibody study this month, falsely claiming that an “FDA approved” test would tell them if they had immunity and could “return to work without fear,” according to an email obtained by BuzzFeed News.

The email, sent to a listserv for Ardis G. Egan Junior High School in the city of Los Altos on Friday, April 2, advertised a study set to begin that weekend. With the subject line “COVID-19 antibody testing - FREE,” the email described how participants could gain “peace of mind” and “know if you are immune.” The results would help researchers calculate the virus’s spread throughout the surrounding county of Santa Clara, according to the message sent by Catherine Su, a radiation oncologist married to Jay Bhattacharya, the Stanford professor of medicine leading the study.

Weeks later, early results from Bhattacharya’s team would conclude that, based on the tests, the area had 50 to 85 times more infections than reported cases. That finding, along with their claim that the coronavirus would therefore have a lower fatality rate than previously thought, made national headlines. But almost as immediately, the study came under fire from scientists, who said it was based on a heavily flawed data analysis that ignored questions about the antibody test’s false positive rate — as well as a problematic Facebook recruitment strategy.

The email reveals that the researchers did not disclose another way participants were recruited that could have further skewed the results. In addition to targeting a specific demographic of parents in a wealthy part of Silicon Valley — making it even less likely that the participants represented a random sample — the email falsely claimed that the study’s antibody test was FDA approved, and was worded in a way that might have disproportionately attracted people who had previously been sick. It also misrepresented what participants could learn about their health from the testing.

Bhattacharya’s explanation is that he didn't know about the email and didn't approve it. Sure, so she invented all of this out of thin air with no knowledge or guidance from her husband? How did she even have enough knowledge about the study to be able to do this kind of promotion? And as an MD wife of a professor of medicine, surely she knows that making false claims about FDA approval and using unapproved means of solicitation for the study could compromise its integrity. Sure smells fishy to me.

[teejers1]

Ahem -- for those who were skeptical of my acting in bad faith accusation:

[tweet]https://twitter.com/CT_Bergstrom/status/1253752435467710464?s=20[/tweet]

A Stanford University professor’s wife invited parents in a wealthy enclave of Northern California to sign up for her husband’s coronavirus antibody study this month, falsely claiming that an “FDA approved” test would tell them if they had immunity and could “return to work without fear,” according to an email obtained by BuzzFeed News.

The email, sent to a listserv for Ardis G. Egan Junior High School in the city of Los Altos on Friday, April 2, advertised a study set to begin that weekend. With the subject line “COVID-19 antibody testing - FREE,” the email described how participants could gain “peace of mind” and “know if you are immune.” The results would help researchers calculate the virus’s spread throughout the surrounding county of Santa Clara, according to the message sent by Catherine Su, a radiation oncologist married to Jay Bhattacharya, the Stanford professor of medicine leading the study.

Weeks later, early results from Bhattacharya’s team would conclude that, based on the tests, the area had 50 to 85 times more infections than reported cases. That finding, along with their claim that the coronavirus would therefore have a lower fatality rate than previously thought, made national headlines. But almost as immediately, the study came under fire from scientists, who said it was based on a heavily flawed data analysis that ignored questions about the antibody test’s false positive rate — as well as a problematic Facebook recruitment strategy.

The email reveals that the researchers did not disclose another way participants were recruited that could have further skewed the results. In addition to targeting a specific demographic of parents in a wealthy part of Silicon Valley — making it even less likely that the participants represented a random sample — the email falsely claimed that the study’s antibody test was FDA approved, and was worded in a way that might have disproportionately attracted people who had previously been sick. It also misrepresented what participants could learn about their health from the testing.

Bhattacharya’s explanation is that he didn't know about the email and didn't approve it. Sure, so she invented all of this out of thin air with no knowledge or guidance from her husband? How did she even have enough knowledge about the study to be able to do this kind of promotion? And as an MD wife of a professor of medicine, surely she knows that making false claims about FDA approval and using unapproved means of solicitation for the study could compromise its integrity. Sure smells fishy to me.

I'm certainly not as learned as the many folks here who edify us with their lengthy posts on all things Covid-research, study, etc, reflecting such learnedness.  
So pardon the (undoubted) ignorance of my question.

I get why the population being tested should be "representative," to the extent possible.  What I don't get is why the motive for having the test would necessarily adversely affect the representative nature of the test.  In other words, let's say the advertising was:  "come get an antibody test; if you're found to have it, you're immune and can go back to work."  

How does that necessarily affect the representative nature of those being tested?

And since you can't force people to participate in a study, how can you guarantee "representativeness" of the participants?  Is it all about having them answer a questionnaire (where you assume answers are truthful) and then trying to take group having representative characteristics which the questions are trying to identify?

[teejers1]

Ahem -- for those who were skeptical of my acting in bad faith accusation:

[tweet]https://twitter.com/CT_Bergstrom/status/1253752435467710464?s=20[/tweet]

A Stanford University professor’s wife invited parents in a wealthy enclave of Northern California to sign up for her husband’s coronavirus antibody study this month, falsely claiming that an “FDA approved” test would tell them if they had immunity and could “return to work without fear,” according to an email obtained by BuzzFeed News.

The email, sent to a listserv for Ardis G. Egan Junior High School in the city of Los Altos on Friday, April 2, advertised a study set to begin that weekend. With the subject line “COVID-19 antibody testing - FREE,” the email described how participants could gain “peace of mind” and “know if you are immune.” The results would help researchers calculate the virus’s spread throughout the surrounding county of Santa Clara, according to the message sent by Catherine Su, a radiation oncologist married to Jay Bhattacharya, the Stanford professor of medicine leading the study.

Weeks later, early results from Bhattacharya’s team would conclude that, based on the tests, the area had 50 to 85 times more infections than reported cases. That finding, along with their claim that the coronavirus would therefore have a lower fatality rate than previously thought, made national headlines. But almost as immediately, the study came under fire from scientists, who said it was based on a heavily flawed data analysis that ignored questions about the antibody test’s false positive rate — as well as a problematic Facebook recruitment strategy.

The email reveals that the researchers did not disclose another way participants were recruited that could have further skewed the results. In addition to targeting a specific demographic of parents in a wealthy part of Silicon Valley — making it even less likely that the participants represented a random sample — the email falsely claimed that the study’s antibody test was FDA approved, and was worded in a way that might have disproportionately attracted people who had previously been sick. It also misrepresented what participants could learn about their health from the testing.

Bhattacharya’s explanation is that he didn't know about the email and didn't approve it. Sure, so she invented all of this out of thin air with no knowledge or guidance from her husband? How did she even have enough knowledge about the study to be able to do this kind of promotion? And as an MD wife of a professor of medicine, surely she knows that making false claims about FDA approval and using unapproved means of solicitation for the study could compromise its integrity. Sure smells fishy to me.

I'm certainly not as learned as the many folks here who edify us with their lengthy posts on all things Covid-research, study, etc, reflecting such learnedness.  
So pardon the (undoubted) ignorance of my question.

I get why the population being tested should be "representative," to the extent possible.  What I don't get is why the motive for having the test would necessarily adversely affect the representative nature of the test.  In other words, let's say the advertising was:  "come get an antibody test; if you're found to have it, you're immune and can go back to work."  

How does that necessarily affect the representative nature of those being tested?

And since you can't force people to participate in a study, how can you guarantee "representativeness" of the participants?  Is it all about having them answer a questionnaire (where you assume answers are truthful) and then trying to take group having representative characteristics which the questions are trying to identify?

[BostonCard]

All valid questions, Teejers.

As you point out its easy to see how an unrepresentative sample could lead to the wrong conclusions, but that doesn't mean that an unrepresentative sample did significantly bias the results.  At this point, all we have is speculation about the directionality, but that could be wrong, or the magnitude could be small.  That's why I push back against people who dismiss the study out of hand (while maintaining a healthy degree of skepticism).

But at least in theory, imagine two people.  The first person had a cough and fever in late February but never got tested because they weren't sick enough and didn't have any direct contacts to China.  Now they are curious to know if that might have been COVID-19.  The second person has not been sick, and has been socially isolating for a while.  The first person may be more motivated to find out if those symptoms were COVID-19.  So, if the first person has a 5% chance of having had COVID-19 and the second person has <1% chance of having had COVID-19, but the first person is five times as likely to show up and get tested, you have a situation where the testing may show a higher prevalence of COVID-19 than actually exists in the county.

That being said, you could also imagine the opposite scenario.  Relative wealthy Palo Alto residents who work in tech and can work from home (and thus shelter in place) show up, even though they are low risk, but minority San Jose residents who work at a grocery store and thus are more likely to have been exposed are too busy or don't understand it or don't trust medical research studies and are thus less likely to show up and get tested, even though they are more likely to have been exposed.

The authors of the study tried to adjust for the second scenario (by over-weighting minority participants), but who knows if the adjustment was right or if they underadjusted or overadjusted.

BC

[BostonCard]

All valid questions, Teejers.

As you point out its easy to see how an unrepresentative sample could lead to the wrong conclusions, but that doesn't mean that an unrepresentative sample did significantly bias the results.  At this point, all we have is speculation about the directionality, but that could be wrong, or the magnitude could be small.  That's why I push back against people who dismiss the study out of hand (while maintaining a healthy degree of skepticism).

But at least in theory, imagine two people.  The first person had a cough and fever in late February but never got tested because they weren't sick enough and didn't have any direct contacts to China.  Now they are curious to know if that might have been COVID-19.  The second person has not been sick, and has been socially isolating for a while.  The first person may be more motivated to find out if those symptoms were COVID-19.  So, if the first person has a 5% chance of having had COVID-19 and the second person has <1% chance of having had COVID-19, but the first person is five times as likely to show up and get tested, you have a situation where the testing may show a higher prevalence of COVID-19 than actually exists in the county.

That being said, you could also imagine the opposite scenario.  Relative wealthy Palo Alto residents who work in tech and can work from home (and thus shelter in place) show up, even though they are low risk, but minority San Jose residents who work at a grocery store and thus are more likely to have been exposed are too busy or don't understand it or don't trust medical research studies and are thus less likely to show up and get tested, even though they are more likely to have been exposed.

The authors of the study tried to adjust for the second scenario (by over-weighting minority participants), but who knows if the adjustment was right or if they underadjusted or overadjusted.

BC

[akiddoc]

https://www.sfgate.com/bayarea/article/W...225414.php

Now in sfgate.

[akiddoc]

https://www.sfgate.com/bayarea/article/W...225414.php

Now in sfgate.

[dabigv13]

Important work being done to evaluate some of the antibody tests on the market by the below group.

https://covidtestingproject.org/

They've tested 10 different antibody tests so far, including the one imported by Premier, which was the test used in the Santa Clara and LA studies. In testing the specificity of the test, they checked 108 samples from 2018, before Covid19 existed, and found 3 positives, for a false positive rate of 2.8%, much higher than the 0.5% the Santa Clara paper indicates (2 positives out of 371, though this was by the manufacturer; they also validated with 30 samples at Stanford and had no false positives).

So essentially their entire findings could be explained by the false positives of their test, and their study doesn't meaningfully contribute to our knowledge of covid prevalence in SCC (or LA), though a prevalence of 1-4% remains plausible.

[dabigv13]

Important work being done to evaluate some of the antibody tests on the market by the below group.

https://covidtestingproject.org/

They've tested 10 different antibody tests so far, including the one imported by Premier, which was the test used in the Santa Clara and LA studies. In testing the specificity of the test, they checked 108 samples from 2018, before Covid19 existed, and found 3 positives, for a false positive rate of 2.8%, much higher than the 0.5% the Santa Clara paper indicates (2 positives out of 371, though this was by the manufacturer; they also validated with 30 samples at Stanford and had no false positives).

So essentially their entire findings could be explained by the false positives of their test, and their study doesn't meaningfully contribute to our knowledge of covid prevalence in SCC (or LA), though a prevalence of 1-4% remains plausible.

[BostonCard]

Thanks.  I would like to see testing for specificity done in much larger samples.  The 95% CI for three positive tests in 108 is 0.0058 to 0.079 (inother words .58% to 7.9%), which is very broad and doesn't really help answer the question as to whether the Stanford study result was real or can be explained with false positives.

I'd like to see the tests looked at in at least 2000 known negative samples, which would narrow down the CI to ± 0.5% if the point estimate is 1% (the CI is broader if the specificity is lower, and narrower if the specificity is higher), but if the specificity is lower than 99%, then the tests are probably not fit for purpose for low prevalence community antibody testing.

BC

[BostonCard]

Thanks.  I would like to see testing for specificity done in much larger samples.  The 95% CI for three positive tests in 108 is 0.0058 to 0.079 (inother words .58% to 7.9%), which is very broad and doesn't really help answer the question as to whether the Stanford study result was real or can be explained with false positives.

I'd like to see the tests looked at in at least 2000 known negative samples, which would narrow down the CI to ± 0.5% if the point estimate is 1% (the CI is broader if the specificity is lower, and narrower if the specificity is higher), but if the specificity is lower than 99%, then the tests are probably not fit for purpose for low prevalence community antibody testing.

BC