Even very accurate tests in the lab can result in a vast majority of false positives in the field.
This “false positive paradox” from testing people without symptoms led to a “false positive catastrophe” around the world, which massively exaggerated the appearance of the COVID-19 pandemic.
The Food & Drug Administration warned way back in November 2020 about how even COVID-19 antigen tests that were 98% accurate in the lab could still produce 96% or more false positives in the field.
Let me say that again because it’s so counterintuitive: a test that is 98% accurate in the lab can produce 96% or more false positives in the field.
This strange result is actually well-known in epidemiology, and it even has a name: “the false positive paradox.” Even so, it was mostly forgotten during the COVID-19 pandemic despite warnings like the one FDA issued.
COVID-19 was defined, for almost the first time in history, as based on only a positive test result.
When we combine this false positive paradox with a case definition of COVID-19 that requires only a positive test result (see “laboratory criteria”), it’s easy to see how pervasive testing of people without symptoms led to a very large number of “cases” that weren’t actually cases at all.
Even worse, most nations also defined a “COVID-19 hospitalization” and a “COVID-19 death” in a way that required nothing more than a positive test result, and sometimes not even that was required.
So the false positive paradox led, in country after country, to a chain of pandemic data that was based on a fundamental mistake, a mirage. That mistake is the failure to understand the false positive paradox.
I’ll explain now in a little more detail how the false positive paradox works.
It happens when the actual disease is rare in the population being tested. When you test the population at large, regardless of symptoms, you can get dramatically high rates of false positives. Here’s why:
It’s because the test will produce the same percentage of false positives at both high or low disease prevalence (which means how common the disease actually is in the population), but as disease prevalence goes down the true positives decline and the false positives will start to swamp the true positives.
So even very accurate tests can result in a vast majority of false positives at low disease prevalence.
Even during the Delta variant wave in cases in the United States in 2021 and elsewhere the disease prevalence was still quite low (under 1% in the population as a whole).
For example, Sadoff et al. 2021, the published results of the Johnson & Johnson vaccine clinical trial, including almost 40,000 participants in a half-dozen countries, from late September 2020 to late January 2021, found only a 0.5% PCR positive baseline (see Sadoff et al. 2021 supplementary appendix, p. 23).
Similarly, Baden et al., 2020, found a 0.6% background positive PCR test result in the 30,420 clinical trial participants for the Moderna vaccine.
Study participants for this trial were selected based on being at higher risk for exposure to the virus and the testing was conducted from late July to late October 2020. So this 0.6% background should, by basic logic, be a higher figure than for the population as a whole.
Why weren’t public health authorities and experts talking more about these issues? Well, some did, but most just ignored this extremely important issue and accepted the headline pandemic data without digging any deeper.
An analysis by a trio of doctors with expertise in epidemiology — including Dr. Westyn Branch-Elliman at Harvard Medical School as lead author — did discuss it in a July 2021 commentary in US News & World Report.
They warned against widespread COVID-19 testing of school students that fall — specifically because of the certainty that such testing will yield a large majority of false positives.
“Put simply, no test is perfect,” they wrote. “There are errors in which a test is positive but there is no disease (false positives), and in which a test is negative even when the person has the disease (false negatives). When case rates are low (this is another way of saying disease prevalence), the majority — and sometimes even the vast majority — of positive test results are false-positives.”
They then add some detail:
“(Various studies) across (Massachusetts) and across the country have shown us that the probability of COVID-19 in asymptomatic students attending in-person learning was consistently low — less than 0.5% — even before widespread vaccination. Using 0.5 as a (very) generous overestimate and a close-to-perfect (99% specific) diagnostic test, that means for every one true positive test, three will be false-positive. The true specificity of some polymerase chain reaction (PCR) tests is probably closer to 95% (in other words, still very good, but not quite so close to perfect). This more realistic estimate increases the proportion of false-positives test results even more — up to 14 false-positives for every real case of COVID-19 identified by the screening program. As case rates continue to decline, the ratio of real cases to false-positives only gets worse (and worse). Assuming a rate of 1 in 1,000 or 0.1% and a nearly perfect test, there are 14 false-positive tests for every real case found by a screening testing program, and 71 if we use the more realistic estimate of 95% specificity.”
So, to summarize: these doctors were warning that COVID-19 screening in schools would very likely yield 71 out of 72 false positives — just one true positive out of 72 positive test results.
It’s not hard to see how that may be labeled a false positive catastrophe as it led to renewed fear, panic-based responses, shutting schools again or delaying reopening, etc.
An essay in The Guardian by mathematician Tom Chivers gives a great history and explanation of the counterintuitive math behind the false positive paradox. He writes:
“If you get a positive result on a COVID test that only gives a false positive one time in every 1,000, what’s the chance that you’ve actually got COVID?” he asked. “Surely it’s 99.9%, right?
“No! The correct answer is: you have no idea. You don’t have enough information to make the judgment.”
With this kind of COVID-19 testing, it’s crucial to understand the disease prevalence of COVID-19 in the population since it makes all the difference to how many false positives you get.
Now that we have good data about society-wide disease prevalence of COVID-19, as discussed above, we can extend the same argument against school testing.
We also can reasonably conclude that we have also seen a false positive catastrophe in the last three years at the societal level, with probably well over 90% of COVID-19 test results from screening programs being false positives.
This is because much of the COVID-19 testing since the early days of the pandemic has been screening or surveillance testing, which by definition doesn’t consider symptoms before performing a test.
This kind of testing includes these examples:
- School testing as discussed
- University testing
- Travel testing
- Daily or weekly workplace testing
- Sports team testing
- Voluntary self-testing before social engagements
The combination of these screening programs constituted the large majority of all COVID-19 testing because it was so repetitive.
For example, the NFL screening program for the 2020 season included, by itself, 632,370 PCR tests conducted in screening programs for 32 football teams in 24 states (see Mack et al. 2020).
This is not a new problem. In fact, it’s been highlighted in past pandemics as a problem.
The Centers for Disease Control and Prevention’s 2004 guidance from the SARS pandemic, for example, stated: “To decrease the possibility of a false-positive result, testing should be limited to patients with a high index of suspicion for having SARS-CoV disease (i.e. having symptoms or contact with someone who has had the disease).”
So how did this get forgotten and/or ignored for so long, and so catastrophically, during the COVID-19 pandemic? That answer will have to wait for another essay.