\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}



\usepackage{float}








\begin{document}

\title{The ``false positive paradox'' and the risks of testing asymptomatic
people for COVID-19}



\author[1]{Tam Hunt}%
\author[2]{Blaine Williams}%
\author[3]{Daniel Howard}%
\affil[1]{Univ. of California, Santa Barbara}%
\affil[2]{Kaiser Permanente Oahu}%
\affil[3]{Independent Researcher}%


\vspace{-1em}



  
  \date{May 25, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
Widespread screening of asymptomatic people leads to high numbers of
false positives when background prevalence is low, even with accurate
tests. During the Covid-19 pandemic, not only has the background
prevalence been low (vaccine clinical trial baseline testing finds
0.5-0.6\% even during periods of higher prevalence), but the various
COVID-19 tests are not very accurate. When inaccurate tests are combined
with a low background prevalence, this results in a massive and
unacknowledged problem of far more false positive test results than true
positive test results, leading also to inaccurate characterization of
COVID-19 hospitalizations and deaths.%
\end{abstract}%



\sloppy


Tam Hunt, J.D.\textsuperscript{1}

Blaine Williams, M.D.\textsuperscript{2}

Daniel Howard, Ph.D\textsuperscript{3}

\textsuperscript{1} Univ. of California, Santa Barbara

\textsuperscript{2} Kaiser Permanente Oahu

\textsuperscript{3} Independent Researcher

\textbf{Correspondence to:~}Tam
Hunt~\href{mailto:tam.hunt@psych.ucsb.edu}{\nolinkurl{tam.hunt@psych.ucsb.edu}}

It is well-known that widespread testing of people with a low
probability of having the disease at issue will lead to high levels of
false positives, even with accurate tests (Skittrall et al. 2020;
Bokhorst et al. 2012; Dinnes et al. 2021; Madrigal et al. 2020). This
has been described as the ``false positive paradox'' (Flender 2019).
It's a paradox because even quite accurate tests can lead to high levels
of false positives when used widely in a population with low actual
prevalence of a given disease.

For example, Skittrall et al. 2020 calculated that hypothetically
screening 100,000 people chosen randomly from the general UK population
in June 2020 would result in 25 times more false positives than true
positives (50 false positives and 2 true positives), even with a test
thought to have a very high 99.95\% specificity.

Widespread screening during previous outbreaks and pandemics has
generally not been recommended because of the potential for high false
positives. The Center for Disease Control (CDC)'s 2004 guidance from the
SARS pandemic (CDC 2004), for example, stated: ``To decrease the
possibility of a false-positive result, testing should be limited to
patients with a high index of suspicion for having SARS-CoV disease.''

\emph{The debate around screening asymptomatic individuals}

The World Health Organization (WHO) and CDC did, however, recommend
testing of asymptomatic people early in the COVID-19 pandemic, but the
CDC revised this guidance in August of 2020 to recommend not testing
asymptomatics even after potential exposure, only to reverse course
again after public and expert pushback in the U.S. (Fang, M. 2020).

The US Food and Drug Administration (FDA) issued a strongly worded
letter to healthcare providers in November 2020 warning about the
potential for false positives from antigen testing, describing the
problems associated with screening populations with a low background
prevalence of COVID-19 (FDA 2020). The letter reminds practitioners
that: ``As disease prevalence decreases, the percent of test results
that are false positives increase.''

CDC's most recent (March 2021) guidance does, however, still recommend
widespread screening, which necessarily includes testing of mostly
asymptomatics, despite the widely known issues regarding such policies.
CDC's guidance states: ``Rapid, point-of care serial screening can
identify asymptomatic cases and help interrupt SARS-CoV-2 transmission.
This is especially important when community risk or transmission levels
are substantial or high.''

Many countries, including the UK and the U.S., have engaged in
widespread population testing (Mercer and Salit 2021: COVID-19 testing
is the ``largest global testing programme in history, in which hundreds
of millions of individuals have been tested to date.'').

Not surprisingly, there appears to have been significant internal debate
about this important issue in these agencies. Those arguing for
screening asymptomatic people seem to have overlooked the extremely high
false positive rate that such testing necessarily entails with low
background prevalence (Skittrall et al. 2020; Dinnes et al. 2021).

We explain below why it is almost always unwise to test asymptomatics on
a widespread basis. Such testing can lead to extremely high levels of
false positives even with highly accurate tests. Unfortunately, the
available PCR and antigen tests are not very accurate.~And inaccurate
tests combined with widespread testing of asymptomatics can lead to
catastrophically high levels of false positives, as explained below.

\emph{The false positive paradox regarding prostate cancer}

Let's first, as an illustration of the issue, look at the history of
screening for prostate cancer. In the U.S. screening for prostate cancer
was normal practice, under the common sense notion that it is good to
detect and treat illnesses early. What resulted, however, was a high
level of false positives due to the false positive paradox, and a
growing awareness that most prostate cancers don't grow fast enough to
be an issue for the patient, particularly in the elderly.

Consequently, the CDC, American Cancer Society, American Medical
Association, and most other groups have stopped recommending widespread
testing for prostate cancer, due to potential for false positives,
overdiagnosis, overtreatment, and possible iatrogenic harm.~CDC, for
example, states at its prostate cancer website (CDC Prostate Cancer
website, emphasis in original):

Possible Harm from Screening: False positive test results:~This occurs
when a man has an abnormal PSA test but does not have prostate cancer.
False positive test results often lead to unnecessary tests, like a
biopsy of the prostate. They may cause men to worry about their health.
Older men are more likely to have false positive test results. Possible
Harms from Diagnosis: Screening finds prostate cancer in some men who
would never have had symptoms from their cancer in their lifetime.
Treatment of men who would not have had symptoms or died from prostate
cancer can cause them to have complications from treatment, but not
benefit from treatment. This is called~\emph{overdiagnosis\textbf{.}}

\emph{What is the background prevalence for COVID-19?}

Even a test with a very high 99\% specificity (1\% chance of false
positives), when used to screen asymptomatic populations with a low
background rate of actual infection, will yield high levels of false
positives.

The background rate of COVID-19 infection, even during high points, has
always been relatively low. For example, Sadoff et al. 2021, the
published results of the Johnson \& Johnson vaccine clinical trial,
including almost 40,000 participants in half a dozen countries, from
late September 2020 to late January 2021, found a 0.5\% PCR positive
baseline (see Sadoff et al. 2021 supplementary appendix, p. 23).

Similarly, Baden et al., 2020, found a 0.6\% background positive PCR
test result in the 30,420 clinical trial participants for the Moderna
vaccine, after initial testing. Study participants for this trial were
selected based on being at higher risk for exposure to the virus and the
testing was conducted from late July to late October 2020.

In the UK, the Government's survey of the population in June 2020 found
about 1 in 2,200 people with an active infection in the study window,
which is 0.05\%, an order of magnitude lower than the vaccine trials
just mentioned (Connors and Williams 2020).

Voysey et al. 2021, the published results of the Astrazeneca vaccine
trial, found 1.65\% baseline antibody test positive results in their
20,675 study participants, but the antibody test (also known as serology
testing) measures the presence of any past infection, not current
infections, so this will necessarily be a significantly higher number
than a snapshot in time of current infections, as is the case for PCR or
antigen tests. The Astrazeneca trial study did not include PCR testing
at baseline like the Moderna and J\&J trials did.

\emph{Why we shouldn't test asymptomatics for COVID-19}

Common sense would suggest that a test with 99\% specificity would
return only about 1 in a 100 false positive results. But this is not how
it works. The false positive rate is far higher when disease prevalence
is as low as the studies just cited have found. In other words: the
Positive Predictive Value of screening testing is very low when
background prevalence is low (Bokhorst et al. 2012; Skittrall et al.
2020; Dinnes et al. 2021).

Here's why: If we test 1,000 people randomly in a population where 1\%
have the illness at issue, and our test is 99\% specific to that
illness, we will have one true positive and one false positive for each
100 tests. So testing 1,000 people results in 10 true positives and 10
false positives.

Figure 1 below examines COVID-19 antigen testing, a type of test
commonly used for screening, including primarily asymptomatics, since
antigen tests became available in the summer of 2020. The figure is
based on the British Medical Journal (BMJ)'s COVID-19 test accuracy
interactive calculator (Watson 2020; see link for the calculator in the
references; it's educational to play with the calculator to see how
different inputs affect false results, also shown in Figure 3).

In populating the three cells in the calculator (at the top of the
image) we've conservatively assumed 1\% pre-test probability of active
infection, which is, based on the data reviewed above, a higher level of
active infection than was found in the large vaccine clinical trials.

We also assumed 58\% sensitivity and 99\% specificity, which are the
findings of a recent Cochrane meta-analysis combining 64 published
studies of antigen test accuracy, when used to test asymptomatics
(Dinnes, J. et al. 2021).

The result in this scenario is 50\% false positives (1 true positive and
1 false positive) --- even with a 99\% specificity test. There would
theoretically be zero false negatives, so the risk of missing actual
infections is not at issue.

This result is not surprising because the numbers are the same as the
hypothetical scenario just discussed. The Dinnes meta-analysis concludes
similarly, but for a lower background prevalence and a slightly higher
test specificity (99.6\%): ``At 0.5\% prevalence applying the same tests
in asymptomatic people would result in {[}Positive Predictive Value{]}
of 11\% to 28\% meaning that between 7 in 10 and 9 in 10 positive
results will be false positives, and between 1 in 2 and 1 in 3 cases
will be missed.''

\emph{Figure 1. False positives account for 50\% (1/2) of the positive
results, even with a 99\% sensitivity test when screening if the active
infection level in the population is at~1\%.}\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image1/image1}

\end{center}
\end{figure}

50\% is the same as random chance. In other words, this 99\% specificity
test can do no better than a coin flip when declaring a positive result.
So screening in this scenario is not warranted because data that is no
better than a coin flip is not data --- it's random chance.

However, the situation is much worse than this because neither PCR nor
antigen tests are close to a 99\% specificity level in practice, for
various reasons (Braunstein et al. 2021). Lee 2020 performed a lab
analysis of the CDC PCR test accuracy, which was widely used in the
first months of the pandemic, and found it had a 70\% specificity (i.e.
30\% false positives) and 80\% sensitivity (20\% false negatives). This
level of inaccuracy matches the CDC's own internal report that found
33\% false results when its PCR test was released in late February 2020,
as reported on by National Public Radio (Temple-Raston 2020).

\emph{Why intuition is a poor guide regarding testing}

Intuitively, and in an emergency situation, we may think that a 70-80\%
accuracy rate is far from perfect but may still be ``good enough.'' But
this is where common sense and intuition gets us -- and the public --
into trouble. If we input these figures in the BMJ calculator, we obtain
a catastrophic 30 out of 31 false positives (Figure 2).

In other words, at a 1\% pre-test probably (background prevalence), just
one out of 31 positive test results is a true positive. And, again, we
have zero false negatives, so the tests are not missing true positives
in this scenario.

\emph{Figure 2. Using Lee 2020 findings regarding CDC's PCR test
inaccuracies when screening a population with active COVID-19 infection
level of~1\%.}\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image2/image2}

\end{center}
\end{figure}

This dynamic is a large part of why there have been so many allegedly
asymptomatic carriers of the virus: 1) a ``confirmed case'' was defined
by the CDC as anyone who tested positive (CDC Interim Case Definition
2020); 2) however, with highly inaccurate tests and widespread testing
of asymptomatic individuals, the large majority of ``cases'' seem to
have been false positives (Braunstein et al. 2021 makes a similar
point).

Figure 3 summarizes the false results for the two scenarios already
discussed, but in a range from 1-20\% pre-test probability. False
positives remain very high through 5\% and higher background prevalence
and false negatives remain low.

Figure 3. \emph{Summary of results: a) antigen test sensitivity of 58\%
and specificity of 99\%; b) PCR test sensitivity of 80\% and specificity
of 70\%. False positive results dominate in real-world low prevalence
conditions.}

a)\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image3/image3}

\end{center}
\end{figure}

b)\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image4/image4}

\end{center}
\end{figure}

The end result of these various policy mis-steps is fear of asymptomatic
transmission leading to release of rushed and highly flawed tests
(Bandler et al. 2020), which were used to screen widely in asymptomatic
populations, which then seemed to find large numbers of asymptomatic
carriers (who were actually in the large majority of cases false
positives), creating a vicious cycle of faulty data.

This problem relates to more than just misidentifying positive COVID-19
cases; it also is relevant to data on hospitalizations and death rates.
After testing became widely available, it became standard practice to
test all patients admitted to hospitals in the U.S., regardless of
symptoms. While this may have been a necessarily cautious step in order
to minimize outbreaks in hospitals, it significantly inflated
hospitalizations and deaths attributed to COVID-19. A positive test
result was the primary basis for defining COVID-19 hospitalizations and
deaths since no symptoms were required to designate a COVID-19
hospitalization or death as such.

In other words, since the CDC and WHO case definitions took the
unprecedented step of defining a ``confirmed case'' as simply a positive
lab test result, and then most jurisdictions also defined a COVID-19
hospitalization and a COVID-19 death in the same manner, if the large
majority of positive test results are false positives, it is necessary
to re-examine the pandemic surveillance data chain from the beginning.

We'll close with a homework problem for the reader: think through how to
apply this Bayesian thinking to the J\&J baseline PCR testing, which as
mentioned above, found 0.5\% baseline PCR test positives at the
beginning of the trial period. How many of those positive results were
true positives? How many were false positives?

\textbf{References}

\begin{enumerate}
\tightlist
\item
  Baden LR, El Sahly HM, Essink B, Kotloff K, Frey S, Novak R, et al.
  Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med.
  2021;384(5):403--16.
\item
  Bandler J et al. ``Inside the fall of the CDC.'' ProPublica. 2020 Oct
  15.
\item
  Bokhorst LP, Zhu X, Bul M, Bangma CH, Schr\selectlanguage{ngerman}öder FH, Roobol MJ. Positive
  predictive value of prostate biopsy indicated by
  prostate-specific-antigen-based prostate cancer screening: trends over
  time in a European randomized trial*. BJU International.
  2012;110(11):1654--60.
\item
  Braunstein GD, Schwartz L, Hymel P, Fielding J. False Positive Results
  With SARS-CoV-2 RT-PCR Tests and How to Evaluate a RT-PCR-Positive
  Test for the Possibility of a False Positive Result. Journal of
  Occupational \& Environmental Medicine. 2021;63(3).
\item
  CDC 2004, Supplement F: Laboratory Guidance - CDC {[}Internet{]}. CDC
  Notice regarding 2003 SARS. {[}cited 2021Jun15{]}. Available from:
  https://www.cdc.gov/sars/guidance/f-lab/downloads/f-lab-full.pdf
\item
  CDC Coronavirus Disease 2019 (COVID-19)2020 Interim Case Definition,
  Approved April 5, 2020 {[}Internet{]}. CDC Interim Case Definition.
  CDC; 2020 {[}cited 2021Jun15{]}. Available from:
  https://wwwn.cdc.gov/nndss/conditions/coronavirus-disease-2019-covid-19/case-definition/2020/
\item
  CDC Prostate Cancer website. What Are the Benefits and Harms of
  Screening for Prostate Cancer? {[}Internet{]}. CDC Prostate Cancer
  Screening. CDC; 2020 {[}cited 2021Jun15{]}. Available from:
  https://www.cdc.gov/cancer/prostate/basic\_info/benefits-harms.htm.
\item
  Connors E, Williams C. Coronavirus (COVID-19) Infection Survey pilot:
  2 July 2020 {[}Internet{]}. Coronavirus (COVID-19) Infection Survey
  pilot - Office for National Statistics. Office for National
  Statistics; 2020 {[}cited 2021Jun15{]}. Available from:
  https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/coronaviruscovid19infectionsurveypilot/2july2020
\item
  Dinnes\_J, Deeks\_JJ, Berhane\_S, Taylor\_M, Adriano\_A, Davenport\_C,
  Dittrich\_S, Emperador\_D, Takwoingi\_Y, Cunningham\_J, Beese\_S,
  Domen\_J, Dretzke\_J, Ferrante di Ru(ano\_L, Harris\_IM, Price\_MJ,
  Taylor-Phillips\_S, Hoo-\_L, Leeflang\_MMG, McInnes\_MDF, Spijker\_R,
  Van den Bruel\_A, Cochrane COVID-19 Diagnostic Test Accuracy Group.
  Rapid, point-of-care antigen and molecular-based tests for diagnosis
  of SARS-CoV-2 infection. Cochrane Database of Systematic Reviews 2021,
  Issue 3. Art. No.: CD013705.
\item
  Fang M. CDC Quietly Changes Testing Guidelines To Exclude People With
  No Symptoms {[}Internet{]}. HuffPost. HuffPost; 2020 {[}cited
  2021Jun15{]}. Available from:
  https://www.huffpost.com/entry/cdc-testing-guidelines-coronavirus-no-symptoms\_n\_5f4667fac5b64f17e136efd2
\item
  FDA 2020. Potential for False Positive Results with SARS-CoV-2 Antigen
  Tests {[}Internet{]}. U.S. Food and Drug Administration. FDA; {[}cited
  2021Jun15{]}. Available from:
  https://www.fda.gov/medical-devices/letters-health-care-providers/potential-false-positive-results-antigen-tests-rapid-detection-sars-cov-2-letter-clinical-laboratory?utm\_medium=email\&utm\_source=govdelivery
\item
  Flender S. The false positive paradox {[}Internet{]}. Medium. Towards
  Data Science; 2019 {[}cited 2021Jun15{]}. Available from:
  https://towardsdatascience.com/the-false-positive-paradox-f86448a524bc
\item
  Lee, S. Testing for SARS-CoV-2 in cellular components by routine
  nested RT-PCR followed by DNA sequencing. International Journal of
  Geriatrics and Rehabilitation 2(1):69- 96, July 17, 2020.
\item
  Madrigal A, Meyer R. Why Trump's Rapid-Testing Plan Worries Scientists
  {[}Internet{]}. The Atlantic. Atlantic Media Company; 2020 {[}cited
  2021Jun21{]}. Available from:
  https://www.theatlantic.com/health/archive/2020/10/do-rapid-antigen-tests-have-accuracy-problem/616681/
\item
  Mercer TR, Salit M. Testing at scale during the COVID-19 pandemic.
  Nature Reviews Genetics. 2021 May 4.
\item
  Sadoff J, Gray G, Vandebosch A, Cárdenas V, Shukarev G, Grinsztejn B,
  Goepfert PA, Truyers C, Fennema H, Spiessens B, Offergeld K, Scheper
  G, Taylor KL, Robb ML, Treanor J, Barouch DH, Stoddard J, Ryser MF,
  Marovich MA, Neuzil KM, Corey L, Cauwenberghs N, Tanner T, Hardt K,
  Ruiz-Guiñazú J, Le Gars M, Schuitemaker H, Van Hoof J, Struyf F,
  Douoguih M; ENSEMBLE Study Group. Safety and Efficacy of Single-Dose
  Ad26.COV2.S Vaccine against Covid-19. N Engl J Med. 2021 Jun
  10;384(23):2187-2201. doi: 10.1056/NEJMoa2101544. Epub 2021 Apr 21.
  PMID: 33882225.
\item
  Skittrall JP, Fortune MD, Jalal H, Zhang H, Enoch DA, Brown NM, et al.
  Diagnostic tool or screening programme? Asymptomatic testing for
  SARS-CoV-2 needs clear goals and protocols. The Lancet Regional Health
  - Europe. 2021;1:100002.
\item
  Temple-Raston D. CDC Report: Officials Knew Coronavirus Test Was
  Flawed But Released It Anyway {[}Internet{]}. NPR. NPR; 2020 {[}cited
  2021Jun15{]}. Available from:
  https://www.npr.org/2020/11/06/929078678/cdc-report-officials-knew-coronavirus-test-was-flawed-but-released-it-anyway
\item
  Voysey M, Clemens SAC, Madhi SA, Weckx LY, Folegatti PM, Aley PK,
  Angus B, Baillie VL, Barnabas SL, Bhorat QE, Bibi S, Briner C, Cicconi
  P, Collins AM, Colin-Jones R, Cutland CL, Darton TC, Dheda K, Duncan
  CJA, Emary KRW, Ewer KJ, Fairlie L, Faust SN, Feng S, Ferreira DM,
  Finn A, Goodman AL, Green CM, Green CA, Heath PT, Hill C, Hill H,
  Hirsch I, Hodgson SHC, Izu A, Jackson S, Jenkin D, Joe CCD, Kerridge
  S, Koen A, Kwatra G, Lazarus R, Lawrie AM, Lelliott A, Libri V, Lillie
  PJ, Mallory R, Mendes AVA, Milan EP, Minassian AM, McGregor A,
  Morrison H, Mujadidi YF, Nana A, O'Reilly PJ, Padayachee SD, Pittella
  A, Plested E, Pollock KM, Ramasamy MN, Rhead S, Schwarzbold AV, Singh
  N, Smith A, Song R, Snape MD, Sprinz E, Sutherland RK, Tarrant R,
  Thomson EC, Török ME, Toshner M, Turner DPJ, Vekemans J, Villafana TL,
  Watson MEE, Williams CJ, Douglas AD, Hill AVS, Lambe T, Gilbert SC,
  Pollard AJ; Oxford COVID Vaccine Trial Group. Safety and efficacy of
  the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim
  analysis of four randomised controlled trials in Brazil, South Africa,
  and the UK. Lancet. 2021 Jan 9;397(10269):99-111. doi:
  10.1016/S0140-6736(20)32661-1. Epub 2020 Dec 8. Erratum in: Lancet.
  2021 Jan 9;397(10269):98. PMID: 33306989; PMCID: PMC7723445.
\item
  Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result.
  BMJ. 2020;:m1808. Online at
  https://www.bmj.com/content/369/bmj.m1808/infographic.
\end{enumerate}

\selectlanguage{english}
\FloatBarrier
\end{document}

