The regulators have accepted non-randomized studies as the
basis for licensing of new treatments
A notion that that large treatment effects can sometimes obviate the
need for data from RCTs has also been accepted by the FDA and the EMA.
The agencies established special pathways-the EMA’s PRIME (Priority
Medicines) and Adaptive Pathways programs
and the FDA’s “Breakthrough
Therapy Designation” programs - designed to support approval of drugs
that demonstrate substantial improvement over the existing therapies,
which may not require further testing in RCTs. 14 We
recently analyzed drug approvals by the EMA and FDA based on
non-randomized drug comparisons and confirmed that larger effect sizes
in non-randomized studies are associated with higher rates of licensing
approval.15,16 Overall, we found that between 7 to
10% of drug approvals are based on non-RCT comparisons and that between
2% and 4% of these approvals displayed “dramatic” effects15,16, defined as relative risks (RR)>28, RR ≥5 17, or
RR≥10.9
Although the probability of approval increased with larger effect sizes,
we could not identify a specific threshold of effect size above which
the regulatory agencies would always grant licensing approval.15,16 Similarly, to date, the agencies have not
formally quantified the effect size necessary to preclude the need for
additional testing in RCTs before granting licensing approval. Thus, the
treatment effect size that is dramatic enough to convince the FDA and
EMA that the treatment differences observed are real and free of bias
and random error and hence can be approved without RCTs remain unclear.
The proposed definitions of dramatic effects- (RR)>28, RR ≥5 17, or RR≥109- have not been theoretically or empirically
justified; instead, they represent a decision rule based onheuristics - powerful, rule-of-thumb, decision-making strategies
that are often more accurate than complex statistical
models.10,11,18
The heuristic theory of decision-making has been linked to SDT12,13 and the threshold model of
decision-making12 to show how seemingly unrelated
theories in different disciplines can lead to discovery of new
relationships and explanations. Here, we argue that when the direct,
empirical evaluation of a treatment effect is not possible, an
alternative approach is to employ SDT to define the circumstances under
which the “signal” (e.g., treatment effect) is credible and
reproducibly detected to allow approval of new drugs without further
testing in RCTs.19,20 21
We rely on the generalizability of SDT to account for two heuristics
that we believe influence the FDA and the EMA’s approval of new
treatments: 1) the likelihood of approval without testing in RCTs will
increase if the difference in treatment effect between the experimental
and control arm is at least one logarithm of magnitude (i.e., reflecting
heuristic based on the Weber-Fechner law ) 2021 22; 2) the specific threshold of
effect size above which the agencies will not require further RCTs will
reflect heuristic known as recognition
heuristic .23 We focus on the effect size heuristics,
but also discuss the heuristic related to the use of p-values. Recently,
there has been an increasing attempt to modify a century-old inferential
rule-of-thumb to reject the null hypothesis at p≤0.0524; some authors vehemently oppose a hard cut-off for
p-values, 25 while others propose a new heuristic rule
of p≤0.005 24 as an acceptable evidentiary standard
against a null hypothesis.