Discussion
We sought to address one of the most important clinical question in contemporary clinical research: how large of an effect size is large enough to allow approval of treatments without further testing in RCTs? To address this question, we illustrate the application of the signal detection and heuristic theory of decision-making to interpret the effect size that regulatory agencies may use to approve treatment without further testing in RCTs. We propose that two heuristics can explain the agencies’ decision-making: first, signal isrecognized as large, and second, the magnitude of that signal is assessed via the Weber-Fechner law. Our findings suggest that when the difference between novel treatments and historical controls is at least one logarithm of magnitude, the veracity of testing in non-RCTs seems to be established.38 These findings based on the convergence of the Weber-Fechner and recognition heuristics agree with the heuristic rule suggested by Glasziou et al9: further RCTs may not be necessary when RR of experimental treatment is ≥ 10 in comparison with control.
Theories of decision making are divided into those that deal with ‘large-’ or ‘small’- world phenomena.39 In a small world, time constraint is not an issue, decision-makers have access to the best available evidence – ideally from well-designed and powered RCTs – regarding all competing management alternatives, consequences and probabilities. Signal detection theory is a prototype of the small world theories and is a normative theory that provides a framework for how people “should” or “ought to” make their decisions and draw inferences. (This is also known as the theory of“ought”). 40-42
In contrast, in a ‘large’ or real-world context, decision-makers are typically under time constraints, with limited knowledge about the complete set of alternatives, consequences, and probabilities. This means that making rational inferences requires adaptation to environment/context (adaptive or ecological rationality ) and respecting epistemological, environmental and computationalconstraints of human brains.11,40 Because finding the optimum solution to a given problem can be resource and computationally intensive, adaptive behaviors typically rely onsatisficing (finding a good enough solution), rather than striving to find a “perfect” solution (via optimising/maximizing procedures). 40,43 The principle behind satisficing is that there must exist a point (threshold) at which obtaining more information or engaging in more computation becomes overly costly and thereby detrimental. Identifying this threshold, at which a decision-maker should stop searching for more information, is often accomplished by using “heuristics”11 for implementation of bounded rationality.44 The heuristic theory of decision-making is a descriptive theory, which helps explain how people actually make their decisions (also known as theory of“is”). 40-42 Surprisingly, simple heuristic-based inferential and decision-making strategies are often more accurate than more complex statistical models (the phenomenon known as “less-is-more”).11
Recently, we12 and others13,36integrated small-world SDT with heuristics decision-making to show how connecting apparently unrelated theories in different disciplines likely leads to discovery of new relationships. In this paper, we extend the theory integration program45 to the application of the Weber-Fechner law and recognition heuristics in order to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-RCT studies without further testing in RCTs. By integrating heuristic reasoning with SDT, it is sometimes possible to derive “ought” rule from “is”observations.40-42,46 That is, if had observed high discriminability of Weber-Fechner, or recognition heuristic, we may then argue that these empirically derived observations may, in turn, be normatively used by drug developers and practitioners alike: one log effect size magnitude could serve as a benchmark to decide if further testing in RCTs should be pursued, or as a guide in interpretation of the results reported in non-RCT studies.
Throughout this study, we found some support for “one logarithm of treatment magnitude” rule, but we should acknowledge the study limitations. First, the strength of evidence supporting high accuracy related to the decision to pursue further RCTs based on the one log effect size is moderate. Second, as discussed in the papers leading to this one15,16, in addition to effect size, other factors play a role in the decision to grant licensing approval; these seem to include issues such as approval for rare diseases where few effective treatment exists, risk tolerance in the attempt to strike a balance between failing to approve effective drugs and approving ineffective or dangerous drugs 47, political pressures like conflict of interest, feasibility of undertaking of RCTs, small sample sizes and bias in the assessment of control event rates, as outlined above.
Nevertheless, it is clear that the larger the effect size, the higher the probability that treatments will be approved without further testing in RCTs. 15,16 When integration of multiple factors are difficult people resort to heuristics, which are often defining characteristics of psychology of decision-making. 10However, one of the reasons that we could not provide more definitive evidence related to the specific effect size above which drugs should be approved based solely on non-RCT data is that our database, even most comprehensive to date, is relatively small (n=134). Mere exposure to “dramatic effects” does not account for the mechanism of recognition heuristic.35 Rather, repeated experience and internalization of the rule is required for the ease of retrieval to rely on recognition for making inferences from memory about the phenomenon of interest.35 We suspect that as databases- and experience- with approval of drugs based on non-RCTs increase, regulators and practicing physicians will encounter many more instances that will help improve the quality of recognition memory and the use of the methods described here will be more applicable.