Strengths and Limitations
In addition to assisting trialists with selecting a holistic set of
outcomes while designing clinical trials, the generated heatmap serves
as an easy-to-interpret visual illustration that can be appended to any
systematic review, giving readers quick insight into what outcomes are
not properly represented by the studies of interest. This will assist
systematic reviewers and decision-makers with assessing the quality of
outcome reporting, in tandem with the assessment of the conduct of
clinical trials, thereby enabling them to make more informed assessments
regarding new interventions. The main limitation to the tool is that it
involves an element of subjectivity to determine what constitutes
comprehensiveness. Since it is the responsibility of the user to define
what the standard is, through a guided process on the ‘Master Sheet’ of
the tool, the heatmap output must be interpreted in the context of the
user’s defined benchmarks. To ensure the identification of outcomes is
truly relevant, it is recommended that the tool is used by individuals
with expertise in the conditions they are investigating and have
extensive insight into what outcomes would be important in making
tangible changes. This level of subjectivity is not exclusive to our
tool, however, and is a fairly consistent attribute of many assessment
schemes, such as Cochrane’s ROB tool34 and the
Newcastle-Ottawa Scale for assessing the quality of observational
studies18. This limitation notwithstanding, the COR
tool fills a unique void that is not addressed by recommended ROB tools,
and unlikely to be addressed by the development of COS.