7 Deadly Sins: Proposed Guidelines for Reporting Clinical Trials Methodology Research
Danielle Popp, PhD; Janet B. W. Williams, PhD; and Michael J. Detke, MD, PhD
From MedAvante, Inc, Hamilton, New Jersey (Dr Popp); College of Physicians and Surgeons, Columbia University, New York, New York (Dr Williams); and Indiana University School of Medicine, Indianapolis (Dr Detke).
This poster presentation was supported by MedAvante, Inc.
Background: Clinical trial failure rates in several disease areas are about 50%. Many methodological approaches for increasing signal detection have been proposed but reporting variations deter comparisons. Standardization will alleviate this and reduce reporting bias in studies evaluating clinical trial methodology (CTM).
Methods: We propose 7 guidelines for standardized reporting and illustrate how misuse or omission influences interpretation of results.
Results: Report interrater reliability (IRR). IRR should be reported in studies with multiple raters and observations. For severity assessments, the ICC is required to accurately assess IRR. If individual items from a single severity scale are treated as separate observations, ICCs may be inversely related to IRR.
Use appropriate statistical tests. Kappa does not capture concordance on continuous variables. Using a fixed criterion to indicate rater agreement with a “gold standard” score can inflate reports of IRR.
Include effect size measures. Measures of effect size should be reported regardless of statistical significance to allow readers to determine clinical relevance, regardless of sample size, and compare both within and across studies.
Identify a priori and post-hoc analyses. Methodological comparisons should be identified a priori or reported as post-hoc. Failure to do so can lead to over-interpretation of exploratory analyses performed on small subsets of data.
Acknowledge and correct for multiple comparisons. All analyses should be reported if multiple comparisons are performed on a single sample. Reporting a significant result on a subset of data without indicating total comparisons made across the entire data set or correcting for multiplicity may lead to over-interpretation of false positives.
Include inferential statistics for means comparisons. Statements concerning differences or patterns in means should be substantiated with inferential statistics.
Correct interpretation of null hypothesis testing (NHT). NHT is commonly misinterpreted, such as concluding that smaller p values indicate more important effects, or that a non-significant p value represents a finding of no difference.
Conclusions: Guidelines for standardized reporting of methodology research may reduce reporting bias. Empirical research evaluating effectiveness of new methods to increase signal detection holds important consequences for CTM.
Understand importance of standardized reporting
Identify information necessary to compare results across methodologies
Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. J Pharmacol Pharmacother. 2010;1(2):100–107. PubMed
Müller MJ, Szegedi A. Effect of interrater reliability of psychopathologic assessment on power and sample size calculations in clinical trials. J Clin Psychopharmacol. 2002;22(3):318–325. PubMed