J Stroke.  2022 Jan;24(1):49-56. 10.5853/jos.2021.02740.

The Challenge of Designing Stroke Trials That Change Practice: MCID vs. Sample Size and Pragmatism

• Goyal M
• McDonough R
• Fisher M
• Ospel J

• KMID: 2525330
• DOI: http://doi.org/10.5853/jos.2021.02740

Abstract

Randomized controlled trials (RCT) are the basis for evidence-based acute stroke care. For an RCT to change practice, its results have to be statistically significant and clinically meaningful. While methods to assess statistical significance are standardized and widely agreed upon, there is no clear consensus on how to assess clinical significance. Researchers often refer to the minimal clinically important difference (MCID) when describing the smallest change in outcomes that is considered meaningful to patients and leads to a change in patient management. It is widely accepted that a treatment should only be adopted when its effect on outcome is equal to or larger than the MCID. There are however situations in which it is reasonable to decide against adopting a treatment, even when its beneficial effect matches or exceeds the MCID, for example when it is resource- intensive and associated with high costs. Furthermore, while the MCID represents an important concept in this regard, defining it for an individual trial is difficult as it is highly context specific. In the following, we use hypothetical stroke trial examples to review the challenges related to MCID, sample size and pragmatic considerations that researchers face in acute stroke trials, and propose a framework for designing meaningful stroke trials that have the potential to change clinical practice.

Keyword

Ischemic stroke; Ischemia; Sample size

Figure

• 

  Figure 1. Relationship between effect size, power, and sample size. Sample size increases with decreasing effect size. Higher power (power=the probability that the trial will detect a significant difference if this difference truly exists) also requires higher sample sizes.

• 

  Figure 2. (A) Different trial result scenarios for a superiority design, using a minimal clinically important difference (MCID) of 5% as an example, showing how the observed result may be clinically meaningful, statistically significant, neither, or both. The solid vertical line represents a difference in outcomes of 0%, indicating no treatment effect. Upper row: The difference in outcomes between treatment and control arm is exactly 0, and the CI does not include the MCID. Thus, the difference is not statistically significant and shows no clinically relevant effect. Second row: The difference is approximately 6% in favor of the new treatment and the CI crosses both 0% and 5% (i.e. contains the null-effect line and the MCID). Thus, the result is not statistically significant. Whether the treatment leads to a clinically relevant difference cannot be determined since there is a chance that the treatment effect is larger than the MCID (since parts of the CI are to the right of the dashed line). Third row: The difference in outcome is approximately 3% in favor of the new treatment and the CI neither contains the MCID nor 0% (the null-effect line). Thus, the difference is statistically significant but not clinically relevant. Fourth row: The difference between the two arms is 5% in favor of the new treatment and the CI contains the MCID (vertical dashed line) but not 0% (the vertical line). Thus, the difference is statistically significant but just reaching clinical relevance, whether it is truly clinically relevant cannot be inferred from this result since parts of the confidence interval are to the left of the dashed line. Other factors such as cost will influence adoption. Lowest row: The difference between the two arms is approximately 8% and the CI neither contains the MCID nor 0% (the CI is entirely right to the MCID). Thus, the difference is statistically significant and clinically relevant. (B) Different trial result scenarios for a non-inferiority trial design, a risk ratio (RR) of 1 (vertical solid line) indicates no difference between the two treatments, and the RR should be >0.95 (between 0.95 and 1) in order for the new treatment to be clinically accepted as a valid alternative to the established treatment. Upper row: The point estimate for the RR is 1 (indicating no effect) and the CI boundary neither includes the MCID nor the non-inferiority margin (the CI is located to the right of the MCID and non-inferiority margin), indicating statistical non-inferiority and clinical acceptance of the new treatment. Second row: The point estimate for the RR is 0.90, and the CI includes the non-inferiority margin, and the MCID. Thus, statistical non-inferiority is not proven and it is unclear whether the new treatment could be a clinically acceptable alternative to the established treatment. Third row: The point estimate for the RR is 0.88 and the CI is entirely to the right of the statistical non-inferiority margin (as chosen by the investigators) and entirely to the left of the MCID. Thus, statistical non-inferiority is proven; however, the new treatment does not constitute a clinically acceptable alternative, since the difference between the treatments favors the established treatment and is larger than the MCID. Fourth row: the point estimate for the RR is 0.84 and the CI contains the statistical non-inferiority margin and is located entirely to the left of the MCID. Thus, statistical non-inferiority is not proven and the new treatment does not constitute a clinically acceptable alternative, since the difference between the treatments favors the established treatment and is larger than the MCID. Lowest row: The point estimate for the RR is 1.1 and the CI is located entirely to the right of the statistical non-inferiority margin and the MCID: Thus, statistical non-inferiority is proven and the new treatment constitutes a clinically acceptable alternative to the established treatment. The non-inferiority margin chosen by *the trialists (RR, 0.85) differs from †the MCID (RR, 0.95).

• 

  Figure 3. Hypothetical workflow for minimal clinically important difference (MCID) determination using a multidisciplinary committee that is collaborating and in close exchange with guideline committees. The trial investigators make their recommendation to the committee for approval prior commencement of the study. The committee could take input from additional experts such as physicians and healthcare policymakers, patient representatives, ethicists, etc. Following the assessment, the multidisciplinary committee would either approve the trial proposal if the conclusion is that the trial would likely change clinical practice in case of a positive result, or, if they think this is not the case, provide guidance on how to revise the trial proposal. EBM, evidence-based medicine.

• 

  Figure 4. Network meta-analysis framework using the direct-to-endovascular treatment (EVT) question as an example. In theory, every thrombolytic agent (drugs A–E) would need to be directly compared to EVT alone. Such a direct comparison may not be available for all drugs. In this example, direct comparisons with EVT alone are only available for drugs A, B and C but not for drugs D and E. However, drug D has directly been compared to drug A and drug E to drug B. Network meta-analysis take all indirect and direct evidence into account and thereby allow us to compare drugs D and E with EVT alone despite the fact that there is no trial comparing them directly.

Reference

References

1. Berkhemer OA, Fransen PS, Beumer D, van den Berg LA, Lingsma HF, Yoo AJ, et al. A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med. 2015; 372:11–20.

2. Campbell BC, Mitchell PJ, Kleinig TJ, Dewey HM, Churilov L, Yassi N, et al. Endovascular therapy for ischemic stroke with perfusion-imaging selection. N Engl J Med. 2015; 372:1009–1018.

3. Goyal M, Demchuk AM, Menon BK, Eesa M, Rempel JL, Thornton J, et al. Randomized assessment of rapid endovascular treatment of ischemic stroke. N Engl J Med. 2015; 372:1019–1030.

4. Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016; 387:1723–1731.

5. National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. Tissue plasminogen activator for acute ischemic stroke. N Engl J Med. 1995; 333:1581–1587.

6. Saver JL, Goyal M, Bonafe A, Diener HC, Levy EI, Pereira VM, et al. Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke. N Engl J Med. 2015; 372:2285–2295.

7. Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, et al. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ. 2018; 363:k3750.

8. de Vet HCW, Mokkink LB, Terwee CB. Minimal clinically important difference (MCID). In : Michalos AC, editor. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer Netherlands;2014. https://doi.org/10.1007/978-94-007-0753-5_1812.

9. Draak T, de Greef B, Faber CG, Merkies I; PeriNomS study group. The minimum clinically important difference: which direction to take. Eur J Neurol. 2019; 26:850–855.

10. Alemayehu C, Mitchell G, Nikles J. Barriers for conducting clinical trials in developing countries: a systematic review. Int J Equity Health. 2018; 17:37.

11. Chow SC, Shao J, Wang H, Lokhnygina Y. Sample Size Calculations in Clinical Research. 3rd ed. Boca Raton, FL: CRC Press;2020.

12. Moher D, Schulz KF, Altman D; CONSORT Group. The CONSORT Statement: revised recommendations for improving the quality of reports of parallel-group randomized trials 2001. Explore (NY). 2005; 1:40–45.

13. Walder B, Maillard J, Lübbeke A. Minimal clinically important difference: a novel approach to measure changes in outcome in perioperative medicine. Eur J Anaesthesiol. 2015; 32:77–78.

14. Zarin DA, Goodman SN, Kimmelman J. Harms from uninformative clinical trials. JAMA. 2019; 322:813–814.

15. D’Agostino RB Sr, Massaro JM, Sullivan LM. Non-inferiority trials: design concepts and issues. The encounters of academic consultants in statistics. Stat Med. 2003; 22:169–186.

16. Fisher M, Albers GW, Donnan GA, Furlan AJ, Grotta JC, Kidwell CS, et al. Enhancing the development and approval of acute stroke therapies: Stroke Therapy Academic Industry roundtable. Stroke. 2005; 36:1808–1813.

17. Hong KS, Ali LK, Selco SL, Fonarow GC, Saver JL. Weighting components of composite end points in clinical trials: an approach using disability-adjusted life-years. Stroke. 2011; 42:1722–1729.

18. Chaimani A, Salanti G, Leucht S, Geddes JR, Cipriani A. Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article. Evid Based Ment Health. 2017; 20:88–94.

19. Higgins JP, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods. 2012; 3:98–110.