Introduction to the sample size estimation methods of multi-reader and multi-case design in radiological diagnostic test and software implementation_Chinese Journal of Evidence-Based Medicine

Authors：

WAN Huiqin ¹ , XIANG Man ¹ , PAN Zhemin ¹ , QIN Yingyi ² , HE Qian ² ,  HE Jia ^1,2

1. Tongji University School of Medicine, Shanghai 200092, P. R. China;
2. Department of Health Statistics, Naval Medical University, Shanghai 200433, P. R. China;

Corresponding author：

HE Jia, Email: hejia63@yeah.net

Keywords：

Radiological diagnostic test; Multi-reader multi-case study; OR method; DBM method; Sample size estimation

DOI：

10.7507/1672-2531.202407148

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To explore two methods of sample size estimation in multi-reader multi-case study of radiological diagnostic test and realize them by software. Methods Demonstration programs were conducted in R software using the Van Dyke dataset, calculating combinations of readers and cases using the OR and DBM methods. These serve as pilot test results for multi-reader multi-case studies, providing a reference for parameter settings in subsequent formal experiments. Results When the effect size was 0.044, 6 readers and 247 cases could yield 0.80 power, while with an effect size of 0.088, only 6 readers and 44 cases were needed to reach 80.5% power. The sample sizes calculated using the OR method and the DBM method were consistent, and the same sample size calculation results could be obtained through conversion between the two methods. Conclusion For the estimation of sample size in multi-reader multi-case studies, R software provides a convenient and mature software package for sample size estimation using multi-reader multi-case designs in radiological diagnostic tests, thereby offering a reference for selecting appropriate sample size estimation and statistical analysis methods in radiological diagnostic tests.

1.	贾凯丽, 王雪梅. 医学影像人工智能新进展. 国际放射医学核医学杂志, 2020, 44(1): 27-31.
2.	金征宇. 人工智能医学影像应用: 现实与挑战. 放射学实践, 2018, 33(10): 989-991.
3.	Obuchowski NA, Rockette HE. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 1995, 24(2): 285-308.
4.	尚美霞, 阎小妍, 李雪迎, 等. 采用多阅片者多病例设计评估AI辅助医疗产品临床试验的样本量估算和应用. 中国卫生统计, 2022, 39(1): 14-18.
5.	Malik B, Iuanow E, Klock J. An exploratory multi-reader, multi-case study comparing transmission ultrasound to mammography on recall rates and detection rates for breast cancer lesions. Acad Radiol, 2022, (Suppl 1): S10-S18.
6.	Platisa L, Vansteenkiste E, Goossens B, et al. Optimization of medical imaging display systems: using the channelized hotelling observer for detecting lung nodules - experimental study. Proceedings of SPIE Medical Imaging, 2009.
7.	Han PL, Jiang L, Cheng JL, et al. Artificial intelligence-assisted diagnosis of congenital heart disease and associated pulmonary arterial hypertension from chest radiographs: a multi-reader multi-case study. Eur J Radiol, 2024, 171: 111277.
8.	尚美霞, 姚晨, 阎小妍, 等. 影像诊断试验中多阅片者研究的设计与分析. 中国卫生统计, 2014, 31(2): 331-335.
9.	Obuchowski NA, Bullen J. Multireader diagnostic accuracy imaging studies: fundamentals of design and analysis. Radiology, 2022, 303(1): 26-34.
10.	FDA. Clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device applied to radiology images and radiology device data in premarket notification (510(k)) submissions: guidance for Industry and Food and Drug Administration Staff.
11.	国家药品监督管理局医疗器械技术审评中心. 深度学习辅助决策医疗器械软件审评要点(2019年第7号). 2019.
12.	国家药品监督管理局. 乳腺X射线系统注册技术审查指导原则(2021年第42号). 2021.
13.	Hillis SL, Obuchowski NA, Berbaum KS. Power estimation for multireader ROC methods an updated and unified approach. Acad Radiol, 2011, 18(2): 129-142.
14.	Wang L, Wang H, Xia C, et al. Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products. Expert Rev Med Devices, 2020, 17(9): 899-918.
15.	Gallas BD, Chan HP, D'Orsi CJ, et al. Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 2012, 19(4): 463-477.
16.	Hillis SL, Obuchowski NA, Schartz KM, et al. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med, 2005, 24(10): 1579-1607.
17.	Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol, 2008, 15(5): 647-661.
18.	Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med, 2007, 26(3): 596-619.
19.	Lenth RV. Some practical guidelines for effective sample size determination. Am Stat. 2001, 55(3): 187-193.
20.	Basol M, Goksuluk D, Karaagaoglu E. Comparing the diagnostic performance of methods used in a full-factorial design multi-reader multi-case studies. Comput Stat, 2023, 38(3): 1537-1553.
21.	Obuchowski NA. Multi-reader multi-modality ROC studies: hypothesis testing and sample size estimation using an ANOVA approach with dependent observations. With rejoinder. Acad Radiol, 1995, 2(6): 522-529.
22.	Efron B, Tibshirani RJ. An introduction to the bootstrap. statistics and applied probability. Chapman & Hall/CRC, 1993.
23.	Delong ER, Delong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 1988, 44(3): 837-845.
24.	Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol, 1992, 27(9): 723-731.
25.	Chen W, Petrick NA, Sahiner B. Hypothesis testing in noninferiority and equivalence MRMC ROC studies. Acad Radiol, 2012, 19(9): 1158-1165.
26.	Obuchowski NA. Sample size tables for receiver operating characteristic studies. AJR Am J Roentgenol, 2000, 175(3): 603-608.
27.	Hillis SL. A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Stat Med, 2014, 33(2): 330-360.
28.	Van Dyke CW. Cine MRI in the diagnosis of thoracic aortic dissection. 79th RSNA Meetings. 1993.
29.	Dorfman DD, Berbaum KS, Lenth RV, et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol, 1998, 5(9): 591-602.
30.	Hillis SL, Schartz KM. Multireader sample size program for diagnostic studies: demonstration and methodology. J Med Imaging (Bellingham), 2018, 5(4): 045503.
31.	Yasaka K, Akai H, Kunimatsu A, et al. Deep learning for staging liver fibrosis on CT: a pilot study. Eur Radiol, 2018, 28(11): 4578-4585.

1. 贾凯丽, 王雪梅. 医学影像人工智能新进展. 国际放射医学核医学杂志, 2020, 44(1): 27-31.
2. 金征宇. 人工智能医学影像应用: 现实与挑战. 放射学实践, 2018, 33(10): 989-991.
3. Obuchowski NA, Rockette HE. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 1995, 24(2): 285-308.
4. 尚美霞, 阎小妍, 李雪迎, 等. 采用多阅片者多病例设计评估AI辅助医疗产品临床试验的样本量估算和应用. 中国卫生统计, 2022, 39(1): 14-18.
5. Malik B, Iuanow E, Klock J. An exploratory multi-reader, multi-case study comparing transmission ultrasound to mammography on recall rates and detection rates for breast cancer lesions. Acad Radiol, 2022, (Suppl 1): S10-S18.
6. Platisa L, Vansteenkiste E, Goossens B, et al. Optimization of medical imaging display systems: using the channelized hotelling observer for detecting lung nodules - experimental study. Proceedings of SPIE Medical Imaging, 2009.
7. Han PL, Jiang L, Cheng JL, et al. Artificial intelligence-assisted diagnosis of congenital heart disease and associated pulmonary arterial hypertension from chest radiographs: a multi-reader multi-case study. Eur J Radiol, 2024, 171: 111277.
8. 尚美霞, 姚晨, 阎小妍, 等. 影像诊断试验中多阅片者研究的设计与分析. 中国卫生统计, 2014, 31(2): 331-335.
9. Obuchowski NA, Bullen J. Multireader diagnostic accuracy imaging studies: fundamentals of design and analysis. Radiology, 2022, 303(1): 26-34.
10. FDA. Clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device applied to radiology images and radiology device data in premarket notification (510(k)) submissions: guidance for Industry and Food and Drug Administration Staff.
11. 国家药品监督管理局医疗器械技术审评中心. 深度学习辅助决策医疗器械软件审评要点(2019年第7号). 2019.
12. 国家药品监督管理局. 乳腺X射线系统注册技术审查指导原则(2021年第42号). 2021.
13. Hillis SL, Obuchowski NA, Berbaum KS. Power estimation for multireader ROC methods an updated and unified approach. Acad Radiol, 2011, 18(2): 129-142.
14. Wang L, Wang H, Xia C, et al. Toward standardized premarket evaluation of computer aided diagnosis/detection products: insights from FDA-approved products. Expert Rev Med Devices, 2020, 17(9): 899-918.
15. Gallas BD, Chan HP, D'Orsi CJ, et al. Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 2012, 19(4): 463-477.
16. Hillis SL, Obuchowski NA, Schartz KM, et al. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Stat Med, 2005, 24(10): 1579-1607.
17. Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol, 2008, 15(5): 647-661.
18. Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med, 2007, 26(3): 596-619.
19. Lenth RV. Some practical guidelines for effective sample size determination. Am Stat. 2001, 55(3): 187-193.
20. Basol M, Goksuluk D, Karaagaoglu E. Comparing the diagnostic performance of methods used in a full-factorial design multi-reader multi-case studies. Comput Stat, 2023, 38(3): 1537-1553.
21. Obuchowski NA. Multi-reader multi-modality ROC studies: hypothesis testing and sample size estimation using an ANOVA approach with dependent observations. With rejoinder. Acad Radiol, 1995, 2(6): 522-529.
22. Efron B, Tibshirani RJ. An introduction to the bootstrap. statistics and applied probability. Chapman & Hall/CRC, 1993.
23. Delong ER, Delong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 1988, 44(3): 837-845.
24. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol, 1992, 27(9): 723-731.
25. Chen W, Petrick NA, Sahiner B. Hypothesis testing in noninferiority and equivalence MRMC ROC studies. Acad Radiol, 2012, 19(9): 1158-1165.
26. Obuchowski NA. Sample size tables for receiver operating characteristic studies. AJR Am J Roentgenol, 2000, 175(3): 603-608.
27. Hillis SL. A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Stat Med, 2014, 33(2): 330-360.
28. Van Dyke CW. Cine MRI in the diagnosis of thoracic aortic dissection. 79th RSNA Meetings. 1993.
29. Dorfman DD, Berbaum KS, Lenth RV, et al. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol, 1998, 5(9): 591-602.
30. Hillis SL, Schartz KM. Multireader sample size program for diagnostic studies: demonstration and methodology. J Med Imaging (Bellingham), 2018, 5(4): 045503.
31. Yasaka K, Akai H, Kunimatsu A, et al. Deep learning for staging liver fibrosis on CT: a pilot study. Eur Radiol, 2018, 28(11): 4578-4585.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesIntroduction to the sample size estimation methods of multi-reader and multi-case design in radiological diagnostic test and software implementation

Abstract Full text Figures/Tables Video References Cited by

Format

Content