Machine learning-based diagnostic test accuracy research: measurement indicators_Chinese Journal of Evidence-Based Medicine

Authors：

XIONG Yutao ¹ , ZHONG Chenglan ² , ZENG Wei ¹ , GUO Jixiang ² ,  ZHANG Tao ³ , HUANG Yan ⁴ , TANG Wei ¹ ,  LIU Chang ^1,4

1. State Key Laboratory of Oral Diseases & National Clinical Research Center for Oral Diseases & Department of Oral and Maxillofacial Surgery & Department of Evidence-Based Dentistry & "Medicine + Manufacturing" Center, West China Hospital of Stomatology, Sichuan University, Chengdu 610041, P. R. China;
2. Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China;
3. Department of Epidemiology and Health Statistics, West China School of Public Health, Sichuan University, Chengdu 610041, P. R. China;
4. Department of Information Management & Department of Dental Informatics, West China Hospital of Stomatology, Sichuan University, Chengdu 610041, P. R. China;

Corresponding author：

ZHANG Tao, Email: scdxzhangtao@163.com; LIU Chang, Email: liu_chang_92@sina.com

Keywords：

Diagnostic test accuracy study; Artificial intelligence; Machine learning; Deep learning; Descriptive statistics; Inferential statistics

DOI：

10.7507/1672-2531.202302048

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Machine learning-based diagnostic tests have certain differences of measurement indicators with traditional diagnostic tests. In this paper, we elaborate the definitions, calculation methods and statistical inferences of common measurement indicators of machine learning-based diagnosis models in detail. We hope that this paper will be helpful for clinical researchers to better evaluate machine learning diagnostic models.

Citation： XIONG Yutao, ZHONG Chenglan, ZENG Wei, GUO Jixiang, ZHANG Tao, HUANG Yan, TANG Wei, LIU Chang. Machine learning-based diagnostic test accuracy research: measurement indicators. Chinese Journal of Evidence-Based Medicine, 2023, 23(8): 963-969. doi: 10.7507/1672-2531.202302048 Copy

1.	吴景玲, 葛龙, 张俊华, 等. 多个诊断性试验准确性的比较: 网状Meta分析方法介绍. 中国循证医学杂志, 2017, 17(8): 987-992.
2.	Bahar RC, Merkaj S, Cassinelli Petersen GI, et al. Machine learning models for classifying high- and low-grade gliomas: a systematic review and quality of reporting analysis. Front Oncol, 2022, 12: 856231.
3.	Majanga V, Viriri S. Dental images' segmentation using threshold connected component analysis. Comput Intell Neurosci, 2021, 2021: 2921508.
4.	Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol, 2004, 11(2): 178-189.
5.	Thiel-Egenter C, Alvarez N, Holderegger R, et al. Break zones in the distributions of alleles and species in alpine plants. J Biogeogr, 2011, 38(4): 772-782.
6.	Yuan Y, Chao M, Lo YC. Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging, 2017, 36(9): 1876-1886.
7.	Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans Pattern Anal Mach Intell, 2015, 37(11): 2153-2163.
8.	Yao J, Zeng W, He T, et al. Automatic localization of cephalometric landmarks based on convolutional neural network. Am J Orthod Dentofacial Orthop, 2022, 161(3): 250-259.
9.	Böhning D, Holling H, Patilea V. A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Stat Methods Med Res, 2011, 20(5): 541-550.
10.	Lian L, Zhu T, Zhu F, et al. Deep learning for caries detection and classification. Diagnostics (Basel), 2021, 11(9): 1672.
11.	Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 1989, 16(4): 561-568.
12.	Everingham M, van Gool L, Williams C, et al. The PASCAL visual object classes challenge 2007.
13.	Everingham M, van Gool L, Williams C, et al. Visual object classes challenge 2010 (VOC2010).
14.	Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977, 33(1): 159-174.
15.	纪守领, 杜天宇, 邓水光, 等. 深度学习模型鲁棒性研究综述. 计算机学报, 2022, 45(1): 190-206.
16.	Castelvecchi D. Can we open the black box of AI. Nature, 2016, 538(7623): 20-23.
17.	杨桂元. 中心极限定理及其在统计分析中的应用. 统计与信息论坛, 2000, (3): 13-15.

1. 吴景玲, 葛龙, 张俊华, 等. 多个诊断性试验准确性的比较: 网状Meta分析方法介绍. 中国循证医学杂志, 2017, 17(8): 987-992.
2. Bahar RC, Merkaj S, Cassinelli Petersen GI, et al. Machine learning models for classifying high- and low-grade gliomas: a systematic review and quality of reporting analysis. Front Oncol, 2022, 12: 856231.
3. Majanga V, Viriri S. Dental images' segmentation using threshold connected component analysis. Comput Intell Neurosci, 2021, 2021: 2921508.
4. Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol, 2004, 11(2): 178-189.
5. Thiel-Egenter C, Alvarez N, Holderegger R, et al. Break zones in the distributions of alleles and species in alpine plants. J Biogeogr, 2011, 38(4): 772-782.
6. Yuan Y, Chao M, Lo YC. Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging, 2017, 36(9): 1876-1886.
7. Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans Pattern Anal Mach Intell, 2015, 37(11): 2153-2163.
8. Yao J, Zeng W, He T, et al. Automatic localization of cephalometric landmarks based on convolutional neural network. Am J Orthod Dentofacial Orthop, 2022, 161(3): 250-259.
9. Böhning D, Holling H, Patilea V. A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Stat Methods Med Res, 2011, 20(5): 541-550.
10. Lian L, Zhu T, Zhu F, et al. Deep learning for caries detection and classification. Diagnostics (Basel), 2021, 11(9): 1672.
11. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 1989, 16(4): 561-568.
12. Everingham M, van Gool L, Williams C, et al. The PASCAL visual object classes challenge 2007.
13. Everingham M, van Gool L, Williams C, et al. Visual object classes challenge 2010 (VOC2010).
14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977, 33(1): 159-174.
15. 纪守领, 杜天宇, 邓水光, 等. 深度学习模型鲁棒性研究综述. 计算机学报, 2022, 45(1): 190-206.
16. Castelvecchi D. Can we open the black box of AI. Nature, 2016, 538(7623): 20-23.
17. 杨桂元. 中心极限定理及其在统计分析中的应用. 统计与信息论坛, 2000, (3): 13-15.

Previous Article
Evaluation of clinical guidelines and consensus on the quality of central venous catheters
Next Article
Current status and challenges of living guidelines

Chinese Journal of Evidence-Based Medicine

Machine learning-based diagnostic test accuracy research: measurement indicators

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content