Evaluation of the accuracy of the large language model for risk of bias assessment in analytical studies_Chinese Journal of Evidence-Based Medicine

Authors：

SONG Tian , GUO Jing , LIU Bin , GONG Zhaoyuan , HU Mingzhi , CHEN Lin ,  CHE Qianzi ,  SHI Nannan

Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, P. R. China;

Corresponding author：

CHE Qianzi, Email: cheqianzi123@126.com; SHI Nannan, Email: 13811839164@vip.126.com

Keywords：

Large language models (LLMs); Risk of bias; Analytical studies; Automatic analysis; Meta-analysis

DOI：

10.7507/1672-2531.202411028

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To systematically review the accuracy and consistency of large language models (LLMs) in assessing risk of bias in analytical studies. Methods The cohort and case-control studies related to COVID-19 based on the team's published systematic review of clinical characteristics of COVID-19 were included. Two researchers independently screened the studies, extracted data, and assessed risk of bias of the included studies with the LLM-based BiasBee model (version Non-RCT) used for automated evaluation. Kappa statistics and score differences were used to analyze the agreement between LLM and human evaluations, with subgroup analysis for Chinese and English studies. Results A total of 210 studies were included. Meta-analysis showed that LLM scores were generally higher than those of human evaluators, particularly in representativeness of exposed cohorts (△=0.764) and selection of external controls (△=0.109). Kappa analysis indicated slight agreement in items such as exposure assessment (κ=0.059) and adequacy of follow-up (κ=0.093), while showing significant discrepancies in more subjective items, such as control selection (κ=−0.112) and non-response rate (κ=−0.115). Subgroup analysis revealed higher scoring consistency for LLMs in English-language studies compared to that of Chinese-language studies. Conclusion LLMs demonstrate potential in risk of bias assessment; however, notable differences remain in more subjective tasks. Future research should focus on optimizing prompt engineering and model fine-tuning to enhance LLM accuracy and consistency in complex tasks.

1.	Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis. Hoboken: Wiley, 2009.
2.	Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
3.	Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.
4.	Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
5.	郭玉杰, 张雪芹, 孙文宇, 等. 自动化文献筛选工具在系统评价中的应用. 协和医学杂志, 2024, 15(4): 921-926.
6.	Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel), 2023, 11(6): 887.
7.	Hasan B, Saadi S, Rajjoub NS, et al. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment. BMJ Evid Based Med, 2024, 29(6): 394-398.
8.	Lai H, Ge L, Sun M, et al. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open, 2024, 7(5): e2412687.
9.	Feng H, Du S, Zhu G, et al. Leveraging large language models for automated Chinese essay scoring. Cham: Springer, 2024.
10.	Song T, Guo J, Liu B, et al. Trends in symptom prevalence and sequential onset of SARS-CoV-2 infection from 2020 to 2022 in East and Southeast Asia: a trajectory pattern exploration based on summary data. Arch Public Health, 2024, 82(1): 125.
11.	Huang H, Tang T, Zhang D, et al. Not all languages are created equal in LLMs: improving multilingual capability by cross-lingual-thought prompting. 2023.
12.	Tian S, Qianzi C, Ning L, et al. The distribution pattern of symptoms among different variants of COVID-19. 2023.
13.	Safary Official. Safary - streamlining automation for all systematic reviews and analysis. 2024.
14.	Treviño-Juarez AS. Assessing risk of bias using ChatGPT-4 and Cochrane ROB2 tool. Med Sci Educ, 2024, 34(3): 691-694.
15.	Roberts RH, Ali SR, Hutchings HA, et al. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform, 2023, 30(1): e100830.
16.	Issaiy M, Ghanaati H, Kolahi S, et al. Methodological insights into ChatGPT's screening performance in systematic reviews. BMC Med Res Methodol, 2024, 24(1): 78.
17.	Zhai X, Nyaaba M, Ma W. Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science? Sci Educ (Dordr), 2024.
18.	Christopher JR, Martin R, Julia B, et al. Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers. 2023.
19.	杨智荣, 孙凤, 詹思延. 偏倚风险评估系列: (一)概述. 中华流行病学杂志, 2017, 38(7): 983-987.
20.	Mirbabaie M, Stieglitz S, Frick NRJ. Artificial intelligence in disease diagnostics: a critical review and classification on the current state of research guiding future direction. Health Technol (Berl), 2021, 11(4): 693-731.
21.	Ho N, Schmid L, Yun SY. Large language models are reasoning teachers. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
22.	Ranganath K, Piyush K, Omesh T. Enhancing trust in large language models with uncertainty-aware fine-tuning. The Thirteenth International Conference on Learning Representations. 2024.
23.	Zhang J, Muhamed A, Anantharaman A, et al. ReAugKD: retrieval-augmented knowledge distillation for pre-trained language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
24.	Zhang S, Liang Y, Wang S, et al. Towards understanding and improving knowledge distillation for neural machine translation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
25.	Ortega L. Understanding second language acquisition. 2014.
26.	Hanauer DI, Sheridan CL, Englander K. Linguistic injustice in the writing of research articles in English as a second language: data from Taiwanese and Mexican Researchers. Writ Commun, 2019, 36(1): 136-154.
27.	Khurana D, Koli A, Khatter K, et al. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl, 2023, 82(3): 3713-3744.
28.	Clark K, Khandelwal U, Levy O, et al. What does bert look at? An analysis of BERT’s attention. 2019.

1. Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis. Hoboken: Wiley, 2009.
2. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
3. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.
4. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
5. 郭玉杰, 张雪芹, 孙文宇, 等. 自动化文献筛选工具在系统评价中的应用. 协和医学杂志, 2024, 15(4): 921-926.
6. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel), 2023, 11(6): 887.
7. Hasan B, Saadi S, Rajjoub NS, et al. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment. BMJ Evid Based Med, 2024, 29(6): 394-398.
8. Lai H, Ge L, Sun M, et al. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open, 2024, 7(5): e2412687.
9. Feng H, Du S, Zhu G, et al. Leveraging large language models for automated Chinese essay scoring. Cham: Springer, 2024.
10. Song T, Guo J, Liu B, et al. Trends in symptom prevalence and sequential onset of SARS-CoV-2 infection from 2020 to 2022 in East and Southeast Asia: a trajectory pattern exploration based on summary data. Arch Public Health, 2024, 82(1): 125.
11. Huang H, Tang T, Zhang D, et al. Not all languages are created equal in LLMs: improving multilingual capability by cross-lingual-thought prompting. 2023.
12. Tian S, Qianzi C, Ning L, et al. The distribution pattern of symptoms among different variants of COVID-19. 2023.
13. Safary Official. Safary - streamlining automation for all systematic reviews and analysis. 2024.
14. Treviño-Juarez AS. Assessing risk of bias using ChatGPT-4 and Cochrane ROB2 tool. Med Sci Educ, 2024, 34(3): 691-694.
15. Roberts RH, Ali SR, Hutchings HA, et al. Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards. BMJ Health Care Inform, 2023, 30(1): e100830.
16. Issaiy M, Ghanaati H, Kolahi S, et al. Methodological insights into ChatGPT's screening performance in systematic reviews. BMC Med Res Methodol, 2024, 24(1): 78.
17. Zhai X, Nyaaba M, Ma W. Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science? Sci Educ (Dordr), 2024.
18. Christopher JR, Martin R, Julia B, et al. Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers. 2023.
19. 杨智荣, 孙凤, 詹思延. 偏倚风险评估系列: (一)概述. 中华流行病学杂志, 2017, 38(7): 983-987.
20. Mirbabaie M, Stieglitz S, Frick NRJ. Artificial intelligence in disease diagnostics: a critical review and classification on the current state of research guiding future direction. Health Technol (Berl), 2021, 11(4): 693-731.
21. Ho N, Schmid L, Yun SY. Large language models are reasoning teachers. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
22. Ranganath K, Piyush K, Omesh T. Enhancing trust in large language models with uncertainty-aware fine-tuning. The Thirteenth International Conference on Learning Representations. 2024.
23. Zhang J, Muhamed A, Anantharaman A, et al. ReAugKD: retrieval-augmented knowledge distillation for pre-trained language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
24. Zhang S, Liang Y, Wang S, et al. Towards understanding and improving knowledge distillation for neural machine translation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
25. Ortega L. Understanding second language acquisition. 2014.
26. Hanauer DI, Sheridan CL, Englander K. Linguistic injustice in the writing of research articles in English as a second language: data from Taiwanese and Mexican Researchers. Writ Commun, 2019, 36(1): 136-154.
27. Khurana D, Koli A, Khatter K, et al. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl, 2023, 82(3): 3713-3744.
28. Clark K, Khandelwal U, Levy O, et al. What does bert look at? An analysis of BERT’s attention. 2019.

Chinese Journal of Evidence-Based Medicine

Latest ArticlesEvaluation of the accuracy of the large language model for risk of bias assessment in analytical studies

Abstract Full text Figures/Tables Video References Cited by

Format

Content