Large Language Models (LLMs) are highly sophisticated deep learning models pre-trained on massive datasets, with ChatGPT representing a prominent application of LLMs in the field of generative models. Since the release of ChatGPT at the end of 2022, generative chatbots have become widely employed across various medical disciplines. As a crucial discipline guiding clinical practices, the usage of generative chatbots like ChatGPT in Evidence-Based Medicine (EBM) is gradually increasing. However, the potential, challenges, and intricacies of their application in the domain of EBM remain unclear. This paper aims to explore and discuss the prospects, challenges, and considerations associated with the application of ChatGPT in the field of EBM through a review of relevant literature. The discussion spans four aspects: evidence generation, synthesis, assessment, dissemination, and implementation, providing researchers with insights into the latest developments and future research suggestions.
Objective This article takes the construction of the Huaxi Hongyi Medical University model as the core, explores its application effect in assisting the generation of medical records, and provides a practical path for the construction and application of artificial intelligence in medical institutions. Methods Through strategies such as multimodal data fusion, domain adaptation training, and localization of hardware adaptation, a large-scale medical model with 72 billion parameters was constructed. Combined with technologies such as speech recognition, knowledge graphs, and reinforcement learning, an application system for assisting in the generation of medical records was developed. Results Taking the assisted generation of discharge records as an example, in the pilot department, the average completion time of writing was reduced from 21 minutes to 5 minutes (a decrease of 76.2%), the accuracy rate of the model output reached 92.4%, and the annotation consistency Kappa coefficient was 0.85. Conclusion The model of medical institutions constructing independently controllable large-scale medical models and incubating various applications based on them can provide a reference path for the artificial intelligence construction of similar institutions.
Objective To systematically review the accuracy and consistency of large language models (LLM) in assessing risk of bias in analytical studies. Methods The cohort and case-control studies related to COVID-19 based on the team's published systematic review of clinical characteristics of COVID-19 were included. Two researchers independently screened the studies, extracted data, and assessed risk of bias of the included studies with the LLM-based BiasBee model (version Non-RCT) used for automated evaluation. Kappa statistics and score differences were used to analyze the agreement between LLM and human evaluations, with subgroup analysis for Chinese and English studies. Results A total of 210 studies were included. Meta-analysis showed that LLM scores were generally higher than those of human evaluators, particularly in representativeness of exposed cohorts (△=0.764) and selection of external controls (△=0.109). Kappa analysis indicated slight agreement in items such as exposure assessment (κ=0.059) and adequacy of follow-up (κ=0.093), while showing significant discrepancies in more subjective items, such as control selection (κ=−0.112) and non-response rate (κ=−0.115). Subgroup analysis revealed higher scoring consistency for LLM in English-language studies compared to that of Chinese-language studies. Conclusion LLM demonstrate potential in risk of bias assessment; however, notable differences remain in more subjective tasks. Future research should focus on optimizing prompt engineering and model fine-tuning to enhance LLM accuracy and consistency in complex tasks.