ObjectiveThis study proposes employing large language models (LLMs) for medical literature quality assessment, exploring their potential to establish a standardized and scalable intelligent evaluation framework for off-label drug use (OLDU). MethodsThe study used two freely available LLMs platforms in China, DeepSeek-R1 and Doubao. Following the medical literature quality assessment tools recommended in the evidence-based evaluation specification for off-label drug use issued by the Guangdong Pharmaceutical Association, we selected the Jadad scale and the MINORS criteria. These tools were employed to assess the quality of the two most prevalent types of medical literature in OLDU evidence evaluation: randomized controlled trials (RCTs) and non-randomized controlled trials (non-RCTs). Utilizing chain-of-thought (CoT) prompting techniques, we developed standardized evaluation templates. The quality scores generated by the LLMs were then compared against those reported in systematic reviews or assigned by clinical pharmacists. ResultsFor RCT, DeepSeek-R1 demonstrated consistency with human assessments in quality appraisal. However, discrepancies exist between the Doubao model and manual evaluation results, with three repeated evaluations yielding inconsistent outcomes and inaccurate identification of "allocation concealment" items. For Non-RCT, all models achieved concordant quality assessment outcomes with human evaluators, while demonstrating unique capacity to detect systematic evaluation inaccuracies attributable to human subjective bias. ConclusionThis study demonstrates that prompt engineering-driven LLMs can efficiently conduct quality assessments of medical literature. However, the selection of models requires rigorous validation against domain-specific benchmarks, alongside mandatory expert validation of scoring outputs. Our findings further reveal the necessity of refining current quality appraisal criteria through granular operational definitions, thereby facilitating standardized automation. This approach not only enhances the efficiency and transparency of evidence-based decision-making for OLDU but also extends to systematic reviews and rapid health technology assessments. By replacing traditional literature quality evaluation models with automated scoring mechanisms, it enables a paradigm shift in the efficiency of evidence processing.
ObjectiveTo investigate the capability and demands of medical students of medical information retrieval, and to provide information for medical information literacy education.MethodsWe conducted an online questionnaire survey among medical students from 15th to 21st July, 2019, aiming to analyze basic characteristics of the subjects and their ways and capability to acquire medical information, and evaluated their demands and satisfaction about relevant courses.ResultsFour hundred and nine valid questionnaires were collected. The average age of the respondents was (21±3) years old. Among the respondents, 279 (68.2%) were female, 290 (70.9%) were undergraduates, and 187 (45.7%) majored in clinical medicine. The mostly accessed ways to obtain medical information were textbooks (87.8%) and search engines (84.4%). Among Chinese literature databases, the most commonly used was China National Knowledge Infrastructure (67.7%), followed by Wanfang (54.3%). As for English literature databases, the most commonly used was PubMed (66.0%), while the proportions of respondents who used other databases was less than 1/4. As for the problems when obtaining information, 84.8% of the respondents supposed that no access to some online databases was the main problem. Obtaining information from search engine scored the highest [(3.21±1.00) points, 5 points totally] when they were asked to self-evaluate their ability to obtain medical information by different methods. In terms of training demands, the need for courses about literature authenticity and accuracy evaluation was the highest [(4.05±1.07) points, 5 points totally]. The trainings were expected to be conducted within 1-2 years after enrollment (83.9%). Training through professional courses (86.8%) was the most welcomed training form, followed by courses on MOOC (51.3%), an online course platform. More than 50% of the respondents were satisfied with the current relevant courses and trainings.ConclusionBoth the students’ capability of medical information retrieval and the design of relevant trainings should be improved to better prepare the medical students for further clinical practice and scientific research.
Objective To investigate an evaluation method of medical literature applicability to clinical work， and provide a convenient way for physicians to search for the best evidence. Methods Delphi method was used to choose appropriate evaluating indexes， analytic hierarchy process was performed to determine the weighing of each index， and the formula to calculate medical literature applicability was formed. The practicability of this formula was evaluated by consistency checking between the formula’s results and experts’ opinions on literature applicability. Results Five evaluating indexes were determined， including literature’s publishing year （X1）， whether the target questions were covered （X2）, sample size （X3）， trial category （X4）， and journal level （X5）. The formula to calculate medical literature applicability was Y=3.93 X1＋11.78 X2＋14.83 X3＋44.53 X4＋24.93 X5. The result of consistency checking showed that the formula’s results were highly consistent with experts’ opinions （Kappa=0.75， P＜0.001）. Conclusion The applicability formula is a valuable tool to evaluate medical literature applicability.