Measurement properties studies of patient-reported outcome measures (PROMs) aims to validate the measurement properties of PROMs. In the process of designing and statistical analysis of these measurement properties studies, bias will occur if there are any defects, which will affect the quality of PROMs. Therefore, the COSMIN (consensus-based standards for the selection of health measurement instruments) team has developed the COSMIN risk of bias (COSMIN-RoB) checklist to evaluate risk of bias of studies on measurement properties of PROMs. The checklist can be used to develop systematic reviews of PROMs measurement properties, and for PROMs developers, it can also be used to guide the research design in the measurement tool development process for reducing bias. At present, similar assessment tools are lacking in China. Therefore, this article aims to introduce the primary contents of COSMIN-RoB checklist and to interpret how to evaluate risk of bias of the internal structure studies of PROMs with examples.
Objective To systematically review the accuracy and consistency of large language models (LLM) in assessing risk of bias in analytical studies. Methods The cohort and case-control studies related to COVID-19 based on the team's published systematic review of clinical characteristics of COVID-19 were included. Two researchers independently screened the studies, extracted data, and assessed risk of bias of the included studies with the LLM-based BiasBee model (version Non-RCT) used for automated evaluation. Kappa statistics and score differences were used to analyze the agreement between LLM and human evaluations, with subgroup analysis for Chinese and English studies. Results A total of 210 studies were included. Meta-analysis showed that LLM scores were generally higher than those of human evaluators, particularly in representativeness of exposed cohorts (△=0.764) and selection of external controls (△=0.109). Kappa analysis indicated slight agreement in items such as exposure assessment (κ=0.059) and adequacy of follow-up (κ=0.093), while showing significant discrepancies in more subjective items, such as control selection (κ=−0.112) and non-response rate (κ=−0.115). Subgroup analysis revealed higher scoring consistency for LLM in English-language studies compared to that of Chinese-language studies. Conclusion LLM demonstrate potential in risk of bias assessment; however, notable differences remain in more subjective tasks. Future research should focus on optimizing prompt engineering and model fine-tuning to enhance LLM accuracy and consistency in complex tasks.
ObjectivesTo comprehensively evaluate the methodological quality and applicability of the results of systematic reviews on acupuncture treatment for primary depression.MethodsWeb of Science, EMbase, PubMed, The Cochrane Library, CNKI, CBM, WanFang Data and VIP databases were electronically searched to collect systematic reviews/meta-analyses on acupuncture treatment for primary depression from inception to December 5th, 2018. Two researchers independently screened and extracted data by using tools of AMSTAR 2 to evaluate the methodological quality, using ROBIS to assess risk of bias, and using CASP-S.R to evaluate the applicability of the results.ResultsA total of 18 systematic reviews/meta-analyses were included, and all focused on acupuncture intervention, including 2 primary outcome indicators. According to AMSTAR 2 evaluation results, there were 4 high quality studies, 12 medium quality studies and 2 low quality studies; ROBIS results found 10 high bias risk studies, 7 low bias risk studies and 1 unclear; CASP-S.R showed only 4 design studies applicable to local individuals, and there were no studies on the relationship between design benefits, hazards and costs.ConclusionsThe quality of systematic reviews/meta-analyses for acupuncture treatment of primary depression is moderate, however with a certain bias. Most studies may not directly benefit local individuals. All studies have no relationship with cost hazards. It is expected for further reviewers to strictly follow systematic evaluation method to improve research quality and reduce bias, while the applicability of the systematic review to individuals from different regions should be considered as well as the relationship between the benefit and cost hazard. In addition, more valid RCTs are required to provide higher quality evidence and explore correlated and comprehensive mechanism.
This study aims to introduce how to use the PROBAST (prediction model risk of bias assessment tool) to evaluate risk of bias and applicability of the study of diagnostic or prognostic predictive models, including the introduction of the background, the scope of application and use of the tool. This tool mainly involves the four areas of participants, predictors, outcomes and analyses. The risk of bias in the research is evaluated through the four areas, while the applicability is evaluated in the first three. PROBAST provides a standardized approach to evaluate the critical appraisal of the study of diagnostic or prognostic predictive models, which screens qualified literature for data analysis and helps to establish a scientific basis for clinical decision-making.
ObjectiveTo systematically interpret the updated risk of bias in non-randomized studies of interventions version 2 (ROBINS-I V2) in 2024, summarizing its key improvements, operational procedures, and clinical application value. MethodsThrough literature review and case studies, the improvements of ROBINS-I V2 were compared with the 2016 version, including the expansion of bias domains, refinement of signaling questions, and optimization of decision flowcharts. A retrospective study in stomatology was used to demonstrate the practical application of the tool. ResultsThe ROBINS-I V2 tool has restructured the hierarchy and refined the definitions of bias domains, optimized the evaluation processes across seven risk-of-bias dimensions, and minimized subjective judgment errors through standardized decision flowcharts. ConclusionROBINS-I V2 significantly improves the rigor of bias assessment in non-randomized intervention studies through its scientific design and standardized workflow. It is recommended for evidence quality grading and decision-making support in clinical research.
Evidence synthesis is the process of systematically gathering, analyzing, and integrating available research evidence. The quality of evidence synthesis depends on the quality of the original studies included. Validity assessment, also known as risk of bias assessment, is an essential method for assessing the quality of these original studies. Currently, there are numerous validity assessment tools available, but some of them lack a rigorous development process and evaluation. The application of inappropriate validity assessment tools to assessing the quality of the original studies during the evidence synthesis process may compromise the accuracy of study conclusions and mislead the clinical practice. To address this dilemma, the LATITUDES Network, a one-stop resource website for validity assessment tools, was established in September 2023, led by academics at the University of Bristol, U.K. This Network is dedicated to collecting, sorting and promoting validity assessment tools to improve the accuracy of original study validity assessments and increase the robustness and reliability of the results of evidence synthesis. This study introduces the background of the establishment of the LATITUDES Network, the included validity assessment tools, and the training resources for the use of validity assessment tools, in order to provide a reference for domestic scholars to learn more about the LATITUDES Network, to better use the appropriate validity assessment tools to conduct study quality assessments, and to provide references for the development of validity assessment tools.
With the rapid development of artificial intelligence (AI) and machine learning technologies, the development of AI-based prediction models has become increasingly prevalent in the medical field. However, the PROBAST tool, which is used to evaluate prediction models, has shown growing limitations when assessing models built on AI technologies. Therefore, Moons and colleagues updated and expanded PROBAST to develop the PROBAST+AI tool. This tool is suitable for evaluating prediction model studies based on both artificial intelligence methods and regression methods. It covers four domains: participants and data sources, predictors, outcomes, and analysis, allowing for systematic assessment of quality in model development, risk of bias in model evaluation, and applicability. This article interprets the content and evaluation process of the PROBAST+AI tool, aiming to provide references and guidance for domestic researchers using this tool.
The COSMIN community updated the COSMIN-RoB checklist on reliability and measurement error in 2021. The updated checklist can be applied to the assessment of all types of outcome measurement studies, including clinician-reported outcome measures (ClinPOMs), performance-basd outcome measurement instruments (PerFOMs), and laboratory values. In order to help readers better understand and apply the updated COSMIN-RoB checklist and provide methodological references for conducting systematic reviews of ClinPOMs, PerFOMs and laboratory values, this paper aimed to interpret the updated COSMIN-RoB checklist on reliability and measurement error studies.
High-quality randomized controlled trials are the best source of evidence to explain the relationship between health interventions and outcomes. However, in cases where they are insufficient, indirect, or inappropriate, researchers may need to include non-randomized studies of interventions to strengthen the evidence body and improve the certainty (quality) of evidence. The latest research from the GRADE working group provides a way for researchers to integrate randomized and non-randomized evidence. The present paper introduced the relevant methods to provide guidance for systematic reviewers, health technology assessors, and guideline developers.
The current issue of air pollution has pushed the development of the corresponding observational air pollution studies. The World Health Organization has developed a new risk of bias (RoB) assessment instrument and a related guideline for assessing the risk of potential bias in observational air pollution studies. This study introduced the background, methods, uses, advantages and disadvantages, precautions, and usage scenarios of the RoB instrument. It is expected to provide researchers with corresponding quality evaluation tools when writing related systematic review and meta-analysis, which will also help provide reporting standards for observational air pollution studies, thereby improving the quality of studies.