Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.Conclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

Protocol of the COSMIN study: consensus-based standards for the selection of health measurement instruments

Background: Choosing an adequate measurement instrument depends on the proposed use of the instrument, the concept to be measured, the measurement properties (e.g. internal consistency, reproducibility, content and construct validity, responsiveness, and interpretability), the requirements, the burden for subjects, and costs of the available instruments. As far as measurement properties are concerned, there are no sufficiently specific standards for the evaluation of measurement properties of instruments to measure health status, and also no explicit criteria for what constitutes good measurement properties. In this paper we describe the protocol for the COSMIN study, the objective of which is to develop a checklist that contains COnsensus-based Standards for the selection of health Measurement INstruments, including explicit criteria for satisfying these standards. We will focus on evaluative health related patient-reported outcomes (HR-PROs), i.e. patient-reported health measurement instruments used in a longitudinal design as an outcome measure, excluding health care related PROs, such as satisfaction with care or adherence. The COSMIN standards will be made available in the form of an easily applicable checklist.Method: An international Delphi study will be performed to reach consensus on which and how measurement properties should be assessed, and on criteria for good measurement properties. Two sources of input will be used for the Delphi study: (1) a systematic review of properties, standards and criteria of measurement properties found in systematic reviews of measurement instruments, and (2) an additional literature search of methodological articles presenting a comprehensive checklist of standards and criteria. The Delphi study will consist of four (written) Delphi rounds, with approximately 30 expert panel members with different backgrounds in clinical medicine, biostatistics, psychology, and epidemiology. The final checklist will subsequently be field-tested by assessing the inter-rater reproducibility of the checklist.Discussion: Since the study will mainly be anonymous, problems that are commonly encountered in face-to-face group meetings, such as the dominance of certain persons in the communication process, will be avoided. By performing a Delphi study and involving many experts, the likelihood that the checklist will have sufficient credibility to be accepted and implemented will increase.

The psychosocial difficulties in brain disorders that explain short term changes in health outcomes.

BACKGROUND: This study identifies a set of psychosocial difficulties that are associated with short term changes in health outcomes across a heterogeneous set of brain disorders, neurological and psychiatric. METHODS: Longitudinal observational study over approximately 12 weeks with three time points of assessment and 741 patients with depression, bipolar disorders, multiple sclerosis, parkinson's disease, migraine, traumatic brain injury and stroke. The data on disability was collected with the checklist of the International Classification of Functioning, Disability and Health. The selected health outcomes were the Short Form 36 and the World Health Organization Disability Assessment Schedule. Multilevel models for change were applied controlling for age, gender and disease severity. RESULTS: The psychosocial difficulties that explain the variability and change over time of the selected health outcomes were energy and drive, sleep, and emotional functions, and a broad range of activities and participation domains, such as solving problems, conversation, areas of mobility and self-care, relationships, community life and recreation and leisure. CONCLUSIONS: Our findings are of interest to researchers and clinicians for interventions and health systems planning as they show that in addition to difficulties that are diagnostic criteria of these disorders, there are other difficulties that explain small changes in health outcomes over short periods of time.

The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content

Background: The COSMIN checklist (COnsensus-based Standards for the selection of health status Measurement INstruments) was developed in an international Delphi study to evaluate the methodological quality of studies on measurement properties of health-related patient reported outcomes (HR-PROs). In this paper, we explain our choices for the design requirements and preferred statistical methods for which no evidence is available in the literature or on which the Delphi panel members had substantial discussion. Methods: The issues described in this paper are a reflection of the Delphi process in which 43 panel members participated. Results: The topics discussed are internal consistency (relevance for reflective and formative models, and distinction with unidimensionality), content validity (judging relevance and comprehensiveness), hypotheses testing as an aspect of construct validity (specificity of hypotheses), criterion validity (relevance for PROs), and responsiveness (concept and relation to validity, and (in) appropriate measures).Conclusions: We expect that this paper will contribute to a better understanding of the rationale behind the items, thereby enhancing the acceptance and use of the COSMIN checklist.

Predictors and consequences of adherence to the treatment of pediatric patients with attention-deficit/hyperactivity disorder in Central Europe and East Asia.

PURPOSE: To assess baseline predictors and consequences of medication non-adherence in the treatment of pediatric patients with attention-deficit/hyperactivity disorder (ADHD) from Central Europe and East Asia. PATIENTS AND METHODS: Data for this post-hoc analysis were taken from a 1-year prospective, observational study that included a total of 1,068 newly-diagnosed pediatric patients with ADHD symptoms from Central Europe and East Asia. Medication adherence during the week prior to each visit was assessed by treating physicians using a 5-point Likert scale, and then dichotomized into either adherent or non-adherent. Clinical severity was measured by the Clinical Global Impressions-ADHD-Severity (CGI-ADHD) scale and the Child Symptom Inventory-4 (CSI-4) Checklist. Health-Related Quality of Life (HRQoL) was measured using the Child Health and Illness Profile-Child Edition (CHIP-CE). Regression analyses were used to assess baseline predictors of overall adherence during follow-up, and the impact of time-varying adherence on subsequent outcomes: response (defined as a decrease of at least 1 point in CGI), changes in CGI-ADHD, CSI-4, and the five dimensions of CHIP-CE. RESULTS: Of the 860 patients analyzed, 64.5% (71.6% in Central Europe and 55.5% in East Asia) were rated as adherent and 35.5% as non-adherent during follow-up. Being from East Asia was found to be a strong predictor of non-adherence. In East Asia, a family history of ADHD and parental emotional distress were associated with non-adherence, while having no other children living at home was associated with non-adherence in Central Europe as well as in the overall sample. Non-adherence was associated with poorer response and less improvement on CGI-ADHD and CSI-4, but not on CHIP-CE. CONCLUSION: Non-adherence to medication is common in the treatment of ADHD, particularly in East Asia. Non-adherence was associated with poorer response and less improvement in clinical severity. A limitation of this study is that medication adherence was assessed by the treating clinician using a single item question.