Inter-rater reliability study of the modified Oxford Grading Scale and the Peritron manometer

Objective To evaluate the inter-rater reliability of the modified Oxford Grading Scale and the Peritron manometer. Design All participants were evaluated twice, first by one examiner and 30 days later by a second examiner. Measurements of vaginal squeeze pressure were compared with the results from the palpation test. Participants Nineteen women with a mean age of 23.7 years (range 21 to 28 years). Results Inter-rater reliability for vaginal palpation was fair (kappa = 0.33, 95% confidence interval 0.09 to 0.57). Using the Peritron manometer, the difference between examiners was less than 10 cmH(2)O in 11 of the 19 (58%) cases. The palpation test did not differentiate between weak, moderate, good and strong muscle contractions. This study found fair inter-rater reliability for the modified Oxford Grading Scale and moderate inter-rater reliability for the Peritron manometer. Conclusions The inter-rater reliability of vaginal squeeze pressure measurement using the Peritron manometer is acceptable and can be used in re-evaluations performed by different examiners in clinical practice. However, for research purposes, the ideal situation would be for a single examiner to assess and re-assess the subject. Vaginal palpation is important in the clinical assessment of correctness of a pelvic floor muscle contraction, but this study does not support the use of the modified Oxford Grading Scale as a reliable and valid method to measure and differentiate pelvic floor muscle strength. (C) 2010 Chartered Society of Physiotherapy. Published by Elsevier Ltd. All rights reserved.

Diagnostic interview for genetic studies (DIGS): inter-rater and test-retest reliability of the French version.

The National Institute of Mental Health developed the semi-structured Diagnostic Interview for Genetic Studies (DIGS) for the assessment of major mood and psychotic disorders and their spectrum conditions. The DIGS was translated into French in a collaborative effort of investigators from sites in France and Switzerland. Inter-rater and test-retest reliability of the French version have been established in a clinical sample in Lausanne. Excellent inter-rater reliability was found for schizophrenia, bipolar disorder, major depression, and unipolar schizoaffective disorder while fair inter-rater reliability was demonstrated for bipolar schizoaffective disorder. Using a six-week test-retest interval, reliability for all diagnoses was found to be fair to good with the exception of bipolar schizoaffective disorder. The lower test-retest reliability was the result of a relatively long test-retest interval that favored incomplete symptom recall. In order to increase reliability for lifetime diagnoses in persons not currently affected, best-estimate procedures using additional sources of diagnostic information such as medical records and reports from relatives should supplement DIGS information in family-genetic studies. Within such a procedure, the DIGS appears to be a useful part of data collection for genetic studies on major mood disorders and schizophrenia in French-speaking populations.

The inter-rater reliability of the Pulmonary Embolism Severity Index.

The Pulmonary Embolism Severity Index (PESI) is a validated clinical prognostic model for patients with acute pulmonary embolism (PE). Our goal was to assess the PESI's inter-rater reliability in patients diagnosed with PE. We prospectively identified consecutive patients diagnosed with PE in the emergency department of a Swiss teaching hospital. For all patients, resident and attending physician raters independently collected the 11 PESI variables. The raters then calculated the PESI total point score and classified patients into one of five PESI risk classes (I-V) and as low (risk classes I/II) versus higher-risk (risk classes III-V). We examined the inter-rater reliability for each of the 11 PESI variables, the PESI total point score, assignment to each of the five PESI risk classes, and classification of patients as low versus higher-risk using kappa (κ) and intra-class correlation coefficients (ICC). Among 48 consecutive patients with an objective diagnosis of PE, reliability coefficients between resident and attending physician raters were > 0.60 for 10 of the 11 variables comprising the PESI. The inter-rater reliability for the PESI total point score (ICC: 0.89, 95% CI: 0.81-0.94), PESI risk class assignment (κ: 0.81, 95% CI: 0.66-0.94), and the classification of patients as low versus higher-risk (κ: 0.92, 95% CI: 0.72-0.98) was near perfect. Our results demonstrate the high reproducibility of the PESI, supporting the use of the PESI for risk stratification of patients with PE.

Validation study of the French version of the observer-rater scales measuring coping and cognitive errors (CE-CAP) on a non-clinical population

This study examined the validity and reliability of the French version of two observer-rated measures developed to assess cognitive errors (cognitive errors rating system [CERS]) [6] and coping action patterns (coping action patterns rating system [CAPRS]) [22,24]. The CE measures 14 cognitive errors, broken down according to their valence positive or negative (see the definitions by A.T. Beck), and the CAP measures 12 coping categories, based on an comprehensive review literature, each broken down into three levels of action (affective, behavioural, cognitive). Thirty (N = 30) subjects recruited in a community sample participated in the study. They were interviewed according to a standardized clinical protocol: these interviews were transcribed and analysed with both observer-rated systems. Results showed that the inter-rater reliability of the two measures is good and that their internal validity is satisfactory, due to a non-significant canonical correlation between CAP and CE. With regard to discriminant validity, we found a non-significant canonical correlation between CAPRS and CISS, one of most widely used self-report questionnaire measuring coping. The same can be said for the correlation with a self-report questionnaire measuring symptoms (SCL-90-R). These results confirm the absence of confounds in the assessment of cognitive errors and of coping as assessed by these observer-rated scales and add an argument in favour of the French validation of the CE-CAP rating scales. (C) 2010 Elsevier Masson SAS. All rights reserved.

Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.Conclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

Pavement Evaluation Using the Road Rater Deflection Dish, MLR-89-2, 1989

In recent years the Iowa Department of Transportation has shifted emphasis from the construction of new roads to the maintenance and preservation of existing highways. A need has developed for evaluating pavements structurally to select the correct rehabilitation strategy and to properly design a pavement overlay if necessary. Road Rater non-destructive testing has fulfilled this need and has been used successfully to evaluate pavement and subgrade conditions and to design asphaltic concrete overlays and portland cement concrete overlays. The Iowa Road Rater Design Method has been simplified so that it may be easily understood and used by various individuals who are involved in pavement restoration and management. Road Rater evaluation techniques have worked well to date and have been verified by pavement coring, soils sampling and testing. Void detection testing has also been performed, and results indicate that the Road Rater can be used to locate pavement voids and that Road Rater evaluation techniques are reasonably accurate. The success of Road Rater research and development has made dynamic deflection test data an important pavement management input.

Structural Analysis and Overlay Design of Pavements Using the Road Rater, MLR-84-3, 1986

In recent years the Iowa DOT has shifted emphasis from the construction of new roads to the maintenance and preservation of existing highways. A need has developed for analyzing pavements structurally to select the correct rehabilitation strategy and to properly design a pavement overlay if necessary. This need has been fulfilled by Road Rater testing which has been used successfully on all types of pavements to evaluate pavement and subgrade conditions and to design asphaltic concrete overlays. The Iowa Road Rater Design Method has been simplified so that it may be easily understood and used by the widely diverse groups of individuals which may be involved in pavement restoration and management. Road Rater analysis techniques have worked well to date and have been verified by pavement coring, soils sampling and testing, and pavement removal by block sampling. Void detection testing has also been performed experimentally in Iowa, and results indicate that the Road Rater can be used to locate pavement voids and that Road Rater analysis techniques are reasonably accurate. The success of Road Rater research and development has made deflection test data one of the most important pavement management inputs.

Assessing inter-rater reliability when the raters are fixed: two concepts and two estimates.

Intraclass correlation (ICC) is an established tool to assess inter-rater reliability. In a seminal paper published in 1979, Shrout and Fleiss considered three statistical models for inter-rater reliability data with a balanced design. In their first two models, an infinite population of raters was considered, whereas in their third model, the raters in the sample were considered to be the whole population of raters. In the present paper, we show that the two distinct estimates of ICC developed for the first two models can both be applied to the third model and we discuss their different interpretations in this context.

Correlation of the Road Rater and the Dynatest Falling Weight Deflectometer, MLR-91-04, 1991

The Falling Weight Deflectometer (FWD) has become the "standard" for deflection testing of pavements. Iowa has used a Road Rater since 1976 to obtain deflection information. A correlation between the Road Rater and the FWD was needed if Iowa was going to continue with the Road Rater. Comparative deflection testing was done using a Road Rater Model 400 and a Pynatest 8000 FWD on 26 pavement sections. The SHRP contractor, Braun Intertec Pavement, Inc., provided the FWD testing. The r^2 for the linear correlations ranged from 0.90 to 0.99 for the different pavement types and sensor locations.

Diagnostic interview for genetic studies (DIGS): inter-rater and test-retest reliability of alcohol and drug diagnoses

The semi-structured diagnostic interview for genetic studies (DIGS) was developed to assess major mood and psychotic disorders and their spectrum manifestations in genetic studies. Our research group developed a French version of the DIGS and tested its inter-rater and test-retest reliability in psychiatric patients. In this article, we present estimates of the reliability of substance use and antisocial personality disorders. High kappa coefficients for inter-rater reliability were found for drug and alcohol as well as antisocial personality diagnoses and slightly lower kappas for test-retest reliability. Combined with evidence of the reliability of major mood and psychotic disorders, these findings support the suitability of the DIGS for studies of familial aggregation and comorbidity of psychiatric disorders including substance use and antisocial personality disorders.

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse.

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W = 0.84) and for overall evaluation (Kendall's W = 0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.