Intra and inter-examiner reliability of the subacromial impingement index

The present study aimed to assess the reliability of intra and inter-examiner subacromial impingement index (SII) measures obtained from radiographs. Thirty-six individuals were enrolled and divided into two groups: control group, composed of 18 volunteers in good general health without shoulder problems, and a group of 18 patients with subacromial impingement syndrome (SIS). Radiographic images were taken with the dominant upper limb in neutral rotation, while the volunteers held their arm at 90A degrees of abduction in the frontal plane. The beam of radiation at 30A degrees craniocaudal inclination was used to provide an antero-posterior image view. Three blinded examiners each performed three measurements from the subacromial space (SS) and the anatomical neck of the humerus (NH). The SII was calculated as the ratio of the SS and the NH measures. The mean values of SII were compared using t-tests. The intra-class correlation coefficient (ICC) was used to assess intra- and inter-examiner reliability of the measures. The mean values of SII were greater for the control group (0.12) than for the SIS group (0.08; p = 0.0071). SII measurements showed excellent intra (0.96-0.99) and inter-examiner reliability (0.94) for both the control and SIS group. The results of this study show the potential use of the SII; a greater mean value for the control group compared to the SIS group and excellent reliability for intra- and inter-examiner measurement. Validation studies of the index should be conducted to correlate the index with clinical findings from subacromial impingement syndrome.

Inter-rater reliability study of the modified Oxford Grading Scale and the Peritron manometer

Objective To evaluate the inter-rater reliability of the modified Oxford Grading Scale and the Peritron manometer. Design All participants were evaluated twice, first by one examiner and 30 days later by a second examiner. Measurements of vaginal squeeze pressure were compared with the results from the palpation test. Participants Nineteen women with a mean age of 23.7 years (range 21 to 28 years). Results Inter-rater reliability for vaginal palpation was fair (kappa = 0.33, 95% confidence interval 0.09 to 0.57). Using the Peritron manometer, the difference between examiners was less than 10 cmH(2)O in 11 of the 19 (58%) cases. The palpation test did not differentiate between weak, moderate, good and strong muscle contractions. This study found fair inter-rater reliability for the modified Oxford Grading Scale and moderate inter-rater reliability for the Peritron manometer. Conclusions The inter-rater reliability of vaginal squeeze pressure measurement using the Peritron manometer is acceptable and can be used in re-evaluations performed by different examiners in clinical practice. However, for research purposes, the ideal situation would be for a single examiner to assess and re-assess the subject. Vaginal palpation is important in the clinical assessment of correctness of a pelvic floor muscle contraction, but this study does not support the use of the modified Oxford Grading Scale as a reliable and valid method to measure and differentiate pelvic floor muscle strength. (C) 2010 Chartered Society of Physiotherapy. Published by Elsevier Ltd. All rights reserved.

Inter-rater reliability of the evaluation of muscular chains associated with posture alterations in scoliosis

Background: In the Global postural re-education (GPR) evaluation, posture alterations are associated with anterior or posterior muscular chain impairments. Our goal was to assess the reliability of the GPR muscular chain evaluation. Methods: Design: Inter-rater reliability study. Fifty physical therapists (PTs) and two experts trained in GPR assessed the standing posture from photographs of five youths with idiopathic scoliosis using a posture analysis grid with 23 posture indices (PI). The PTs and experts indicated the muscular chain associated with posture alterations. The PTs were also divided into three groups according to their experience in GPR. Experts' results (after consensus) were used to verify agreement between PTs and experts for muscular chain and posture assessments. We used Kappa coefficients (K) and the percentage of agreement (%A) to assess inter-rater reliability and intra-class coefficients (ICC) for determining agreement between PTs and experts. Results: For the muscular chain evaluation, reliability was moderate to substantial for 12 PI for the PTs (% A: 56 to 82; K: 0.42 to 0.76) and perfect for 19 PI for the experts. For posture assessment, reliability was moderate to substantial for 12 PI for the PTs (% A > 60%; K: 0.42 to 0.75) and moderate to perfect for 18 PI for the experts (% A: 80 to 100; K: 0.55 to 1.00). The agreement between PTs and experts was good for most muscular chain evaluations (18 PI; ICC: 0.82 to 0.99) and PI (19 PI; ICC: 0.78 to 1.00). Conclusions: The GPR muscular chain evaluation has good reliability for most posture indices. GPR evaluation should help guide physical therapists in targeting affected muscles for treatment of abnormal posture patterns.

Intra-Rater and Inter-Rater Reliability of a Medical Record Abstraction Study on Transition of Care after Childhood Cancer.

BACKGROUND The abstraction of data from medical records is a widespread practice in epidemiological research. However, studies using this means of data collection rarely report reliability. Within the Transition after Childhood Cancer Study (TaCC) which is based on a medical record abstraction, we conducted a second independent abstraction of data with the aim to assess a) intra-rater reliability of one rater at two time points; b) the possible learning effects between these two time points compared to a gold-standard; and c) inter-rater reliability. METHOD Within the TaCC study we conducted a systematic medical record abstraction in the 9 Swiss clinics with pediatric oncology wards. In a second phase we selected a subsample of medical records in 3 clinics to conduct a second independent abstraction. We then assessed intra-rater reliability at two time points, the learning effect over time (comparing each rater at two time-points with a gold-standard) and the inter-rater reliability of a selected number of variables. We calculated percentage agreement and Cohen's kappa. FINDINGS For the assessment of the intra-rater reliability we included 154 records (80 for rater 1; 74 for rater 2). For the inter-rater reliability we could include 70 records. Intra-rater reliability was substantial to excellent (Cohen's kappa 0-6-0.8) with an observed percentage agreement of 75%-95%. In all variables learning effects were observed. Inter-rater reliability was substantial to excellent (Cohen's kappa 0.70-0.83) with high agreement ranging from 86% to 100%. CONCLUSIONS Our study showed that data abstracted from medical records are reliable. Investigating intra-rater and inter-rater reliability can give confidence to draw conclusions from the abstracted data and increase data quality by minimizing systematic errors.

Intra- and inter-observer reliability of MRI examination of intervertebral disc abnormalities in patients with cervical myelopathy

Purpose: Intervertebral cervical disc herniation (CDH) is a relatively common disorder that can coexist with degenerative changes to worsen cervicogenic myelopathy. Despite the frequent disc abnormalities found in asymptomatic populations, magnetic resonance imaging (MRI) is considered excellent at detecting cervical spine myelopathy (CSM) associated with disc abnormality. The objective of this study was to investigate the intra- and inter-observer reliability of MRI detection of CSM in subjects who also had co-existing intervertebral disc abnormalities. Materials and methods: Seven experienced radiologists reviewed twice the MRI of 10 patients with clinically and/or imaging determined myelopathy. MRI assessment was performed individually, with and without operational guidelines. A Fleiss Kappa statistic was used to evaluate the intra- and inter-observer agreement. Results: The study found high intra-observer percent agreement but relatively low Kappa values on selected variables. Inter-observer reliability was also low and neither observation was improved with operational guidelines. We believe that those low values may be associated with the base rate problem of Kappa. Conclusion: In conclusion, this study demonstrated high intra-observer percent agreement in MR examination for intervertebral disc abnormalities in patients with underlying cervical myelopathy, but differing levels of intra- and inter-observer Kappa agreement among seven radiologists. (c) 2007 Elsevier Ireland Ltd. All rights reserved.

The inter-rater reliability of the Pulmonary Embolism Severity Index.

The Pulmonary Embolism Severity Index (PESI) is a validated clinical prognostic model for patients with acute pulmonary embolism (PE). Our goal was to assess the PESI's inter-rater reliability in patients diagnosed with PE. We prospectively identified consecutive patients diagnosed with PE in the emergency department of a Swiss teaching hospital. For all patients, resident and attending physician raters independently collected the 11 PESI variables. The raters then calculated the PESI total point score and classified patients into one of five PESI risk classes (I-V) and as low (risk classes I/II) versus higher-risk (risk classes III-V). We examined the inter-rater reliability for each of the 11 PESI variables, the PESI total point score, assignment to each of the five PESI risk classes, and classification of patients as low versus higher-risk using kappa (κ) and intra-class correlation coefficients (ICC). Among 48 consecutive patients with an objective diagnosis of PE, reliability coefficients between resident and attending physician raters were > 0.60 for 10 of the 11 variables comprising the PESI. The inter-rater reliability for the PESI total point score (ICC: 0.89, 95% CI: 0.81-0.94), PESI risk class assignment (κ: 0.81, 95% CI: 0.66-0.94), and the classification of patients as low versus higher-risk (κ: 0.92, 95% CI: 0.72-0.98) was near perfect. Our results demonstrate the high reproducibility of the PESI, supporting the use of the PESI for risk stratification of patients with PE.

Assessing inter-rater reliability when the raters are fixed: two concepts and two estimates.

Intraclass correlation (ICC) is an established tool to assess inter-rater reliability. In a seminal paper published in 1979, Shrout and Fleiss considered three statistical models for inter-rater reliability data with a balanced design. In their first two models, an infinite population of raters was considered, whereas in their third model, the raters in the sample were considered to be the whole population of raters. In the present paper, we show that the two distinct estimates of ICC developed for the first two models can both be applied to the third model and we discuss their different interpretations in this context.

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse.

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W = 0.84) and for overall evaluation (Kendall's W = 0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.

The inter-rater reliability of the Pulmonary Embolism Severity Index

The Pulmonary Embolism Severity Index (PESI) is a validated clinical prognostic model for patients with acute pulmonary embolism (PE). Our goal was to assess the PESI's inter-rater reliability in patients diagnosed with PE. We prospectively identified consecutive patients diagnosed with PE in the emergency department of a Swiss teaching hospital. For all patients, resident and attending physician raters independently collected the 11 PESI variables. The raters then calculated the PESI total point score and classified patients into one of five PESI risk classes (I-V) and as low (risk classes I/II) versus higher-risk (risk classes III-V). We examined the inter-rater reliability for each of the 11 PESI variables, the PESI total point score, assignment to each of the five PESI risk classes, and classification of patients as low versus higher-risk using kappa ( ) and intra-class correlation coefficients (ICC). Among 48 consecutive patients with an objective diagnosis of PE, reliability coefficients between resident and attending physician raters were > 0.60 for 10 of the 11 variables comprising the PESI. The inter-rater reliability for the PESI total point score (ICC: 0.89, 95% CI: 0.81-0.94), PESI risk class assignment ( : 0.81, 95% CI: 0.66-0.94), and the classification of patients as low versus higher-risk ( : 0.92, 95% CI: 0.72-0.98) was near perfect. Our results demonstrate the high reproducibility of the PESI, supporting the use of the PESI for risk stratification of patients with PE.

Assessing the intra- and inter-reader reliability of dynamic ultrasound images in power Doppler ultrasonography

OBJECTIVE: To assess the intra-reader and inter-reader reliabilities of interpreting ultrasonography by several experts using video clips. METHOD: 99 video clips of healthy and rheumatic joints were recorded and delivered to 17 physician sonographers in two rounds. The intra-reader and inter-reader reliabilities of interpreting the ultrasound results were calculated using a dichotomous system (normal/abnormal) and a graded semiquantitative scoring system. RESULTS: The video reading method worked well. 70% of the readers could classify at least 70% of the cases correctly as normal or abnormal. The distribution of readers answering correctly was wide. The most difficult joints to assess were the elbow, wrist, metacarpophalangeal (MCP) and knee joints. The intra-reader and inter-reader agreements on interpreting dynamic ultrasound images as normal or abnormal, as well as detecting and scoring a Doppler signal were moderate to good (kappa = 0.52-0.82). CONCLUSIONS: Dynamic image assessment (video clips) can be used as an alternative method in ultrasonography reliability studies. The intra-reader and inter-reader reliabilities of ultrasonography in dynamic image reading are acceptable, but more definitions and training are needed to improve sonographic reproducibility.

Inter-operator reliability of live football match statistics from OPTA Sportsdata

The aim of the study was to evaluate the inter-operator reliability of OPTA Client System which is used to collect live football match statistics by OPTA Sportsdata Company. Two groups of experienced operators were required to analyze a Spanish league match independently. Results showed that team events coded by independent operators reached a very good agreement (kappa values were 0.92 and 0.94) and average difference of event time was 0.06±0.04 s. The reliability of goalkeeper actions was also at high level, kappa values were 0.92 and 0.86. The high intra-class correlation coefficients (ranged from 0.88 to 1.00) and low standardized typical errors (varied from 0.00 to 0.37) of different match actions and indicators of individual outfield players showed a high level of inter-operator reliability as well. These results suggest that the OPTA Client System is reliable to be used to collect live football match statistics by well trained operators.

Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database

Rationale and aims 'OTseeker' is an online database of randomized controlled trials (RCTs) and systematic reviews relevant to occupational therapy. RCTs are critically appraised and rated for quality using the 'PEDro' scale. We aimed to investigate the inter-rater reliability of the PEDro scale before and after revising rating guidelines. Methods In study 1, five raters scored 100 RCTs using the original PEDro scale guidelines. In study 2, two raters scored 40 different RCTs using revised guidelines. All RCTs were randomly selected from the OTseeker database. Reliability was calculated using Kappa and intraclass correlation coefficients [ICC (model 2,1)]. Results Inter-rater reliability was 'good to excellent' in the first study (Kappas >= 0.53; ICCs >= 0.71). After revising the rating guidelines, the reliability levels were equivalent or higher to those previously obtained (Kappas >= 0.53; ICCs >= 0.89), except for the item, 'groups similar at baseline', which still had moderate reliability (Kappa = 0.53). In study 2, two PEDro scale items, which had their definitions revised, 'less than 15% dropout' and 'point measures and variability', showed higher reliability. In both studies, the PEDro items with the lowest reliability were 'groups similar at baseline' (Kappas = 0.53), 'less than 15% dropout' (Kappas

The inter-rater reliability of the Risk Instrument for Screening in the Community

Predicting risk of adverse healthcare outcomes is important to enable targeted delivery of interventions. The Risk Instrument for Screening in the Community (RISC), designed for use by public health nurses (PHNs), measures the one-year risk of hospitalisation, institutionalisation and death in community-dwelling older adults according to a five-point global risk score: from low (score 1,2), medium (3) and high (4,5). We examined the inter-rater reliability (IRR) of the RISC between student PHNs (n=32) and expert raters using six cases (two low, medium and high-risk), scored before and after RISC training. Correlations increased for each adverse outcome, statistically significantly for institutionalisation (r=0.72 to 0.80,p=0.04) and hospitalisation, (r=0.51 to 0.71,p<0.01) but not death. Training improved accuracy for low-risk but not all high-risk cases. Overall, the RISC showed good IRR, which increased after RISC training. That reliability reduced for some high-risk cases suggests that the training programme requires adjustment to further improve IRR.

Avaliação sensoriomotora do complexo articular do ombro : validação de um protocolo

Introdução: São escassas as medidas objetivas, válidas e consistentes da função sensoriomotora do complexo articular do ombro (CAO). Objetivos: desenvolvimento e validação de um protocolo de avaliação sensoriomotora do CAO. Materiais e métodos: 40 Sujeitos (16 atletas e 24 não atletas) completaram 6 momentos de teste, 3 por cada um de 2 investigadores. Em cada momento avaliou-se a NPA pelo Biodex System 3, na elevação do braço no plano da omoplata a 80º e 160º, e na rotação externa (RE) a 45º e 80º , bem como o controlo postural do CAO avaliou-se pela plataforma Rsscan. Resultados/Discussão: Todas as variáveis apresentaram estabilidade temporal, exceto os erros a 80º na RE (p=0,021) e 80º no PO (p=0,006), nos atletas, para o Investigador 1, e as componentes do CP nos não atletas, para o investigador 2. Os ICC’s foram globalmente baixos, apresentando valores moderados no grupo de não atletas, para o Investigador 2, no comprimento (0,862) e velocidade média do CP (0,852). Excluindo a área do CP no grupo de não atletas (p=0,030), não houve diferenças significativas nas médias entre observadores. Verificaram-se correlações fortes entre as componentes do CP, mas não ocorreram correlações fortes entre estas e as da NPA. Conclusões: Os resultados poucos consistentes, não permitem retirar conclusões sólidas, impossibilitando assumir o protocolo como fiável e válido na avaliação sensoriomotora do CAO.