Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

OBJECTIVE To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN Inter-therapist reliability study. SETTING Research laboratory. PARTICIPANTS 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.

A Satellite Selection Algorithm for Achieving High Reliability of Ambiguity Resolution with GPS and Beidou Constellations

Reliability of carrier phase ambiguity resolution (AR) of an integer least-squares (ILS) problem depends on ambiguity success rate (ASR), which in practice can be well approximated by the success probability of integer bootstrapping solutions. With the current GPS constellation, sufficiently high ASR of geometry-based model can only be achievable at certain percentage of time. As a result, high reliability of AR cannot be assured by the single constellation. In the event of dual constellations system (DCS), for example, GPS and Beidou, which provide more satellites in view, users can expect significant performance benefits such as AR reliability and high precision positioning solutions. Simply using all the satellites in view for AR and positioning is a straightforward solution, but does not necessarily lead to high reliability as it is hoped. The paper presents an alternative approach that selects a subset of the visible satellites to achieve a higher reliability performance of the AR solutions in a multi-GNSS environment, instead of using all the satellites. Traditionally, satellite selection algorithms are mostly based on the position dilution of precision (PDOP) in order to meet accuracy requirements. In this contribution, some reliability criteria are introduced for GNSS satellite selection, and a novel satellite selection algorithm for reliable ambiguity resolution (SARA) is developed. The SARA algorithm allows receivers to select a subset of satellites for achieving high ASR such as above 0.99. Numerical results from a simulated dual constellation cases show that with the SARA procedure, the percentages of ASR values in excess of 0.99 and the percentages of ratio-test values passing the threshold 3 are both higher than those directly using all satellites in view, particularly in the case of dual-constellation, the percentages of ASRs (>0.99) and ratio-test values (>3) could be as high as 98.0 and 98.5 % respectively, compared to 18.1 and 25.0 % without satellite selection process. It is also worth noting that the implementation of SARA is simple and the computation time is low, which can be applied in most real-time data processing applications.

Reliability and validity of physical fitness field tests for adults aged 55 to 70 years

The aim of this study was to examine the reliability and validity of field tests for assessing physical function in mid-aged and young-old people (55-70 y). Tests were selected that required minimal space and equipment and could be implemented in multiple field settings such as a general practitioner's office. Nineteen participants completed 2 field and I laboratory testing sessions. Intra-class correlations showed good reliability for the tests of upper body strength (lift and reach, R=.66), lower body strength (sit to stand, R=.80) and functional capacity (Canadian Step Test, R=.92), but not for leg power (single timed chair rise, R=.28). There was also good reliability for the balance test during 3 stances: parallel (94.7% agreement), semi-tandem (73.7%), and tandem (52.6%). Comparison of field test results with objective laboratory measures found good validity for the sit to stand (cf 1RM leg press, Pearson r=.68, p <.05), and for the step test (cf PWC140, r = -.60, p <.001), but not for the lift and reach (cf 1RM bench press, r=.43, p >.05), balance (r=-.13, -.18, .23) and rate of force development tests (r=-.28). It was concluded that the lower body strength and cardiovascular function tests were appropriate for use in field settings with mid-aged and young-old adults.

Advances in population surveillance for physical activity and sedentary behavior : reliability and validity of time use surveys

Many countries conduct regular national time use surveys, some of which date back as far as the 1960s. Time use surveys potentially provide more detailed and accurate national estimates of the prevalence of sedentary and physical activity behavior than more traditional self-report surveillance systems. In this study, the authors determined the reliability and validity of time use surveys for assessing sedentary and physical activity behavior. In 2006 and 2007, participants (n = 134) were recruited from work sites in the Australian state of New South Wales. Participants completed a 2-day time use diary twice, 7 days apart, and wore an accelerometer. The 2 diaries were compared for test-retest reliability, and comparison with the accelerometer determined concurrent validity. Participants with similar activity patterns during the 2 diary periods showed reliability intraclass correlations of 0.74 and 0.73 for nonoccupational sedentary behavior and moderate/vigorous physical activity, respectively. Comparison of the diary with the accelerometer showed Spearman correlations of 0.57-0.59 and 0.45-0.69 for nonoccupational sedentary behavior and moderate/vigorous physical activity, respectively. Time use surveys appear to be more valid for population surveillance of nonoccupational sedentary behavior and health-enhancing physical activity than more traditional surveillance systems. National time use surveys could be used to retrospectively study nonoccupational sedentary and physical activity behavior over the past 5 decades.

Reliability of a wellness inventory for use among adolescent females aged 12-14 years

Background The wellness construct has application in a number of fields including education, healthcare and counseling, particularly with regard to female adolescents. The effective measurement of wellness in adolescents can assist researchers and practitioners in determining lifestyle behaviors in which they are lacking. Behavior change interventions can then be designed which directly aid in the promotion of these areas. Methods The 5-Factor Wellness Inventory (designed to measure the Indivisible Self model of wellness) is a popular instrument for measuring the broad aspects of wellness amongst adolescents. The instrument comprises 97 items contributing to 17 subscales, five dimension scores, four context scores, total wellness score, and a life satisfaction index. This investigation evaluated the test-retest (intra-rater) reliability of the 5 F-Wel instrument in repeated assessments (seven days apart) among adolescent females aged 12-14 years. Percentages of exact agreement for individual items, and the number of respondents who scored within +/-5, +/-7.5 and +/-10 points for total wellness and the five summary dimension scores were calculated. Results Overall, 46 (95.8%) participants responded with complete data and were included in the analysis. Item agreement ranged from 47.8% to 100% across the 97 items (median 69.9%, interquartile range 60.9%-73.9%). The percentage of respondents who scored within +/-5, +/-7.5 and +/-10 points for total wellness at the re-assessment was 87.0%, 97.8% and 97.8% respectively. The percentage of respondents who scored within +/-5, +/-7.5 and +/-10 for the domain scores at the reassessment ranged between 54.3-76.1%, 78.3-95.7% and 89.1-95.7% respectively across the five dimensions. Conclusions These findings suggest there was considerable variation in agreement between the two assessments on some individual items. However, the total wellness score and the five dimension summary scores remained comparatively stable between assessments.

A test rig for the condition-based maintenance application on the traction chain of very high speed trains

The preventive maintenance of traction equipment for Very High Speed Trains (VHST) nowadays is becoming very expensive owing to the high complexity and quality of these components that require high reliability. An efficient maintenance approach like the Condition-Based Maintenance (CBM) should be implemented to reduce the costs. For this purpose, an experimental full-scale test rig for the CBM of VHST traction equipment has been designed to investigate in detail failures in the main mechanical components of system, i.e. motor, bearings and gearbox. The paper describes the main characteristics of this unique test rig, able to reproduce accurately the train operating conditions, including the relative movements of the motor, the gearbox and the wheel axle. Gearbox, bearing seats and motor are equipped by accelerometers, thermocouples, torque meter and other sensors in different positions. The testing results give important information about the most suitable sensor position and type to be installed for each component and show the effectiveness of the techniques used for the signal analysis in order to identify faults of the gearbox and motor bearings.

The validity and reliability of the Parent Fever Management Scale: A study from Palestine

Background Parental fever phobia and overuse of antipyretics to control fever is increasing. Little is known about childhood fever management among Arab parents. No scales to measure parents’ fever management practices in Palestine are available. Aims The aims of this study were to translate and examine the psychometric properties of the Arabic version of the Parent Fever Management Scale (PFMS). Methods A standard “forward–backward” procedure was used to translate PFMS into Arabic language. It was then validated on a convenience sample of 402 parents between July and October 2012. Descriptive statistics were used, and instrument reliability was assessed for internal consistency using Cronbach's alpha coefficient. Validity was confirmed using convergent and known group validation. Results Applying the recommended scoring method, the median (interquartile range) score of the PFMS was 26 (23-30). Acceptable internal consistency was found (Cronbach’s alpha = 0.733) and the test–retest reliability value was 0.92 (P < 0.001). The chi-squared (χ2) test showed a significant relationship between PFMS groups and frequent daily administration of antipyretic groups (χ2 = 52.86; P < 0.001). The PFMS sensitivity and specificity were 77.67% and 57.75%, respectively. The positive and negative predictive values were 67.89% and 32.11%, respectively. Conclusions The findings of this validation study indicate that the Arabic version of the PFMS is a reliable and valid measure which can be used as a useful tool for health professionals to identify parents’ fever management practices and thus provide targeted education to reduce the unnecessary burden of care they place on themselves when concerned for a febrile child.

Palliative care phase : inter-rater reliability and acceptability in a national study

Background: The concept of palliative care consisting of five distinct, clinically meaningful, phases (stable, unstable, deteriorating, terminal and bereavement) was developed in Australia about 20 years ago and is used routinely for communicating clinical status, care planning, quality improvement and funding. Aim: To test the reliability and acceptability of revised definitions of Palliative Care Phase. Design: Multi-centre cross-sectional study involving pairs of clinicians independently rating patients according to revised definitions of Palliative Care Phase. Setting/participants: Clinicians from 10 Australian palliative care services, including 9 inpatient units and 1 mixed inpatient/community-based service. Results: A total of 102 nursing and medical clinicians participated, undertaking 595 paired assessments of 410 patients, of which 90.7% occurred within 2 h. Clinicians rated 54.8% of patients in the stable phase, 15.8% in the unstable phase, 20.8% in the deteriorating phase and 8.7% in the terminal phase. Overall agreement between clinicians’ rating of Palliative Care Phase was substantial (kappa = 0.67; 95% confidence interval = 0.61–0.70). A moderate level of inter-rater reliability was apparent across all participating sites. The results indicated that Palliative Care Phase was an acceptable measure, with no significant difficulties assigning patients to a Palliative Care Phase and a good fit between assessment of phase and the definition of that phase. The most difficult phase to distinguish from other phases was the deteriorating phase. Conclusion: Policy makers, funders and clinicians can be confident that Palliative Care Phase is a reliable and acceptable measure that can be used for care planning, quality improvement and funding purposes.

Measuring reliability and validity of the ActiGraph GT3X accelerometer for children with cerebral palsy : a feasibility study

PURPOSE The purposes of this study were to: 1) establish inter-instrument reliability between left and right hip accelerometer placement; 2) examine procedural reliability of a walking protocol used to measure physical activity (PA), and; 3) confirm concurrent validity of accelerometers in measuring PA intensity as compared to the gold standard of oxygen consumption measured by indirect calorimetry. METHODS Eight children (mean age: 11.9; SD: 3.2, 75% male) with CP (GMFCS levels I-III) wore ActiGraph GT3X accelerometers on each hip and the Cosmed K4b^{2} portable indirect calorimeter during two measurement sessions in which they performed the six minute walk test (6MWT) at three self-selected speeds (comfortable/slow, brisk, fast). Oxygen consumption (VO2) and accelerometer step and activity count data were recorded. RESULTS Inter-instrument reliability of ActiGraph GT3X accelerometers placed on left and right hips was excellent (ICC=0.96-0.99, CI_{95}: 0.81-0.99). Reproducibility of the protocol was good/excellent (ICC=0.75-0.95, CI_{95}: 0.75-0.98). Concurrent validity of accelerometer count data and VO2 was fair/good (rho=0.67, p< 0.001). The correlation between step count and VO2 was not significant (rho=0.29, p=0.2). CONCLUSION This preliminary research suggests that ActiGraph GT3X accelerometers are reliable and valid devices to monitor PA during walking in children with CP and may be appropriate in rehabilitation research and clinical practice. ActiGraph GTX3 step counts were not valid for this sample and further research is warranted.

Cross-cultural adaptation, reliability and validity of the Turkish version of the spine functional index

Background The Spine Functional Index (SFI) is a patient reported outcome measure with sound clinimetric properties and clinical viability for the determination of whole-spine impairment. To date, no validated Turkish version is available. The purpose of this study is to cross-culturally adapted the SFI for Turkish-speaking patients (SFI-Tk) and determine the psychometric properties of reliability, validity and factor structure in a Turkish population with spine musculoskeletal disorders. Methods The SFI English version was culturally adapted and translated into Turkish using a double forward and backward method according to established guidelines. Patients (n = 285, cervical = l29, lumbar = 151, cervical and lumbar region = 5, 73% female, age 45 ± 1) with spine musculoskeletal disorders completed the SFI-Tk at baseline and after a seven day period for test-retest reliability. For criterion validity the Turkish version of the Functional Rating Index (FRI) was used plus the Neck Disability Index (NDI) for cervical patients and the Oswestry Disability Index (ODI) for back patients. Additional psychometric properties were determined for internal consistency (Chronbach’s α), criterion validity and factor structure. Results There was a high degree of internal consistency (α = 0.85, item range 0.80-0.88) and test-retest reliability (r = 0.93, item range = 0.75-0.95). The factor analysis demonstrated a one-factor solution explaining 24.2% of total variance. Criterion validity with the ODI was high (r = 0.71, p < 0.001) while the FRI and NDI were fair (r = 0.52 and r = 0.58, respectively). The SFI-Tk showed no missing responses with the ‘half-mark’ option used in 11.75% of total responses by 77.9% of participants. Measurement error from SEM and MDC90 were respectively 2.96% and 7.12%. Conclusions The SFI-Tk demonstrated a one-factor solution and is a reliable and valid instrument. The SFI-Tk consists of simple and easily understood wording and may be used to assess spine region musculoskeletal disorders in Turkish speaking patients.

Comparison of kinematic variables obtained by inertial sensors among stroke survivors and healthy older adults in the Functional Reach Test: cross-sectional study

Background Balance dysfunction is one of the most common problems in people who suffer stroke. To parameterize functional tests standardized by inertial sensors have been promoted in applied medicine. The aim of this study was to compare the kinematic variables of the Functional Reach Test (FRT) obtained by two inertial sensors placed on the trunk and lumbar region between stroke survivors (SS) and healthy older adults (HOA) and to analyze the reliability of the kinematic measurements obtained. Methods Cross-sectional study. Five SS and five HOA over 65. A descriptive analysis of the average range as well as all kinematic variables recorded was developed. The intrasubject and intersubject reliability of the measured variables was directly calculated. Results In the same intervals, the angular displacement was greater in the HOA group; however, they were completed at similar times for both groups, and HOA conducted the test at a higher speed and greater acceleration in each of the intervals. The SS values were higher than HOA values in the maximum and minimum acceleration in the trunk and in the lumbar region. Conclusions The SS show less functional reach, a narrower, slower and less accelerated movement during the FRT execution, but with higher peaks of acceleration and speed when they are compared with HOA.

Reliability of point-of-care testing of INR in acute stroke

Background In the emergency department, portable point-of-care testing (POCT) coagulation devices may facilitate stroke patient care by providing rapid International Normalized Ratio (INR) measurement. The objective of this study was to evaluate the reliability, validity, and impact on clinical decision-making of a POCT device for INR testing in the setting of acute ischemic stroke (AIS). Methods A total of 150 patients (50 healthy volunteers, 51 anticoagulated patients, 49 AIS patients) were assessed in a tertiary care facility. The INR's were measured using the Roche Coaguchek S and the standard laboratory technique. Results The interclass correlation coefficient and 95% confidence interval between overall POCT device and standard laboratory value INRs was high (0.932 (0.69 - 0.78). In the AIS group alone, the correlation coefficient and 95% CI was also high 0.937 (0.59 - 0.74) and diagnostic accuracy of the POCT device was 94%. Conclusions When used by a trained health professional in the emergency department to assess INR in acute ischemic stroke patients, the CoaguChek S is reliable and provides rapid results. However, as concordance with laboratory INR values decreases with higher INR values, it is recommended that with CoaguChek S INRs in the > 1.5 range, a standard laboratory measurement be used to confirm the results.

Agreement between physiotherapists rating scapular posture in multiple planes in patients with neck pain: Reliability study

Objective Evaluation of scapular posture is an integral component of the clinical assessment of painful neck disorders. The aim of this study was to evaluate agreement between therapist judgements of scapula posture in multiple biomechanical planes in individuals with neck pain. Design Inter-therapist reliability study. Setting Research laboratory. Participants Fifteen participants with chronic neck pain. Main outcome measures Four physiotherapists recorded ratings of scapular orientation (relative to the thorax) in five different scapula postural planes (plane of scapula, sagittal plane, transverse plane, horizontal plane, and vertical plane) under four test conditions (at rest, and during three isometric shoulder conditions) in all participants. Inter-therapist reliability was expressed using both generalized and paired kappa coefficient. Results Following adjustment for expected agreement and the high prevalence of neutral ratings (81%), on average both the generalised kappa (0.37) as well as Cohen's Kappa for the two therapist pairs (0.45 and 0.42) demonstrated only slight to moderate inter-therapist reliability. Conclusions The findings suggest that ratings of scapular posture in individuals with neck pain by visual inspection has only slight to moderate reliability and should only be used in conjunction with other clinical tests when judging scapula function in these patients.

Palliative care problem severity score: Reliability and acceptability in a national study

Background The Palliative Care Problem Severity Score is a clinician-rated tool to assess problem severity in four palliative care domains (pain, other symptoms, psychological/spiritual, family/carer problems) using a 4-point categorical scale (absent, mild, moderate, severe). Aim To test the reliability and acceptability of the Palliative Care Problem Severity Score. Design: Multi-centre, cross-sectional study involving pairs of clinicians independently rating problem severity using the tool. Setting/participants Clinicians from 10 Australian palliative care services: 9 inpatient units and 1 mixed inpatient/community-based service. Results A total of 102 clinicians participated, with almost 600 paired assessments completed for each domain, involving 420 patients. A total of 91% of paired assessments were undertaken within 2 h. Strength of agreement for three of the four domains was moderate: pain (Kappa = 0.42, 95% confidence interval = 0.36 to 0.49); psychological/spiritual (Kappa = 0.48, 95% confidence interval = 0.42 to 0.54); family/carer (Kappa = 0.45, 95% confidence interval = 0.40 to 0.52). Strength of agreement for the remaining domain (other symptoms) was fair (Kappa = 0.38, 95% confidence interval = 0.32 to 0.45). Conclusion The Palliative Care Problem Severity Score is an acceptable measure, with moderate reliability across three domains. Variability in inter-rater reliability across sites and participant feedback indicate that ongoing education is required to ensure that clinicians understand the purpose of the tool and each of its domains. Raters familiar with the patient they were assessing found it easier to assign problem severity, but this did not improve inter-rater reliability.

Development, reliability and validity of the queensland evaluation of wheelchair skills (QEWS)

- Objectives To develop and test a valid and reliable assessment of wheelchair skills for individuals with spinal cord injuries (SCI); the Queensland Evaluation of Wheelchair Skills (QEWS). - Setting Hospital, Australia. - Methods Phase 1: Four Delphi panel rounds with clinical experts were used to develop the QEWS. Phase 2: Intra-rater and inter-rater reliability of the QEWS items were examined in 100 people with SCI. Phase 3a: Concurrent validity was investigated by examining the association between QEWS total scores and physiotherapists’ global ratings of wheelchair skill performance. Phase 3b: Construct validity was tested in 20 people with recent SCI by examining change in QEWS total scores between when they first mobilised in a wheelchair and scores obtained 10 weeks later. - Results Phase 1: The QEWS was developed. Phase 2: The intra-class correlation coefficients reflecting the intra-rater reliability and the inter-rater reliability for the QEWS total score were 1.00 and 0.98, with scores being within one point of each other 96 and 91% of the time, respectively. Phase 3a: The QEWS total scores were comparable with the global rating of wheelchair skill performance (r2=0.93). Phase 3b: The QEWS scores changed by a median (interquartile range (IQR)) of 4 (1 to 6) points over the 10-week period following first wheelchair mobilisation. - Conclusion The QEWS is a valid and reliable tool for measuring wheelchair skills in individuals with SCI. The QEWS is efficient and practical to administer and does not require specialised equipment.