Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

OBJECTIVE To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN Inter-therapist reliability study. SETTING Research laboratory. PARTICIPANTS 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.

Modeling predictors of risky drug use behavior among male street laborers in urban Vietnam

Background The application of theoretical frameworks for modeling predictors of drug risk among male street laborers remains limited. The objective of this study was to test a modified version of the IMB (Information-Motivation-Behavioral Skills Model), which includes psychosocial stress, and compare this modified version with the original IMB model in terms of goodness-of-fit to predict risky drug use behavior among this population. Methods In a cross-sectional study, social mapping technique was conducted to recruit 450 male street laborers from 135 street venues across 13 districts of Hanoi city, Vietnam, for face-to-face interviews. Structural equation modeling (SEM) was used to analyze data from interviews. Results Overall measures of fit via SEM indicated that the original IMB model provided a better fit to the data than the modified version. Although the former model was able to predict a lesser variance than the latter (55% vs. 62%), it was of better fit. The findings suggest that men who are better informed and motivated for HIV prevention are more likely to report higher behavioral skills, which, in turn, are less likely to be engaged in risky drug use behavior. Conclusions This was the first application of the modified IMB model for drug use in men who were unskilled, unregistered laborers in urban settings. An AIDS prevention program for these men should not only distribute information and enhance motivations for HIV prevention, but consider interventions that could improve self-efficacy for preventing HIV infection. Future public health research and action may also consider broader factors such as structural social capital and social policy to alter the conditions that drive risky drug use among these men.