Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

OBJECTIVE To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN Inter-therapist reliability study. SETTING Research laboratory. PARTICIPANTS 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.

Arabic translation, adaptation and modification of the dialysis symptom index for chronic kidney disease stages four and five

Background Symptom burden in chronic kidney disease (CKD) is poorly understood. To date, the majority of research focuses on single symptoms and there is a lack of suitable multidimensional symptom measures. The purpose of this study was to modify, translate, cross-culturally adapt and psychometrically analyse the Dialysis Symptom Index (DSI). Methods The study methods involved four phases: modification, translation, pilot-testing with a bilingual non-CKD sample and then psychometric testing with the target population. Content validity was assessed using an expert panel. Inter-rater agreement, test-retest reliability and Cronbach’s alpha coefficient were calculated to demonstrate reliability of the modified DSI. Discriminative and convergent validity were assessed to demonstrate construct validity. Results Content validity index during translation was 0.98. In the pilot study with 25 bilingual students a moderate to perfect agreement (Kappa statistic = 0.60-1.00) was found between English and Arabic versions of the modified DSI. The main study recruited 433 patients CKD with stages 4 and 5. The modified DSI was able to discriminate between non-dialysis and dialysis groups (p < 0.001) and demonstrated convergent validity with domains of the Kidney Disease Quality of Life short form. Excellent test-retest and internal consistency (Cronbach’s α = 0.91) reliability were also demonstrated. Conclusion The Arabic version of the modified DSI demonstrated good psychometric properties, measures the multidimensional nature of symptoms and can be used to assess symptom burden at different stages of CKD. The modified instrument, renamed the CKD Symptom Burden Index (CKD-SBI), should encourage greater clinical and research attention to symptom burden in CKD.

UV index measurement and model agreement: uncertainties and limitations

En les últimes dècades, l'increment dels nivells de radiació solar ultraviolada (UVR) que arriba a la Terra (principalment degut a la disminució d'ozó estratosfèric) juntament amb l'augment detectat en malalties relacionades amb l'exposició a la UVR, ha portat a un gran volum d'investigacions sobre la radiació solar en aquesta banda i els seus efectes en els humans. L'índex ultraviolat (UVI), que ha estat adoptat internacionalment, va ser definit amb el propòsit d'informar al públic general sobre els riscos d'exposar el cos nu a la UVR i per tal d'enviar missatges preventius. L'UVI es va definir inicialment com el valor màxim diari. No obstant, el seu ús actual s'ha ampliat i té sentit referir-se a un valor instantani o a una evolució diària del valor d'UVI mesurat, modelitzat o predit. El valor concret d'UVI està afectat per la geometria Sol-Terra, els núvols, l'ozó, els aerosols, l'altitud i l'albedo superficial. Les mesures d'UVI d'alta qualitat són essencials com a referència i per estudiar tendències a llarg termini; es necessiten també tècniques acurades de modelització per tal d'entendre els factors que afecten la UVR, per predir l'UVI i com a control de qualitat de les mesures. És d'esperar que les mesures més acurades d'UVI s'obtinguin amb espectroradiòmetres. No obstant, com que els costs d'aquests dispositius són elevats, és més habitual trobar dades d'UVI de radiòmetres eritemàtics (de fet, la majoria de les xarxes d'UVI estan equipades amb aquest tipus de sensors). Els millors resultats en modelització s'obtenen amb models de transferència radiativa de dispersió múltiple quan es coneix bé la informació d'entrada. No obstant, habitualment no es coneix informació d'entrada, com per exemple les propietats òptiques dels aerosols, la qual cosa pot portar a importants incerteses en la modelització. Sovint, s'utilitzen models més simples per aplicacions com ara la predicció d'UVI o l'elaboració de mapes d'UVI, ja que aquests són més ràpids i requereixen menys paràmetres d'entrada. Tenint en compte aquest marc de treball, l'objectiu general d'aquest estudi és analitzar l'acord al qual es pot arribar entre la mesura i la modelització d'UVI per condicions de cel sense núvols. D'aquesta manera, en aquest estudi es presenten comparacions model-mesura per diferents tècniques de modelització, diferents opcions d'entrada i per mesures d'UVI tant de radiòmetres eritemàtics com d'espectroradiòmeters. Com a conclusió general, es pot afirmar que la comparació model-mesura és molt útil per detectar limitacions i estimar incerteses tant en les modelitzacions com en les mesures. Pel que fa a la modelització, les principals limitacions que s'han trobat és la falta de coneixement de la informació d'aerosols considerada com a entrada dels models. També, s'han trobat importants diferències entre l'ozó mesurat des de satèl·lit i des de la superfície terrestre, la qual cosa pot portar a diferències importants en l'UVI modelitzat. PTUV, una nova i simple parametrització pel càlcul ràpid d'UVI per condicions de cel serens, ha estat desenvolupada en base a càlculs de transferència radiativa. La parametrització mostra una bona execució tant respecte el model base com en comparació amb diverses mesures d'UVI. PTUV ha demostrat la seva utilitat per aplicacions particulars com ara l'estudi de l'evolució anual de l'UVI per un cert lloc (Girona) i la composició de mapes d'alta resolució de valors d'UVI típics per un territori concret (Catalunya). En relació a les mesures, es constata que és molt important saber la resposta espectral dels radiòmetres eritemàtics per tal d'evitar grans incerteses a la mesura d'UVI. Aquest instruments, si estan ben caracteritzats, mostren una bona comparació amb els espectroradiòmetres d'alta qualitat en la mesura d'UVI. Les qüestions més importants respecte les mesures són la calibració i estabilitat a llarg termini. També, s'ha observat un efecte de temperatura en el PTFE, un material utilitzat en els difusors en alguns instruments, cosa que potencialment podria tenir implicacions importants en el camp experimental. Finalment, i pel que fa a les comparacions model-mesura, el millor acord s'ha trobat quan es consideren mesures d'UVI d'espectroradiòmetres d'alta qualitat i s'usen models de transferència radiativa que consideren les millors dades disponibles pel que fa als paràmetres òptics d'ozó i aerosols i els seus canvis en el temps. D'aquesta manera, l'acord pot ser tan alt dins un 0.1º% en UVI, i típicament entre menys d'un 3%. Aquest acord es veu altament deteriorat si s'ignora la informació d'aerosols i depèn de manera important del valor d'albedo de dispersió simple dels aerosols. Altres dades d'entrada del model, com ara l'albedo superficial i els perfils d'ozó i temperatura introdueixen una incertesa menor en els resultats de modelització.

Metabolic syndrome in overweight children from the city of Botucatu - São Paulo State - Brazil: agreement among six diagnostic criteria

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Evaluation of the cervical vertebrae maturation index in lateral cephalograms taken in different head positions

The aim of this study was to evaluate if upward or downward head inclination might interfere with determination of the growth stage, using cervical vertebrae maturation index (CVMI), in order to verify the accuracy of such diagnosis when radiographs are taken with errors. Forty-nine patients, 26 females and 23 males, aged 9 to 15 years, were submitted to 3 lateral cephalograms: normal (NHP), with 15° upward head inclination (NHP-Up), and with 15° downward head inclination (NHP-Down). Three examiners evaluated the CVMI on the 147 cephalograms. The agreement among examiners was higher in the evaluation of cephalograms taken in NHP. The weighted Kappa test revealed moderate to substantial agreement between NHP and NHP-Up and between NHP and NHP-Down. There was greater agreement between NHP-Up and NHP-Down. It may be concluded that the evaluation of the CVMI on cephalograms in NHP is different as compared with radiographs taken with inclinations. Both NHP-Up and NHP-Down exhibited greater disagreement in the interpretation among examiners, since the evaluation method was not designed for cephalograms with positioning errors.

Impact of inter-judge agreement on perceptual judgment of nasality

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Incomplete fissures in severe emphysematous patients evaluated with MDCT: Incidence and interobserver agreement among radiologists and pneumologists

Objective: Pulmonary fissures completeness predicts efficacy in endobronchial valves (EBV) implantation, a new lobar volume reduction therapy for severe emphysematous patients. We assessed the incidence of incomplete fissures and the interobserver agreement in its evaluation with MDCT, in severe emphysematous patients prior to EBV implantation. Materials and Methods: Volumetric thin-section CT scans of 35 patients (CODP GOLD 3/4, heterogeneous emphysema) were retrospectively reviewed by 2 pneumologists, 1 general and 2 experienced chest radiologists, independently and blinded for treatment outcome, and the pulmonary fissures were classified as either complete or incomplete. Interobserver agreement was assessed with Kappa index (KI). Results: Agreement between all readers for the left oblique, right oblique and horizontal fissure was, respectively, moderate (KI = 0.53), fair (KI = 0.37) and moderate (KI = 0.42). Highest agreement (99/105 fissures) was observed among experienced radiologists, being for left oblique, right oblique and horizontal, respectively, almost perfect (KI = 0.79), perfect (KI = 1.0) and moderate (KI = 0.52). These 2 reviewers found that all of 35 patients had at least one incomplete fissure, with a proportion of incomplete fissures assigned as 74/65%, 85/85% and 91/88%, respectively for the left oblique, right oblique and horizontal fissures. Conclusions: Pneumologists and radiologists agreed fairly to moderately in fissures analysis, while the experienced chest radiologists reached the highest clinically adequate agreement of 94%. We believe that clinical routine visual analysis of the fissures integrity can be done with a good degree of confidence in MDCT images, and experienced readers might be required. Also, a higher than expected incidence of incomplete fissures was described in our studied population. (C) 2012 Elsevier Ireland Ltd. All rights reserved.

Comparison of Bispectral Index (BIS) and Patient State Index (PSI) as objective measures of sedation and analgesia in critically ill/injured patients

Critically ill and injured patients require pain relief and sedation to reduce the body's stress response and to facilitate painful diagnostic and therapeutic procedures. Presently, the level of sedation and analgesia is guided by the use of clinical scores which can be unreliable. There is therefore, a need for an objective measure of sedation and analgesia. The Bispectral Index (BIS) and Patient State Index (PSI) were recently introduced into clinical practice as objective measures of the depth of analgesia and sedation. ^ Aim. To compare the different measures of sedation and analgesia (BIS and PSI) to the standard and commonly used modified Ramsay Score (MRS) and determine if the monitors can be used interchangeably. ^ Methods. MRS, BIS and PSI values were obtained in 50 postoperative cardiac surgery patients requiring analgesia and sedation from June to December 2004. The MRS, BIS and PSI values were assessed hourly for up to 6-h by a single observer. ^ The relationship between BIS and PSI values were explored using scatter plots and correlation between MRS, BIS and PSI was determined using Spearman's correlation coefficient. Intra-class correlation (ICC) was used to determine the inter-rater reliability of MRS, BIS and PSI. Kappa statistics was used to further evaluate the agreement between BIS and PSI at light, moderate and deep levels of sedation. ^ Results. There was a positive correlation between BIS and PSI values (Rho = 0.731, p<0.001). Intra-class correlation between BIS and PSI was 0.58, MRS and BIS 0.43 and MRS and PSI 0.27. Using Kappa statistics, agreement between MRS and BIS was 0.35 (95% CI: 0.27–0.43) and for MRS and PSI was 0.21 (95% CI: 0.15–0.28). The kappa statistic for BIS and PSI was 0.45 (95% CI: 0.37–0.52). Receiver operating characteristics (ROC) curves constructed to detect undersedation indicated an area under the curve (AUC) of 0.91 (95% CI = 0.87 to 0.94) for the BIS and 0.84 (95% CI = 0.79 to 0.88) for the PSI. For detection of oversedation, AUC for the BIS was 0.89 (95% CI = 0.84 to 0.92) and 0.80 (95% CI = 0.75 to 0.85) for the PSI. ^ Conclusions. There is a statistically significant positive correlation between the BIS and PSI but poor correlation and poor test agreement between the MRS and BIS as well as MRS and PSI. Both the BIS and PSI demonstrated a high level of prediction for undersedation and oversedation; however, the BIS and PSI can not be considered interchangeable monitors of sedation. ^

Deed of agreement ... established in ... MDCCLXXX : with a copious index, containing all the rules of the company.

Mode of access: Internet.

Validating the functional capacity index: A comparison of predicted versus observed total body scores

Background: The Functional Capacity Index (FCI) was designed to predict physical function 12 months after injury. We report a validation study of the FCI. Methods: This was a consecutive case series registered in the Queensland Trauma Registry who consented to the prospective 12-month telephone-administered follow-up study. FCI scores measured at 12 months were compared with those originally predicted. Results: Complete Abbreviated Injury Scale score information was available for 617 individuals, of whom 587 (95%) could be assigned at least one FCI score (range, 1-17). Agreement between the largest predicted FCI and observed FCI score was poor (kappa = 0.05; 95% confidence interval, 0.00-0.10) and explained only 1% of the variability in observed FCI. Using an encompassing model that included all FCI assignments, agreement remained poor (kappa = 0.05; 95% confidence interval, -0.02-0.12), and the model explained only 9% of the variability in observed FCI. Conclusion: The predicted functional capacity poorly agrees with actual functional outcomes. Further research should consider including other (noninjury) explanatory factors in predicting FCI at 12 months.

Osteoarthritis Index delivered by mobile phone (m-WOMAC) is valid, reliable, and responsive

Objectives: To evaluate the validity, reliability and responsiveness of EDC using the WOMAC® NRS 3.1 Index on Motorola V3 mobile phones. ---------- Methods: Patients with osteoarthritis (OA) undergoing primary unilateral hip or knee joint replacement surgery were assessed pre-operatively and 3-4 months post-operatively. Patients completed the WOMAC® Index in paper (p-WOMAC®) and electronic (m-WOMAC®) format in random order. ---------- Results: 24 men and 38 women with hip and knee OA participated and successfully completed the m-WOMAC® questionnaire. Pearson correlations between the summated total index scores for the p-WOMAC® and m-WOMAC® pre- and post-surgery were 0.98 and 0.99 (p<0.0001). There was no clinically important or statistically significant between-method difference in the adjusted total summated scores, pre- and post-surgery (adjusted mean difference = 4.44, p = 0.474 and 1.73, p = 0.781). Internal consistency estimates of m-WOMAC® reliability were 0.87 – 0.98. The m-WOMAC® detected clinically important, statistically significant (p<0.0001) improvements in pain, stiffness, function and total index score. ---------- Conclusions: Sixty-two patients with hip and knee OA successfully completed EDC by Motorola V3 mobile phone using the m-WOMAC® NRS3.1 Index; completion times averaging only 1-1.5 minutes longer than the p-WOMAC® Index. Data were successfully and securely transmitted from patients in Australia to a server in the USA. There was close agreement and no significant differences between m-WOMAC® and p-WOMAC® scores. This study confirms the validity, reliability and responsiveness of the Exco InTouch engineered, Java-based m-WOMAC® Index application. EDC with the m-WOMAC® Index provides unique opportunities for using quantitative measurement in clinical research and practice.

The relationship between chronic disease self-efficacy and nutritional status, functional ability and quality of life in older adults at risk of hospital readmission

Background and significance: Older adults with chronic diseases are at increasing risk of hospital admission and readmission. Approximately 75% of adults have at least one chronic condition, and the odds of developing a chronic condition increases with age. Chronic diseases consume about 70% of the total Australian health expenditure, and about 59% of hospital events for chronic conditions are potentially preventable. These figures have brought to light the importance of the management of chronic disease among the growing older population. Many studies have endeavoured to develop effective chronic disease management programs by applying social cognitive theory. However, limited studies have focused on chronic disease self-management in older adults at high risk of hospital readmission. Moreover, although the majority of studies have covered wide and valuable outcome measures, there is scant evidence on examining the fundamental health outcomes such as nutritional status, functional status and health-related quality of life. Aim: The aim of this research was to test social cognitive theory in relation to self-efficacy in managing chronic disease and three health outcomes, namely nutritional status, functional status, and health-related quality of life, in older adults at high risk of hospital readmission. Methods: A cross-sectional study design was employed for this research. Three studies were undertaken. Study One examined the nutritional status and validation of a nutritional screening tool; Study Two explored the relationships between participants. characteristics, self-efficacy beliefs, and health outcomes based on the study.s hypothesized model; Study Three tested a theoretical model based on social cognitive theory, which examines potential mechanisms of the mediation effects of social support and self-efficacy beliefs. One hundred and fifty-seven patients aged 65 years and older with a medical admission and at least one risk factor for readmission were recruited. Data were collected from medical records on demographics, medical history, and from self-report questionnaires. The nutrition data were collected by two registered nurses. For Study One, a contingency table and the kappa statistic was used to determine the validity of the Malnutrition Screening Tool. In Study Two, standard multiple regression, hierarchical multiple regression and logistic regression were undertaken to determine the significant influential predictors for the three health outcome measures. For Study Three, a structural equation modelling approach was taken to test the hypothesized self-efficacy model. Results: The findings of Study One suggested that a high prevalence of malnutrition continues to be a concern in older adults as the prevalence of malnutrition was 20.6% according to the Subjective Global Assessment. Additionally, the findings confirmed that the Malnutrition Screening Tool is a valid nutritional screening tool for hospitalized older adults at risk of readmission when compared to the Subjective Global Assessment with high sensitivity (94%), and specificity (89%) and substantial agreement between these two methods (k = .74, p < .001; 95% CI .62-.86). Analysis data for Study Two found that depressive symptoms and perceived social support were the two strongest influential factors for self-efficacy in managing chronic disease in a hierarchical multiple regression. Results of multivariable regression models suggested advancing age, depressive symptoms and less tangible support were three important predictors for malnutrition. In terms of functional status, a standard regression model found that social support was the strongest predictor for the Instrumental Activities of Daily Living, followed by self-efficacy in managing chronic disease. The results of standard multiple regression revealed that the number of hospital readmission risk factors adversely affected the physical component score, while depressive symptoms and self-efficacy beliefs were two significant predictors for the mental component score. In Study Three, the results of the structural equation modelling found that self-efficacy partially mediated the effect of health characteristics and depression on health-related quality of life. The health characteristics had strong direct effects on functional status and body mass index. The results also indicated that social support partially mediated the relationship between health characteristics and functional status. With regard to the joint effects of social support and self-efficacy, social support fully mediated the effect of health characteristics on self-efficacy, and self-efficacy partially mediated the effect of social support on functional status and health-related quality of life. The results also demonstrated that the models fitted the data well with relative high variance explained by the models, implying the hypothesized constructs under discussion were highly relevant, and hence the application for social cognitive theory in this context was supported. Conclusion: This thesis highlights the applicability of social cognitive theory on chronic disease self-management in older adults at risk of hospital readmission. Further studies are recommended to validate and continue to extend the development of social cognitive theory on chronic disease self-management in older adults to improve their nutritional and functional status, and health-related quality of life.

Agreement between self-reported use of in vitro fertilization or ovulation induction, and medical insurance claims in Australian women aged 28–36 years

STUDY QUESTION: What is the self-reported use of in vitro fertilization (IVF) and ovulation induction (OI) in comparison with insurance claims by Australian women aged 28–36 years? SUMMARY ANSWER: The self-reported use of IVF is quite likely to be valid; however, the use of OI is less well reported. WHAT IS KNOWN AND WHAT THIS PAPER ADDS: Population-based research often relies on the self-reported use of IVF and OI because access to medical records can be difficult and the data need to include sufficient personal identifying information for linkage to other data sources. There have been few attempts to explore the reliability of the self-reported use of IVF and OI using the linkage to medical insurance claims for either treatment. STUDY DESIGN: This prospective, population-based, longitudinal study included the cohort of women born during 1973–1978 and participating in the Australian Longitudinal Study on Women's Health (ALSWH) (n = 14247). From 1996 to 2009, participants were surveyed up to five times. PARTICIPANTS AND SETTING: Participants self-reported their use of IVF or OI in two mailed surveys when aged 28–33 and 31–36 years (n = 7280), respectively. This study links self-report survey responses and claims for treatment or medication from the universal national health insurance scheme (i.e. Medicare Australia). MAIN RESULTS AND THE ROLE OF CHANCE: Comparisons between self-reports and claims data were undertaken for all women consenting to the linkage (n = 3375). The self-reported use of IVF was compared with claims for OI for IVF (Kappa, K = 0.83), oocyte collection (K = 0.82), sperm preparation (K = 0.83), intracytoplasmic sperm injection (K = 0.40), fresh embryo transfers (K = 0.82), frozen embryo transfers (K = 0.64) and OI for IVF medication (K = 0.17). The self-reported use of OI was compared with ovulation monitoring (K = 0.52) and OI medication (K = 0.71). BIAS, CONFOUNDING AND OTHER REASONS FOR CAUTION: There is a possibility of selection bias due to the inclusion criteria for participants in this study: (1) completion of the last two surveys in a series of five and (2) consent to the linkage of their responses with Medicare data. GENERALIZABILITY TO OTHER POPULATIONS: The results are relevant to questionnaire-based research studies with infertile women in developed countries. STUDY FUNDING/COMPETING INTEREST(S): ALSWH is funded by the Australian Government Department of Health and Ageing. This research is funded by a National Health and Medical Research Council Centre of Research Excellence grant.