Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

OBJECTIVE To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN Inter-therapist reliability study. SETTING Research laboratory. PARTICIPANTS 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.

Agreement between physiotherapists rating scapular posture in multiple planes in patients with neck pain: Reliability study

Objective Evaluation of scapular posture is an integral component of the clinical assessment of painful neck disorders. The aim of this study was to evaluate agreement between therapist judgements of scapula posture in multiple biomechanical planes in individuals with neck pain. Design Inter-therapist reliability study. Setting Research laboratory. Participants Fifteen participants with chronic neck pain. Main outcome measures Four physiotherapists recorded ratings of scapular orientation (relative to the thorax) in five different scapula postural planes (plane of scapula, sagittal plane, transverse plane, horizontal plane, and vertical plane) under four test conditions (at rest, and during three isometric shoulder conditions) in all participants. Inter-therapist reliability was expressed using both generalized and paired kappa coefficient. Results Following adjustment for expected agreement and the high prevalence of neutral ratings (81%), on average both the generalised kappa (0.37) as well as Cohen's Kappa for the two therapist pairs (0.45 and 0.42) demonstrated only slight to moderate inter-therapist reliability. Conclusions The findings suggest that ratings of scapular posture in individuals with neck pain by visual inspection has only slight to moderate reliability and should only be used in conjunction with other clinical tests when judging scapula function in these patients.

Family interventions in psychosis: a scale to measure therapist adherence

The efficacy of family interventions in psychosis is well documented. UK and USA schizophrenia treatment guidelines advocate the practice of family interventions within routine clinical services. However, less attention has been paid to the study of treatment fidelity and the tools used in its assessment. This study reports the inter-rater reliability of a new scale: Family Intervention in Psychosis-Adherence Scale (FIPAS). This measure is designed to assess therapist adherence to the Kuipers et al. (2002) family intervention in psychosis treatment manual. Reliability ratings were based on a sample of thirteen audiotapes drawn from a randomized controlled trial of family intervention. The results indicated that the majority of items of the FIPAS had acceptable levels of inter-rater reliability. The findings are discussed in terms of their implications for the training and monitoring of the effectiveness of practitioners for family interventions in psychosis.

Evaluation of therapist and client language in Motivational Interviewing (MI) sessions: A secondary analysis of data from the Southern Methodist Alcohol Research Trial (SMART) study

In the United States, “binge” drinking among college students is an emerging public health concern due to the significant physical and psychological effects on young adults. The focus is on identifying interventions that can help decrease high-risk drinking behavior among this group of drinkers. One such intervention is Motivational interviewing (MI), a client-centered therapy that aims at resolving client ambivalence by developing discrepancy and engaging the client in change talk. Of late, there is a growing interest in determining the active ingredients that influence the alliance between the therapist and the client. This study is a secondary analysis of the data obtained from the Southern Methodist Alcohol Research Trial (SMART) project, a dismantling trial of MI and feedback among heavy drinking college students. The present project examines the relationship between therapist and client language in MI sessions on a sample of “binge” drinking college students. Of the 126 SMART tapes, 30 tapes (‘MI with feedback’ group = 15, ‘MI only’ group = 15) were randomly selected for this study. MISC 2.1, a mutually exclusive and exhaustive coding system, was used to code the audio/videotaped MI sessions. Therapist and client language were analyzed for communication characteristics. Overall, therapists adopted a MI consistent style and clients were found to engage in change talk. Counselor acceptance, empathy, spirit, and complex reflections were all significantly related to client change talk (p-values ranged from 0.001 to 0.047). Additionally, therapist ‘advice without permission’ and MI Inconsistent therapist behaviors were strongly correlated with client sustain talk (p-values ranged from 0.006 to 0.048). Simple linear regression models showed a significant correlation between MI consistent (MICO) therapist language (independent variable) and change talk (dependent variable) and MI inconsistent (MIIN) therapist language (independent variable) and sustain talk (dependent variable). The study has several limitations such as small sample size, self-selection bias, poor inter-rater reliability for the global scales and the lack of a temporal measure of therapist and client language. Future studies might consider a larger sample size to obtain more statistical power. In addition the correlation between therapist language, client language and drinking outcome needs to be explored.^

Information technology support to improved construction processes : inter-disciplinarity in research and practice

Research and innovation in the built environment is increasingly taking on an inter-disciplinary nature. The built environment industry and professional practice have long adopted multi and inter-disciplinary practices. The application of IT in Construction is moving beyond the automation and replication of discrete mono and multi-disciplinary tasks to replicate and model the improved inter-disciplinary processes of modern design and construction practice. A major long-term research project underway at the University of Salford seeks to develop IT modelling capability to support the design of buildings and facilities that are buildable, maintainable, operable, sustainable, accessible, and have properties of acoustic, thermal and business support performance that are of a high standard. Such an IT modelling tool has been the dream of the research community for a long time. Recent advances in technology are beginning to make such a modelling tool feasible.----- Some of the key problems with its further research and development, and with its ultimate implementation, will be the challenges of multiple research and built environment stakeholders sharing a common vision, language and sense of trust. This paper explores these challenges as a set of research issues that underpin the development of appropriate technology to support realisable advances in construction process improvements.

Assessing the potential of inter-organisational shared services

Shared Services (SS) involves the convergence and streamlining of an organisation’s functions to ensure timely service delivery as effectively and efficiently as possible. As a management structure designed to promote value generation, cost savings and improved service delivery by leveraging on economies of scale, the idea of SS is driven by cost reduction and improvements in quality levels of service and efficiency. Current conventional wisdom is that the potential for SS is increasing due to the increasing costs of changing systems and business requirements for organisations and in implementing and running information systems. In addition, due to commoditisation of large information systems such as enterprise systems, many common, supporting functions across organisations are becoming more similar than not, leading to an increasing overlap in processes and fuelling the notion that it is possible for organisations to derive benefits from collaborating and sharing their common services through an inter-organisational shared services (IOSS) arrangement. While there is some research on traditional SS, very little research has been done on IOSS. In particular, it is unclear what are the potential drivers and inhibitors of IOSS. As the concepts of IOSS and SS are closely related to that of Outsourcing, and their distinction is sometimes blurred, this research has the first objective of seeking a clear conceptual understanding of the differences between SS and Outsourcing (in motivators, arrangements, benefits, disadvantages, etc) and based on this conceptual understanding, the second objective of this research is to develop a decision model (Shared Services Potential model) which would aid organisations in deciding which arrangement would be more appropriate for them to adopt in pursuit of process improvements for their operations. As the context of the study is on universities in higher education sharing administrative services common to or across them and with the assumption that such services were homogenous in nature, this thesis also reports on a case study. The case study involved face to face interviews from representatives of an Australian university to explore the potential for IOSS. Our key findings suggest that it is possible for universities to share services common across them as most of them were currently using the same systems although independently.

Inter-agency working : good intentions and interaction dynamics

This paper explores inter-agency working and examines the implications of inter-agency operations for delivering multi-domain service outcomes. Cross-agency collaborative approaches to service delivery are suggested to provide the vehicle for achieving integrated service and policy goals. However, it is argued these need to be crafted ‘fit’ for purpose’ and may not be the requisite approach for all joint purposes. Moreover, some commentators suggest that the optimism about these partnership arrangements and cross-agency actions to resolve complex multi-dimensional problems may be misplaced and propose that further research into the actual rather than desired consequences of these arrangements may find that, at times, partnership working creates negative effects. While collaboration and partnerships are often framed as the way to achieve real breakthroughs in service delivery across agencies, there remain key challenges to interagency working. As more and insistent calls for agencies and other community actors to work together in resolving complex social problems are heeded, the implications of working across organizational boundaries need to be further investigated. This paper investigates cases of inter-agency programmes to understand the dimensions and limitations of inter-agency working. The paper concludes by offering a framework for better inter-agency working that has applicability across all sectors.

Health professional partnerships and their impact on Aboriginal health : an occupational therapist's and Aboriginal health worker's perspective

Objective: To describe the extent and nature of demonstrated professional partnerships between occupational therapists and Aboriginal health workers in rural and remote communities of North Queensland. The study identifies ways in which professional partnerships improve client services and enhance occupational therapy outcomes through exploring the aspects of communication, collaboration and bridging cultural boundaries.---------- Design: Data collected via in-depth, semistructured telephone interviews. ---------- Setting: Aboriginal and mainstream health and human service organisations in rural and remote North Queensland. Rural and remote areas were identified using the Accessibility and Remoteness Index of Australia codes. ---------- Participants: Seven participants working in rural and remote areas of North Queensland, comprising four occupational therapists and three Aboriginal health workers. All participants were female. ---------- Results: Participants identified five core themes when describing the extent and nature of professional partnerships between occupational therapists and Aboriginal health workers. Themes include: professional interaction; perception of professional roles; benefits to the client; professional interdependence; and significance of Aboriginal culture. According to participants, when partnerships between occupational therapists and Aboriginal health workers were formed, clients received a more culturally appropriate service, were more comfortable in the presence of the occupational therapist, obtained a greater understanding of occupational therapy assessment and intervention, and felt valued in the health care process. ---------- Conclusions: This study substantiates the necessity for the formation of professional partnerships between occupational therapists and Aboriginal health workers. The findings suggest that participation in professional partnerships has positive implications for occupational therapists working with Aboriginal clients and Aboriginal health workers in rural and remote regions of North Queensland.