Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies

OBJECTIVE To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae. DESIGN Inter-therapist reliability study. SETTING Research laboratory. PARTICIPANTS 30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities. MAIN OUTCOME MEASURES Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists. RESULTS The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively. CONCLUSIONS Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.

Assessing the construct validity and reliability of the parental perception on antibiotics (PAPA) scales

Background: The overuse of antibiotics is becoming an increasing concern. Antibiotic resistance, which increases both the burden of disease, and the cost of health services, is perhaps the most profound impact of antibiotics overuse. Attempts have been made to develop instruments to measure the psychosocial constructs underlying antibiotics use, however, none of these instruments have undergone thorough psychometric validation. This study evaluates the psychometric properties of the Parental Perceptions on Antibiotics (PAPA) scales. The PAPA scales attempt to measure the factors influencing parental use of antibiotics in children. Methods: 1111 parents of children younger than 12 years old were recruited from primary schools’ parental meetings in the Eastern Province of Saudi Arabia from September 2012 to January 2013. The structure of the PAPA instrument was validated using Confirmatory Factor Analysis (CFA) with measurement model fit evaluated using the raw and scaled χ2, Goodness of Fit Index, and Root Mean Square Error of Approximation. Results: A five-factor model was confirmed with the model showing good fit. Constructs in the model include: Knowledge and Beliefs, Behaviors, Sources of information, Adherence, and Awareness about antibiotics resistance. The instrument was shown to have good internal consistency, and good discriminant and convergent validity. Conclusion: The availability of an instrument able to measure the psychosocial factors underlying antibiotics usage allows the risk factors underlying antibiotic use and overuse to now be investigated.

Theory-based self-management programs for promoting recovery in community-dwelling stroke survivors : a systematic review

Background Post-stroke recovery is demanding. Increasing studies have examined the effectiveness of self-management programs for stroke survivors. However no systematic review has been conducted to summarize the effectiveness of theory-based stroke self-management programs. Objectives The aim is to present the best available research evidence about effectiveness of theory-based self-management programs on community-dwelling stroke survivors’ recovery. Inclusion criteria Types of participants All community-residing adults aged 18 years or above, and had a clinical diagnosis of stroke. Types of interventions Studies which examined effectiveness of a self-management program underpinned by a theoretical or conceptual framework for community-dwelling stroke survivors. Types of studies Randomized controlled trials. Types of outcomes Primary outcomes included health-related quality of life and self-management behaviors. Secondary outcomes included physical (activities of daily living), psychological (self-efficacy, depressive symptoms), and social outcomes (community reintegration, perceived social support). Search Strategy A three-step approach was adopted to identify all relevant published and unpublished studies in English or Chinese. Methodological quality The methodological quality of the included studies was assessed using the Joanna Briggs Institute critical appraisal checklist for experimental studies. Data Collection A standardized JBI data extraction form was used. There was no disagreement between the two reviewers on the data extraction results. Data Synthesis There were incomplete details about the number of participants and the results in two studies, which makes it impossible to perform meta-analysis. A narrative summary of the effectiveness of stroke self-management programs is presented. Results Three studies were included. The key issues of concern in methodological quality included insufficient information about random assignment, allocation concealment, reliability and validity of the measuring instruments, absence of intention-to-treat analysis, and small sample sizes. The three programs were designed based on the Stanford Chronic Disease Self-management program and were underpinned by the principles of self-efficacy. One study showed improvement in the intervention group in family and social roles three months after program completion, and work productivity at six months as measured by the Stroke Specific Quality of Life Scale (SSQOL). The intervention group also had an increased mean self-efficacy score in communicating with physicians six months after program completion. The mean changes from baseline in these variables were significantly different from the control group. No significant difference was found in time spent in aerobic exercise between the intervention and control groups at three and six months after program completion. Another study, using SSQOL, showed a significant interaction effect by treatment and time on family roles, fine motor tasks, self-care, and work productivity. However there was no significant interaction by treatment and time on self-efficacy. The third study showed improvement in quality of life, community participation, and depressive symptoms among the participants receiving the stroke self-management program, Stanford Chronic Disease Self-management program, or usual care six months after program completion. However, there was no significant difference between the groups. Conclusions There is inconclusive evidence about the effectiveness of theory-based stroke self-management programs on community-dwelling stroke survivors’ recovery. However the preliminary evidence suggests potential benefits in improving stroke survivors’ quality of life and self-efficacy.

Can Business Continuity Management Create Reliability in Airports? (Issues from Theory into Practice)

The advances made within the aviation industry over the past several decades have significantly improved the availability, affordability and convenience of air travel and have been greatly beneficial in both social and economic terms. Air transport has developed into an irreplaceable service being relied on by millions of people each day and as such airports have become critical elements of national infrastructure to facilitate the movement of people and goods. As components of critical infrastructure (CI), airports are integral parts of a national economy supporting regional as well as national trade, commercial activity and employment. Therefore, any disruption or crisis which impacts the continuity of operations at airports can have significant negative consequences for the airport as a business, for the local economy and other nodes of transport infrastructure as well as for society. Due to the highly dynamic and volatile environment in which airports operate in, the aviation industry has faced many different challenges over the years ranging from terrorist attacks such as September 11, to health crises such as the SARS epidemic to system breakdowns such as the recent computer system outage at Virgin Blue Airlines in Australia. All these events have highlighted the vulnerability of airport systems to a range of disturbances as well as the gravity and widespread impact of any kind of discontinuity in airport functions. Such incidents thus emphasise the need for increasing resilience and reliability of airports and ensuring business continuity in the event of a crisis...

Dimensions and reliability of a hospital safety climate questionnaire in Chinese health-care practice

The aim of the current study was to examine the dimensions and reliability of a hospital safety climate questionnaire in Chinese health-care practice. To achieve this, a cross-sectional survey of health-care professionals was undertaken at a university teaching hospital in Shandong province, China. Our survey instrument demonstrated very high internal consistency, comparing well with previous research in this field conducted in other countries. Factor analysis highlighted four key dimensions of safety climate, which centred on employee personal protection, employee interactions, safetyrelated housekeeping and time pressures. Overall, this study suggests that hospital safety climate represents an important aspect of health-care practice in contemporary China.

A Satellite Selection Algorithm for Achieving High Reliability of Ambiguity Resolution with GPS and Beidou Constellations

Reliability of carrier phase ambiguity resolution (AR) of an integer least-squares (ILS) problem depends on ambiguity success rate (ASR), which in practice can be well approximated by the success probability of integer bootstrapping solutions. With the current GPS constellation, sufficiently high ASR of geometry-based model can only be achievable at certain percentage of time. As a result, high reliability of AR cannot be assured by the single constellation. In the event of dual constellations system (DCS), for example, GPS and Beidou, which provide more satellites in view, users can expect significant performance benefits such as AR reliability and high precision positioning solutions. Simply using all the satellites in view for AR and positioning is a straightforward solution, but does not necessarily lead to high reliability as it is hoped. The paper presents an alternative approach that selects a subset of the visible satellites to achieve a higher reliability performance of the AR solutions in a multi-GNSS environment, instead of using all the satellites. Traditionally, satellite selection algorithms are mostly based on the position dilution of precision (PDOP) in order to meet accuracy requirements. In this contribution, some reliability criteria are introduced for GNSS satellite selection, and a novel satellite selection algorithm for reliable ambiguity resolution (SARA) is developed. The SARA algorithm allows receivers to select a subset of satellites for achieving high ASR such as above 0.99. Numerical results from a simulated dual constellation cases show that with the SARA procedure, the percentages of ASR values in excess of 0.99 and the percentages of ratio-test values passing the threshold 3 are both higher than those directly using all satellites in view, particularly in the case of dual-constellation, the percentages of ASRs (>0.99) and ratio-test values (>3) could be as high as 98.0 and 98.5 % respectively, compared to 18.1 and 25.0 % without satellite selection process. It is also worth noting that the implementation of SARA is simple and the computation time is low, which can be applied in most real-time data processing applications.

Constrained control design for dynamic positioning of marine vehicles with control allocation

In this paper, we address the control design problem of positioning of over-actuated marine vehicles with control allocation. The proposed design is based on a combined position and velocity loops in a multi-variable anti-windup implementation together with a control allocation mapping. The vehicle modelling is considered with appropriate simplifications related to low-speed manoeuvring hydrodynamics and vehicle symmetry. The control design is considered together with a control allocation mapping. We derive analytical tuning rules based on requirements of closed-loop stability and performance. The anti- windup implementation of the controller is obtained by mapping the actuator-force constraint set into a constraint set for the generalized forces. This approach ensures that actuation capacity is not violated by constraining the generalized control forces; thus, the control allocation is simplified since it can be formulated as an unconstrained problem. The mapping can also be modified on-line based on actuator availability to provide actuator-failure accommodation. We provide a proof of the closed-loop stability and illustrate the performance using simulation scenarios for an open-frame underwater vehicle.

A survey of control allocation methods for underwater vehicles

A control allocation system implements a function that maps the desired control forces generated by the vehicle motion controller into the commands of the different actuators. In this article, a survey of control allocation methods for over-actuated underwater vehicles is presented. The methods are applicable for both surface vessels and underwater vehicles. The paper presents a survey of control allocation methods with focus on mathematical representation and solvability of thruster allocation problems. The paper is useful for university students and engineers who want to get an overview of state-of-the art control allocation methods as well as advance methods to solve more complex problems.

An integrated approach for earthwork allocation, sequencing and routing

Planning techniques for large scale earthworks have been considered in this article. To improve these activities a “block theoretic” approach was developed that provides an integrated solution consisting of an allocation of cuts to fills and a sequence of cuts and fills over time. It considers the constantly changing terrain by computing haulage routes dynamically. Consequently more realistic haulage costs are used in the decision making process. A digraph is utilised to describe the terrain surface which has been partitioned into uniform grids. It reflects the true state of the terrain, and is altered after each cut and fill. A shortest path algorithm is successively applied to calculate the cost of each haul, and these costs are summed over the entire sequence, to provide a total cost of haulage. To solve this integrated optimisation problem a variety of solution techniques were applied, including constructive algorithms, meta-heuristics and parallel programming. The extensive numerical investigations have successfully shown the applicability of our approach to real sized earthwork problems.

Selecting volatility forecasting models for portfolio allocation purposes

Techniques for evaluating and selecting multivariate volatility forecasts are not yet understood as well as their univariate counterparts. This paper considers the ability of different loss functions to discriminate between a set of competing forecasting models which are subsequently applied in a portfolio allocation context. It is found that a likelihood-based loss function outperforms its competitors, including those based on the given portfolio application. This result indicates that considering the particular application of forecasts is not necessarily the most effective basis on which to select models.

Antispin thrust allocation for marine vessels

In extreme weather conditions, thrusters on ships and rigs may be subject to severe thrust losses caused by ventilation and in-and-out-of-water events. When a thruster ventilates, air is sucked down from the surface and into the propeller. In more severe cases, parts of or even the whole propeller can be out of the water. These losses vary rapidly with time and cause increased wear and tear in addition to reduced thruster performance. In this paper, a thrust allocation strategy is proposed to reduce the effects of thrust losses and to reduce the possibility of multiple ventilation events. This thrust allocation strategy is named antispin thrust allocation, based on the analogous behavior of antispin wheel control of cars. The proposed thrust allocation strategy is important for improving the life span of the propulsion system and the accuracy of positioning for vessels conducting station keeping in terms of dynamic positioning or thruster-assisted position mooring. Application of this strategy can result in an increase of operational time and, thus, increased profitability. The performance of the proposed allocation strategy is demonstrated with experiments on a model ship.

Benefit of accident reduction considering the improvement of travel time reliability

Traffic accidents often cause lane closure, and diminish stability of travel time as well as the level of road services. On the other hand, research on the implementation of ITS services aiming at the reduction of traffic accidents has made considerable progress lately. However there has been little discussion on the benefits obtained by traffic accident reduction from the view point of travel time reliability. Therefore, in this research, relationships between traffic accidents and travel time reliability are examined, and the benefit of traffic accident reduction is calculated based on the scheduling model under travel time uncertainties. The results show the significance of traffic accident reduction for the improvement of travel time reliability.

A low-complexity flight controller for Unmanned Aircraft Systems with constrained control allocation

In this paper, we propose a framework for joint allocation and constrained control design of flight controllers for Unmanned Aircraft Systems (UAS). The actuator configuration is used to map actuator constraint set into the space of the aircraft generalised forces. By constraining the demanded generalised forces, we ensure that the allocation problem is always feasible; and therefore, it can be solved without constraints. This leads to an allocation problem that does not require on-line numerical optimisation. Furthermore, since the controller handles the constraints, and there is no need to implement heuristics to inform the controller about actuator saturation. The latter is fundamental for avoiding Pilot Induced Oscillations (PIO) in remotely operated UAS due to the rate limit on the aircraft control surfaces.

Test-retest reliability of four physical activity measures used in population surveys

Accurate monitoring of prevalence and trends in population levels of physical activity (PA) is a fundamental public health need. Test-retest reliability (repeatability) was assessed in population samples for four self-report PA measures: the Active Australia survey (AA, N=356), the short International Physical Activity Questionnaire (IPAQ, N=104), the physical activity items in the Behavioral Risk Factor Surveillance System (BRFSS, N=127) and in the Australian National Health Survey (NHS, N=122). Percent agreement and Kappa statistics were used to assess reliability of classification of activity status as 'active', 'insufficiently active' or 'sedentary'. Intraclass correlations (ICCs) were used to assess agreement on minutes of activity reported for each item of each survey and for total minutes. Percent agreement scores for activity status were very good on all four instruments, ranging from 60% for the NHS to 79% for the IPAQ. Corresponding Kappa statistics ranged from 0.40 (NHS) to 0.52 (AA). For individual items, ICCs were highest for walking (0.45 to 0.78) and vigorous activity (0.22 to 0.64) and lowest for the moderate questions (0.16 to 0.44). All four measures provide acceptable levels of test-retest reliability for assessing both activity status and sedentariness, and moderate reliability for assessing total minutes of activity.

Control design for positioning of underwater vehicles with control allocation

In this paper, we address the control design problem of positioning of over-actuated underwater vehicles. The proposed design is based on a control architecture with combined position and velocity loops and a control tuning method based on the decoupled models. We derive analytical tuning rules based on requirements of closed-loop stability, positioning performance, and the vehicle velocity dynamic characteristics. The vehicle modelling is considered from force to motion with appropriate simplifications related to low-speed manoeuvring hydrodynamics and vehicle symmetry. The control design is considered together with a control allocation mapping. This approach makes the control tuning independent of the characteristics of the force actuators and provides the basis for control reconfiguration in the presence of actuator failure. We propose an anti-wind-up implementation of the controller, which ensures that the constraints related to actuation capacity are not violated. This approach simplifies the control allocation problem since the actuator constraints are mapped into generalised force constraints.