Bridging the gap between different statistical approaches: An integrated framework for modelling

This paper proposes a template for modelling complex datasets that integrates traditional statistical modelling approaches with more recent advances in statistics and modelling through an exploratory framework. Our approach builds on the well-known and long standing traditional idea of 'good practice in statistics' by establishing a comprehensive framework for modelling that focuses on exploration, prediction, interpretation and reliability assessment, a relatively new idea that allows individual assessment of predictions. The integrated framework we present comprises two stages. The first involves the use of exploratory methods to help visually understand the data and identify a parsimonious set of explanatory variables. The second encompasses a two step modelling process, where the use of non-parametric methods such as decision trees and generalized additive models are promoted to identify important variables and their modelling relationship with the response before a final predictive model is considered. We focus on fitting the predictive model using parametric, non-parametric and Bayesian approaches. This paper is motivated by a medical problem where interest focuses on developing a risk stratification system for morbidity of 1,710 cardiac patients given a suite of demographic, clinical and preoperative variables. Although the methods we use are applied specifically to this case study, these methods can be applied across any field, irrespective of the type of response.

Simulation of air traffic controllers' behaviour using the operator choice model

The Operator Choice Model (OCM) was developed to model the behaviour of operators attending to complex tasks involving interdependent concurrent activities, such as in Air Traffic Control (ATC). The purpose of the OCM is to provide a flexible framework for modelling and simulation that can be used for quantitative analyses in human reliability assessment, comparison between human computer interaction (HCI) designs, and analysis of operator workload. The OCM virtual operator is essentially a cycle of four processes: Scan Classify Decide Action Perform Action. Once a cycle is complete, the operator will return to the Scan process. It is also possible to truncate a cycle and return to Scan after each of the processes. These processes are described using Continuous Time Probabilistic Automata (CTPA). The details of the probability and timing models are specific to the domain of application, and need to be specified using domain experts. We are building an application of the OCM for use in ATC. In order to develop a realistic model we are calibrating the probability and timing models that comprise each process using experimental data from a series of experiments conducted with student subjects. These experiments have identified the factors that influence perception and decision making in simplified conflict detection and resolution tasks. This paper presents an application of the OCM approach to a simple ATC conflict detection experiment. The aim is to calibrate the OCM so that its behaviour resembles that of the experimental subjects when it is challenged with the same task. Its behaviour should also interpolate when challenged with scenarios similar to those used to calibrate it. The approach illustrated here uses logistic regression to model the classifications made by the subjects. This model is fitted to the calibration data, and provides an extrapolation to classifications in scenarios outside of the calibration data. A simple strategy is used to calibrate the timing component of the model, and the results for reaction times are compared between the OCM and the student subjects. While this approach to timing does not capture the full complexity of the reaction time distribution seen in the data from the student subjects, the mean and the tail of the distributions are similar.

Formal analysis of human-computer interaction using model checking

Experiments with simulators allow psychologists to better understand the causes of human errors and build models of cognitive processes to be used in human reliability assessment (HRA). This paper investigates an approach to task failure analysis based on patterns of behaviour, by contrast to more traditional event-based approaches. It considers, as a case study, a formal model of an air traffic control (ATC) system which incorporates controller behaviour. The cognitive model is formalised in the CSP process algebra. Patterns of behaviour are expressed as temporal logic properties. Then a model-checking technique is used to verify whether the decomposition of the operator's behaviour into patterns is sound and complete with respect to the cognitive model. The decomposition is shown to be incomplete and a new behavioural pattern is identified, which appears to have been overlooked in the analysis of the data provided by the experiments with the simulator. This illustrates how formal analysis of operator models can yield fresh insights into how failures may arise in interactive systems.

Reliability measures for local nodes assessment in classification trees

Most of the modem developments with classification trees are aimed at improving their predictive capacity. This article considers a curiously neglected aspect of classification trees, namely the reliability of predictions that come from a given classification tree. In the sense that a node of a tree represents a point in the predictor space in the limit, the aim of this article is the development of localized assessment of the reliability of prediction rules. A classification tree may be used either to provide a probability forecast, where for each node the membership probabilities for each class constitutes the prediction, or a true classification where each new observation is predictively assigned to a unique class. Correspondingly, two types of reliability measure will be derived-namely, prediction reliability and classification reliability. We use bootstrapping methods as the main tool to construct these measures. We also provide a suite of graphical displays by which they may be easily appreciated. In addition to providing some estimate of the reliability of specific forecasts of each type, these measures can also be used to guide future data collection to improve the effectiveness of the tree model. The motivating example we give has a binary response, namely the presence or absence of a species of Eucalypt, Eucalyptus cloeziana, at a given sampling location in response to a suite of environmental covariates, (although the methods are not restricted to binary response data).

Reliability and validity of a brief physical activity assessment for use by family doctors

Objective: To evaluate the reliability and validity of a brief physical activity assessment tool suitable for doctors to use to identify inactive patients in the primary care setting. Methods: Volunteer family doctors (n = 8) screened consenting patients (n = 75) for physical activity participation using a brief physical activity assessment tool. Inter-rater reliability was assessed within one week (n = 71). Validity was assessed against an objective physical activity monitor (computer science and applications accelerometer; n = 42). Results: The brief physical activity assessment tool produced repeatable estimates of sufficient total physical activity, correctly classifying over 76% of cases (kappa 0.53, 95% confidence interval (CI) 0.33 to 0.72). The validity coefficient was reasonable (kappa 0.40, 95% CI 0.12 to 0.69), with good percentage agreement (71%). Conclusions: The brief physical activity assessment tool is a reliable instrument, with validity similar to that of more detailed self report measures of physical activity. It is a tool that can be used efficiently in routine primary healthcare services to identify insufficiently active patients who may need physical activity advice.

Reliability and validity of arm volume measurements for assessment of lymphedema

Background and Purpose. Arm lymphedema following breast cancer In this study, we assessed the surgery is a continuing problem. reliability and validity of circumferential measurements and water displacement for measuring upper-limb volume. Subjects. Participants included subjects who had had breast cancer surgery, including axillary dissection-19 with and 22 without a diagnosis of arm lymphedema-and 25 control subjects. Methods. Two raters measured each subject by using circumferential tape measurements at specified distances from the fingertips and in relation to anatornic landmarks and by using water displacement. Interrater reliability was calculated by analysis of variance and multilevel modeling. Volumes from circumferential measurements were compared with those from water displacement by use of means and correlation coefficients, respectively. The standard error of measurement, minimum detectable change (MDC), and limits of agreement (LOA) for volumes also were calculated. Results. Arm volumes obtained with these methods had high reliability. Compared with volumes from water displacement, volumes from circumferential measurements had high validity, although these volumes were slightly larger. Expected differences between subjects with and without clinical lymphedema following breast cancer were found. The MDC of volumes or the error associated with a single measure for data based oil anatomic landmarks was lower than that based oil distance from fingertips. The mean LOA with water displacement were lower for data based on anatomic landmarks than for data based on distance from fingertips. Discussion and Conclusion. Volumes calculated from anatomic landmarks are reliable, valid, and more accurate than those obtained from circumferential measurements based on distance from fingertips.

Measurement of functional ability following traumatic brain injury using the Clinical Outcomes Variable Scale: A reliability study

This study determined the inter-tester and intra-tester reliability of physiotherapists measuring functional motor ability of traumatic brain injury clients using the Clinical Outcomes Variable Scale (COVS). To test inter-tester reliability, 14 physiotherapists scored the ability of 16 videotaped patients to execute the items that comprise the COVS. Intra-tester reliability was determined by four physiotherapists repeating their assessments after one week, and three months later. The intra-class correlation coefficients (ICC) were very high for both inter-tester reliability (ICC > 0.97 for total COVS scores, ICC > 0.93 for individual COVS items) and intra-tester reliability (ICC > 0.97). This study demonstrates that physiotherapists are reliable in the administration of the COVS.

Functional assessment and care of children with neurodevelopmental disabilities

Objectives: To determine the relationship between pediatric assessment scores and ratings by parents and teachers regarding the amount of assistance required to complete basic activities of daily living; and to examine the relationship among scores for three commonly used pediatric assessments. Design: Prospective correlational study. 205 children with developmental disabilities. The children ranged in age from 11 to 87 mo and included 72 females and 133 males of diverse socioeconomic and ethnic backgrounds. The children were evaluated by using the Battelle Developmental Inventory Screening Test, Vineland Adaptive Behavior Scales, Functional Independence Measure for Children (WeeFIM(TM) instrument), and the Amount of Assistance Questionnaire, Results: The test-retest reliability coefficients for items on the Amount of Assistance Questionnaire were found to range from 0.82 to 0.97. Correlations among subscale scores and amount of assistance ratings were highest for the WeeFIM instrument and Battelle Developmental Inventory Screening Test. The highest correlation was between WeeFIM total rating and total amount of assistance rating (r = 0.91). Conclusion: Total WeeFIM instrument ratings and severity of disability were the best predictors of amount of assistance ratings provided by parents and teachers.

Development and standardization of the Self-regulation Skills Interview (SRSI): A new clinical assessment tool for acquired brain injury

The Self-regulation Skills Interview (SRSI) is a clinical tool designed to measure a range of metacognitive skills essential for rehabilitation planning, monitoring an individual's progress, and evaluating the outcome of treatment interventions. The results of the present study indicated that the SRSI has sound interrater reliability and test-retest reliability. A principle components analysis revealed three SRSI factors: Awareness, Readiness to Change, and Strategy Behavior. A comparison between a group of 61 participants with acquired brain injury (ABI) and a group of 43 non-brain-injured participants indicated that the participants with ABI had significantly lower levels of Awareness and Strategy Behavior, but that level of Readiness to Change was not significantly different between the two groups. The significant relationship observed between the SRSI factors and measures of neuropsychological functioning confirmed the concurrent validity of the scale and supports the value of the SRSI for post-acute assessment.

The family attitude scale: Reliability and validity of a new scale for measuring the emotional climate of families

Research on outcomes from psychiatric disorders has highlighted the importance of expressed emotion (EE), but its cost-effective measurement remains a challenge. This article describes development of the Family Attitude Scale (FAS), a 30-item instrument that can be completed by any informant. Its psychometric characteristics are reported in parents of undergraduate students and in 70 families with a schizophrenic member. The total FAS had high internal consistency in all samples, and reports of angry behaviour in FAS items showed acceptable inter-rater agreement. The FAS was associated with the reported anger, anger expression and anxiety of respondents. Substantial associations between the parents' FAS and the anger and anger expression of students was also observed. Parents of schizophrenic patients had higher FAS scores than parents of students, and the FAS was higher if disorder duration was longer or patient functioning was poorer. Hostility, high criticism and low warmth on the Camberwell Family Interview (CFI) were associated with a more negative FAS. The highest FAS in the family was a good predictor of a highly critical environment on the CFI. The FAS is a reliable and valid indicator of relationship stress and expressed anger that has wide applicability. (C) 1997 Elsevier Science Ireland Ltd.

Development, reliability, and concurrent validity of the modified inventory of potential reconstructive needs

Documentation of burn sequelae can be a difficult and time-consuming task. To date a reliable and systematic format for recording postburn trauma is lacking. The purpose of this research was two-fold: first, to develop a Modified Inventory of Potential Reconstructive Needs from the original Inventory of Potential Reconstructive Needs to allow methodical documentation of functional and cosmetic burn sequelae in all body surface areas of children with burns and, second, to establish interrater reliability and concurrent validity of the instrument, thus allowing its clinical application. Two raters scored the Modified Inventory of Potential Reconstructive Needs on 41 children with a range of burns types and severity. Excellent interrater reliability was demonstrated for both total (intraclass correlation coefficient = 0.996) and subsection inventory scores. Concurrent validity was also established with total scores showing strong positive correlations (0.73-0.76) with three indicators of burn severity. These findings provide initial support for the tool's clinical applicability, particularly in relation to rehabilitative planning and documentation.