Reliability measures for local nodes assessment in classification trees

Most of the modem developments with classification trees are aimed at improving their predictive capacity. This article considers a curiously neglected aspect of classification trees, namely the reliability of predictions that come from a given classification tree. In the sense that a node of a tree represents a point in the predictor space in the limit, the aim of this article is the development of localized assessment of the reliability of prediction rules. A classification tree may be used either to provide a probability forecast, where for each node the membership probabilities for each class constitutes the prediction, or a true classification where each new observation is predictively assigned to a unique class. Correspondingly, two types of reliability measure will be derived-namely, prediction reliability and classification reliability. We use bootstrapping methods as the main tool to construct these measures. We also provide a suite of graphical displays by which they may be easily appreciated. In addition to providing some estimate of the reliability of specific forecasts of each type, these measures can also be used to guide future data collection to improve the effectiveness of the tree model. The motivating example we give has a binary response, namely the presence or absence of a species of Eucalypt, Eucalyptus cloeziana, at a given sampling location in response to a suite of environmental covariates, (although the methods are not restricted to binary response data).

The Northern HIPASS catalogue - data presentation, completeness and reliability measures

The Northern HIPASS catalogue (NHICAT) is the northern extension of the HIPASS catalogue, HICAT. This extension adds the sky area between the declination (Dec.) range of +2 degrees < delta < +25 degrees 30' to HICAT's Dec. range of -90 degrees < delta < +2 degrees. HIPASS is a blind H I survey using the Parkes Radio Telescope covering 71 per cent of the sky (including this northern extension) and a heliocentric velocity range of - 1280 to 12 700 km s(-1). The entire Virgo Cluster region has been observed in the Northern HIPASS. The galaxy catalogue, NHICAT, contains 1002 sources with nu(hel) > 300 km s(-1). Sources with -300 < nu(hel) < 300 km s(-1) were excluded to avoid contamination by Galactic emission. In total, the entire HIPASS survey has found 5317 galaxies identified purely by their HI content. The full galaxy catalogue is publicly available at http://hipass.aus-vo.org.

Purchasing Power Parities for Industry Comparisons Using Weighted Elteto-Koves-Szulc (EKS) Methods

This study has three main objectives. First, it develops a generalization of the commonly used EKS method to multilateral price comparisons. It is shown that the EKS system can be generalized so that weights can be attached to each of the link comparisons used in the EKS computations. These weights can account for differing levels of reliability of the underlying binary comparisons. Second, various reliability measures and corresponding weighting schemes are presented and their merits discussed. Finally, these new methods are applied to an international data set of manufacturing prices from the ICOP project. Although theoretically superior, it appears that the empirical impact of the weighted EKS method is generally small compared to the unweighted EKS. It is also found that this impact is larger when it is applied at lower levels of aggregation. Finally, the importance of using sector specific PPPs in assessing relative levels of manufacturing productivity is indicated.

Test-retest reliability of four physical activity measures used in population surveys

Accurate monitoring of prevalence and trends in population levels of physical activity (PA) is a fundamental public health need. Test-retest reliability (repeatability) was assessed in population samples for four self-report PA measures: the Active Australia survey (AA, N=356), the short International Physical Activity Questionnaire (IPAQ, N=104), the physical activity items in the Behavioral Risk Factor Surveillance System (BRFSS, N=127) and in the Australian National Health Survey (NHS, N=122). Percent agreement and Kappa statistics were used to assess reliability of classification of activity status as 'active', 'insufficiently active' or 'sedentary'. Intraclass correlations (ICCs) were used to assess agreement on minutes of activity reported for each item of each survey and for total minutes. Percent agreement scores for activity status were very good on all four instruments, ranging from 60% for the NHS to 79% for the IPAQ. Corresponding Kappa statistics ranged from 0.40 (NHS) to 0.52 (AA). For individual items, ICCs were highest for walking (0.45 to 0.78) and vigorous activity (0.22 to 0.64) and lowest for the moderate questions (0.16 to 0.44). All four measures provide acceptable levels of test-retest reliability for assessing both activity status and sedentariness, and moderate reliability for assessing total minutes of activity.

The List of Social Situation Problems: Reliability and validity in an adolescent Spanish-speaking sample

The aim of this study was to analyze the psychometric properties of the Spanish translation of the List of Social Situation Problems (LSSP; S. H. Spence, 1980). The questionnaire was administered to a sample of 388 adolescents between the ages of 12 and 18. Exploratory factor analysis identified four factors: Social Anxiety, Adult Oppositional, Assertiveness, and Making Friends, which accounted for 26.64% of the variance. Internal consistency of the total scale was high (alpha = .86). Correlations between the LSSP and two self-report measures of social anxiety, the Questionnaire about Interpersonal Difficulties for Adolescents (r = .45) and the Social Phobia and Anxiety Inventory (r = .48), were statistically significant. A significant difference was found between LSSP total scores for adolescents with and without social anxiety (d = 1.14), supporting the construct validity of the scale.

Suitability of acoustic perturbation measures in analysing periodic and nearly periodic voice signals

In recent years, acoustic perturbation measurement has gained clinical and research popularity due to the ease of availability of commercial acoustic analysing software packages in the market. However, because the measurement itself depends critically on the accuracy of frequency tracking from the voice signal, researchers argue that perturbation measures are not suitable for analysing dysphonic voice samples, which are aperiodic in nature. This study compares the fundamental frequency, relative amplitude perturbation, shimmer percent and noise-to-harmonic ratio between a group of dysphonic and non-dysphonic subjects. One hundred and twelve dysphonic subjects ( 93 females and 19 males) and 41 non-dysphonic subjects ( 35 females and 6 males) participated in the study. All the 153 voice samples were categorized into type I ( periodic or nearly periodic), type II ( signals with subharmonic frequencies that approach the fundamental frequency) and type III ( aperiodic) signals. Only the type I ( periodic and nearly periodic) voice signals were acoustically analysed for perturbation measures. Results revealed that the dysphonic female group presented significantly lower fundamental frequency, significantly higher relative amplitude perturbation and shimmer percent values than the non-dysphonic female group. However, none of these three perturbation measures were able to differentiate between male dysphonic and male non-dysphonic subjects. The noise-to-harmonic ratio failed to differentiate between the dysphonic and non-dysphonic voices for both gender groups. These results question the sensitivity of acoustic perturbation measures in detecting dysphonia and suggest that contemporary acoustic perturbation measures are not suitable for analysing dysphonic voice signals, which are even nearly periodic. Copyright (C) 2005 S. Karger AG, Basel.

Reliability and validity of a brief physical activity assessment for use by family doctors

Objective: To evaluate the reliability and validity of a brief physical activity assessment tool suitable for doctors to use to identify inactive patients in the primary care setting. Methods: Volunteer family doctors (n = 8) screened consenting patients (n = 75) for physical activity participation using a brief physical activity assessment tool. Inter-rater reliability was assessed within one week (n = 71). Validity was assessed against an objective physical activity monitor (computer science and applications accelerometer; n = 42). Results: The brief physical activity assessment tool produced repeatable estimates of sufficient total physical activity, correctly classifying over 76% of cases (kappa 0.53, 95% confidence interval (CI) 0.33 to 0.72). The validity coefficient was reasonable (kappa 0.40, 95% CI 0.12 to 0.69), with good percentage agreement (71%). Conclusions: The brief physical activity assessment tool is a reliable instrument, with validity similar to that of more detailed self report measures of physical activity. It is a tool that can be used efficiently in routine primary healthcare services to identify insufficiently active patients who may need physical activity advice.

Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database

Rationale and aims 'OTseeker' is an online database of randomized controlled trials (RCTs) and systematic reviews relevant to occupational therapy. RCTs are critically appraised and rated for quality using the 'PEDro' scale. We aimed to investigate the inter-rater reliability of the PEDro scale before and after revising rating guidelines. Methods In study 1, five raters scored 100 RCTs using the original PEDro scale guidelines. In study 2, two raters scored 40 different RCTs using revised guidelines. All RCTs were randomly selected from the OTseeker database. Reliability was calculated using Kappa and intraclass correlation coefficients [ICC (model 2,1)]. Results Inter-rater reliability was 'good to excellent' in the first study (Kappas >= 0.53; ICCs >= 0.71). After revising the rating guidelines, the reliability levels were equivalent or higher to those previously obtained (Kappas >= 0.53; ICCs >= 0.89), except for the item, 'groups similar at baseline', which still had moderate reliability (Kappa = 0.53). In study 2, two PEDro scale items, which had their definitions revised, 'less than 15% dropout' and 'point measures and variability', showed higher reliability. In both studies, the PEDro items with the lowest reliability were 'groups similar at baseline' (Kappas = 0.53), 'less than 15% dropout' (Kappas

A review of the psychometric properties of the Health of the Nation Outcomes Scales (HoNOS) family of measures

BACKGROUND: The Health of the Nation Outcome Scales was developed to routinely measure outcomes for adults with mental illness. Comparable instruments were also developed for children and adolescents (the Health of the Nation Outcome Scales for Children and Adolescents) and older people (the Health of the Nation Outcome Scales 65+). All three are being widely used as outcome measures in the United Kingdom, Australia and New Zealand. There is, however, no comprehensive review of these instruments. This paper fills this gap by reviewing the psychometric properties of each. METHOD: Articles and reports relating to the instruments were retrieved, and their findings synthesised to assess the instruments' validity (content, construct, concurrent, predictive), reliability (test-retest, inter-rater), sensitivity to change, and feasibility/utility. RESULTS: Mostly, the instruments perform adequately or better on most dimensions, although some of their psychometric properties warrant closer examination. CONCLUSION: Collectively, the Health of the Nation Outcome Scales family of measures can assess outcomes for different groups on a range of mental health-related constructs, and can be regarded as appropriate for routinely monitoring outcomes.

Test-retest reliability and practice effects of a rapid screen of mild traumatic brain injury

Test-retest reliabilities and practice affects of measures from the Rapid Screen of Concussion (RSC), in addition to the Digit Symbol Substitution Test (Digit Symbol), were examined. Twenty five male participants were tested three times; each testing session scheduled a week apart. The test-retest reliability estimates for most measures were reasonably good, ranging from .79 to .97. An exception was the delayed word recall test, which has had a reliability estimate of .66 for the first retest, and .59 for the second retest. Practice effects were evident from Times 1 to 2 on the sentence comprehension and delayed recall subtests of the RSC, Digit Symbol and a composite score. There was also a practice effect of the same magnitude found from Time 2 to Time 3 on Digit Symbol, delayed recall and the composite score. Statistics on measures for both the first and second retest intervals, with associated practice affects, are presented to enable the calculation of reliable change indices (RCI). The RCI may be used to assess any improvement in cognitive functioning after mild Traumatic Brain Injury.

Validation and reliability of the Modified Elderly Mobility Scale

Objective: To establish concurrent validity, interrater and test-retest reliability of the Modified Elderly Mobility Scale (MEMS). Methods: Ninety elderly patients were scored on the MEMS. To establish concurrent validity, 75 patients MEMS scores were compared to Functional Independence Measure (FIM) scores using Spearman's correlation. Videotaped patient performances were used to establish interrater and test-retest reliability using percentage absolute agreement and intraclass correlation coefficients (ICCs). Results: The total MEMS score demonstrated a significant association with the motor (r = 0.725) and total FIM scores (r = 0.718). Absolute agreement for interrater reliability was greater than 93% for all test items, with 97 and 98% for the two new measures, respectively. Test-retest reliability demonstrated similar high levels of absolute agreement and had ICCs ranging from 0.870 to 1.0. Conclusions: The MEMS is a quick, valid and reliable test of motor function of elderly patients with a spread of functional levels.