Propensity score matching and variations on the balancing test

Balancing tests are diagnostics designed for use with propensity score methods, a widely used non-experimental approach in the evaluation literature. Such tests provide useful information on whether plausible counterfactuals have been created. Currently, multiple balancing tests exist in the literature but it is unclear which is the most useful. This article highlights the poor size properties of commonly employed balancing tests and attempts to shed some light on the link between the results of balancing tests and bias of the evaluation estimator. The simulation results suggest that in scenarios where the conditional independence assumption holds, a permutation version of the balancing test described in Dehejia and Wahba (Rev Econ Stat 84:151–161, 2002) can be useful in applied study. The proposed test has good size properties. In addition, the test appears to have good power for detecting a misspecification in the link function and some power for detecting an omission of relevant non-linear terms involving variables that are included at a lower order.

The test-retest reliability of the child-initiated pretend play assessment

OBJECTIVE: The purpose of this study is to establish the test–retest reliability of the Child-Initiated Pretend Play Assessment (ChIPPA) (Stagnitti, 2002a; Stagnitti, Unsworth, & Rodger, 2000).

METHOD: The first author rated 38 preschool children ages 4 and 5 years (4 with developmental delay and 34 typically developing) on the ChIPPA. The ChIPPA employs conventional play materials and unstructured play materials to assess three qualities of a child's play ability: elaborateness of play action, ability to substitute objects during play, and the child's need to imitate the modelled actions of the examiner. The ChIPPA was administered twice, at a 2-week interval, to each participant.

RESULTS: Test–retest intraclass correlation coefficients (ICCs) (Type 2,1) calculated for each of the three elaborate play measures ranged from .73 to .84. A test–retest ICC of .56 was obtained for object substitution with unstructured play materials. The test–retest ICC obtained for the combined score for unstructured and conventional play materials was .57. Percentage agreement figures ranging from 63.2% to 84.2% were obtained on test–retest of the object substitution with conventional toys and imitated actions measures. There was no significant difference between test and retest scores for these measures based on a Wilcoxon Matched Pairs Signed-Ranks Test (Wilcoxon Sign Test).

CONCLUSION: Elaborate play scores, object substitution with conventional toys score, and imitation scores on the ChIPPA showed stability over time. Object substitution scores using unstructured materials were the least stable play measures and appeared to be related to the child's play themes. Since play is the primary occupation of children, it is essential that therapists have a reliable measure of play behavior. The test–retest reliability results from the ChIPPA provide evidence that this assessment produces a stable measure of play behavior that can then guide therapists when planning intervention strategies for children.

A comparison of four functional test in discriminating fallers from non-fallers in older people

Purpose : Which functional tests on mobility and balance can better screen older people at risk of falls is unclear. This study aims to compare the Berg Balance Scale (BBS), Tinetti Mobility Score (TMS), Elderly Mobility Scale (EMS) and Timed Up and Go test (TUG) in discriminating fallers from non-fallers in older people.
Method : This was a case-control study involving one rater who conducted a mobility and balance assessment on subjects using the four functional tests in random sequence. Subjects recruited included 17 and 22 older people with a history of single and multiple falls respectively from a public Falls Clinic, and 39 community-dwellers without fall history and whose age, sex and BMI matched those of the fallers. All subjects underwent the mobility and balance assessment within one day.
Results : Single fallers performed better than multiple fallers in all four functional tests but were worse than non-fallers in the BBS, TMS and TUG. The BBS demonstrated the best discriminating ability, with high sensitivity and specificity. The BBS item 'pick up an object from the floor' was the best at screening fallers.
Conclusion : BBS was the most powerful functional test of the four in discriminating fallers from non-faller.

Tests of measurement invariance failed to support the application of the "then-test"

Objective
The use of then-test (retrospective pre-test) scores has frequently been proposed as a solution to potential confounding of change scores because of response shift, as it is assumed that then-test and post-test responses are provided from the same perspective. However, this assumption has not been formally tested using robust quantitative methods. The aim of this study was to compare the psychometric performance of then-test/post-test with traditional pre-test/post-test data and assessing whether the resulting data structures support the application of the then-test for evaluations of chronic disease self-management interventions.

Study Design and Setting
Pre-test, post-test, and then-test data were collected from 314 participants of self-management courses using the Health Education Impact Questionnaire (heiQ). The derived change scores (pre-test/post-test; then-test/post-test) were examined for their psychometric performance using tests of measurement invariance.

Results
Few questionnaire items were noninvariant across pre-test/post-test, with four items identified and requiring removal to enable an unbiased comparison of factor means. In contrast, 12 items were identified and required removal in then-test/post-test data to avoid biased change score estimates.

Conclusion
Traditional pre-test/post-test data appear to be robust with little indication of response shift. In contrast, the weaker psychometric performance of then-test/post-test data suggests psychometric flaws that may be the result of implicit theory of change, social desirability, and recall bias.

Development and validation of the Approach-Iron Skill Test for use in golf

The primary aim of this study was to develop and validate a golf-specific approach-iron test for use with elite and high-level amateur golfers. Elite (n=26) and high-level amateur (n=23) golfers were recruited for this study. The ‘Approach-Iron Skill Test’ requires players to hit a total of 27 shots. Specifically, three shots are hit at each of nine targets on a specially constructed driving range in a randomised order. A real-time launch monitor positioned behind the player, measured the carry distance for each of these shots. A scoring system was developed based on the percentage error index of each shot, meaning that 81 points was the maximum score possible (with a maximum of three points per shot). Two rounds of the test were performed. For both rounds of the test, elite-level golfers scored significantly higher than their high-level amateur counterparts (56.3±5.6 and 58.5±4.6 points versus 46.0±6.3 and 46.1±6.7 points, respectively) (P<0.05). For both elite and high-level players, 95% limits of agreement statistics also indicated that the test showed good test–retest reliability (2.1±7.9 and 0.2±10.8, respectively). Due to the clinimetric properties of the test, we conclude that the Approach-Iron Skill Test is suitable for further examination with the players examined in this study.

Development and measurement properties of a putting skill test for high-level golf

Despite a recent increase in the amount of research investigating performance in golf, a comprehensive putting skill test has not been reported in the peer-reviewed literature. In this study, the Golf Australia Putting Test (GAPT) was developed and a series of measurement properties were assessed. Elite (n = 18) and high-level amateur (HLA; n = 22) participants completed six single putts from various areas on six concentric circles (circle radii = 0.9, 1.5, 3.0, 4.6, 6.1 and 7.6 m). Using a scoring system that rewarded participants for holing putts from longer distances, the maximum score from a single round of the test (i.e. 36 putts) was 27 points. After two rounds of the test were completed by all players, a subsample of participants (elite, n = 15; HLA, n = 7) had their putting performance recorded during tournament play for a period of 90 days to assess criterion (predictive) validity of the test. The reliability, sensitivity and discriminative validity of the GAPT were also assessed. Better agreement between Rounds 1 and 2 scores was noted in the elite group, whilst reliability values were similar for both groups. Further, the GAPT scores were shown to predict players from the elite and high-ability groups with a low classification error. An equation for predicting on-course performance from GAPT scores was also developed. Findings from this study indicate that the GAPT is a valid and reliable tool for high-level players and the GAPT may be used for player evaluation in the field.

Interrater reliability assessment using the Test of Gross Motor Development-2

Objectives: The aim was to examine interrater reliability of the object control subtest from the Test of Gross Motor Development-2 by live observation in a school field setting. Design:: Reliability Study - cross sectional. Methods: Raters were rated on their ability to agree on (1) the raw total for the six object control skills; (2) each skill performance and (3) the skill components. Agreement for the object control subtest and the individual skills was assessed by an intraclass correlation (ICC) and a kappa statistic assessed for skill component agreement. Results: A total of 37 children (65% girls) aged 4-8 years (M= 6.2, SD=0.8) were assessed in six skills by two raters; equating to 222 skill tests. Interrater reliability was excellent for the object control subset (ICC= 0.93), and for individual skills, highest for the dribble (ICC= 0.94) followed by strike (ICC= 0.85), overhand throw (ICC= 0.84), underhand roll (ICC= 0.82), kick (ICC= 0.80) and the catch (ICC= 0.71). The strike and the throw had more components with less agreement. Conclusions: Even though the overall subtest score and individual skill agreement was good, some skill components had lower agreement, suggesting these may be more problematic to assess. This may mean some skill components need to be specified differently in order to improve component reliability.

Development and implementation of Taiwan's child health literacy test

Objectives: The aims of this study were to develop Taiwan's Child Health Literacy Test and to undertake a nation-wide survey in order to determine the current status of Taiwanese sixth graders' health literacy, and to understand the association between health literacy, healthy behavior, and health status. absp Methods: Taiwan's Child Health Literacy Test was developed through the process of concept clarification, a qualitative pilot, a development pilot, and a field test. In the field test, 162,609 sixth graders (56.9%) from 2,235 schools (83.3%) nationwide completed the questionnaire. We also collected the students' dates of birth, BMIs, self-reported health and healthy behaviors. absp Results: The final test consisted of 32 questions with item discrimination of 0.55-1.89 and item difficulty of-1.7-0.41 according to IRT; Cronbach's a was 0.87. Based on this information, the test was deemed appropriate for basic health literacy screening among children. Nation-wide, the average score for sixth graders' health literacy was 23.97 points (total score 32 points), with a correct rate of 74.9%. Those who were "good" in self-reported health scored highest in health literacy (M = 24.29). Health literacy was significantly positively related to healthy behavior (r = .25, p< .05), and negatively to risky behavior (r =-.28, p< .05). absp Conclusions: This study was the first curriculum-based child health literacy test developed from the viewpoints of both teachers and pupils in Taiwan through a rigorous procedure. The nationwide survey results may serve as a reference for decision-makers at the national health education level.