Comprehensive Validation Of A Measure Of Student School Engagement: A Pilot Study Of Middle School Students

Although it may sound reasonable that American education continues to be more effective at sending high school students to college, in a study conducted in 2009, The Council of the Great City Schools states that "slightly more than half of entering ninth grade students arrive performing below grade level in reading and math, while one in five entering ninth grade students is more than two years behind grade level...[and] 25% received support in the form of remedial literacy instruction or interventions" (Council of the Great City Schools, 2009). Students are distracted with technology (Lei & Zhao, 2005), family (Xu & Corno, 2003), medical illnesses (Nielson, 2009), learning disabilities and perhaps the most detrimental to academic success, the very lack of interest in school (Ruch, 1963). In a Johns Hopkins research study, Building a Graduation Nation - Colorado (Balfanz, 2008), warning signs were apparent years before the student dropped out of high school. The ninth grade was often referenced as a critical point that indicated success or failure to graduate high school. The research conducted by Johns Hopkins illustrates the problem: students who become disengaged from school have a much greater chance of dropping out of high school and not graduating. The first purpose of this study was to compare different measurement models of the Student School Engagement (SSE) using Factor Analysis to verify model fit with student engagement. The second purpose was to determine the extent to which the SSE instrument measures student school engagement by investigating convergent validity (via the SSE and Appleton, Christenson, Kim and Reschly's instrument and Fredricks, Blumenfeld, Friedel and Paris's instrument), discriminant validity (via Huebner's Student Life Satisfaction Survey) and criterion-related validity (via the sub-latent variables of Aspirations, Belonging and Productivity and student outcome measures such as achievement, attendance and discipline). Discriminant validity was established between the SSE and the Appleton, Christenson, Kim and Reschly's model and Fredricks, Blumenfeld, Friedel and Paris's (2005) Student Engagement Instruments (SEI). When confirming discriminant validity, the SSE's correlations were weak and statistically not significant, thus establishing discriminant validity with the SLSS. Criterion-related validity was established through structural equation modeling when the SSE was found to be a significant predictor of student outcome measures when both risk score and CSAP scores were used. The third purpose of this study was to assess the factorial invariance of the SSE instrument across gender to ensure the instrument is measuring the intended construct across different groups. Conclusively, configural, weak and metric invariances were established for the SSE as a non-significant change in chi-square indicating that all parameters including the error variances were invariant across groups of gender. Engagement is not a clearly defined psychological construct; it requires more research in order to fully comprehend its complexity. Hopefully, with parental and teacher involvement and a sense of community, student engagement can be nurtured to result in a meaningful attachment to school and academic success.

Validation of the Item-Attribute Matrix in TIMSS-Mathematics Using Multiple Regression and the LSDM

The purposes of this study were (1) to validate of the item-attribute matrix using two levels of attributes (Level 1 attributes and Level 2 sub-attributes), and (2) through retrofitting the diagnostic models to the mathematics test of the Trends in International Mathematics and Science Study (TIMSS), to evaluate the construct validity of TIMSS mathematics assessment by comparing the results of two assessment booklets. Item data were extracted from Booklets 2 and 3 for the 8th grade in TIMSS 2007, which included a total of 49 mathematics items and every student's response to every item. The study developed three categories of attributes at two levels: content, cognitive process (TIMSS or new), and comprehensive cognitive process (or IT) based on the TIMSS assessment framework, cognitive procedures, and item type. At level one, there were 4 content attributes (number, algebra, geometry, and data and chance), 3 TIMSS process attributes (knowing, applying, and reasoning), and 4 new process attributes (identifying, computing, judging, and reasoning). At level two, the level 1 attributes were further divided into 32 sub-attributes. There was only one level of IT attributes (multiple steps/responses, complexity, and constructed-response). Twelve Q-matrices (4 originally specified, 4 random, and 4 revised) were investigated with eleven Q-matrix models (QM1 ~ QM11) using multiple regression and the least squares distance method (LSDM). Comprehensive analyses indicated that the proposed Q-matrices explained most of the variance in item difficulty (i.e., 64% to 81%). The cognitive process attributes contributed to the item difficulties more than the content attributes, and the IT attributes contributed much more than both the content and process attributes. The new retrofitted process attributes explained the items better than the TIMSS process attributes. Results generated from the level 1 attributes and the level 2 attributes were consistent. Most attributes could be used to recover students' performance, but some attributes' probabilities showed unreasonable patterns. The analysis approaches could not demonstrate if the same construct validity was supported across booklets. The proposed attributes and Q-matrices explained the items of Booklet 2 better than the items of Booklet 3. The specified Q-matrices explained the items better than the random Q-matrices.

The Health Behavior Change Competency Checklist (HBCCC): Instrument development and psychometric evaluation.

More than half of morbidity and mortality in the United States can be attributed to behavior-related disease, such as tobacco use, physical inactivity, poor diet and alcohol consumption. Given the increased prevalence of behaviorally related medical health concerns, physician competence in the implementation of effective behavior change strategies is quickly becoming an essential skill. However, only recently have primary care residency programs begun to systematically teach and evaluate motivational interviewing skills critical to influencing health behavior, and use of standardized, objective assessment tools to assess skillfulness has been largely absent. This paper reports the development of a checklist, the Health Behavior Change Competency Checklist (HBCCC). The instrument captures the theoretical model of behavior change, motivational interviewing, in a practical and versatile manner. Psychometric evaluation demonstrated moderate efficacy. Namely, results indicated the HBCCC possesses good reliability, as evidenced by high internal consistency, and adequate construct validity. It also displayed considerable utility and practical application. While these results provide several reasons for confidence in the HBCCC, item revision and additional testing are required in order to establish it as a meaningful and valuable instrument.

Development of the DORI-TBI: An Investigation to Develop a New Screening Measure to Determine Consultation with a School-Based Brain Injury Resource Team

Brain injury is the leading cause of disability and death in children in the United States. Student re-entry into the school setting following a traumatic brain injury is crucial to student success. Multidisciplinary teams within the school district comprised of individuals with expertise in brain injury are ideal in implementing student specific treatment plans given their specialized training and wide range of expertise addressing student needs. Therefore, the purpose of this study is to develop and initially validate a quantitative instrument that school personnel can use to determine if a student, identified as having a traumatic brain injury, will benefit from district-level consultation from a brain injury team. Three studies were designed to investigate the research questions. In study one, the planning and construction of the DORI-TBI was completed. Study two addressed the content validity of the DORI-TBI through a comparison analysis with other referral forms, content review with experts in the field of TBI, and cognitive interviews with professionals to test the usability of the new screening tool. In study three, a field administration was conducted using vignettes to measure construct validity. Results produced a valid and reliable new screening instrument that can aid school-based teams to more efficiently utilize district level consultation with a brain injury support team.

Psychometric Investigation of the Check-In, Check-Out Fidelity of Implementation Measure

Suspension and expulsion are utilized frequently and disproportionality in schools in the United States. Many schools utilize Positive Behavioral Interventions and Supports (PBIS), a tiered framework to prevent problem behavior and reduce the use of discipline practices (Sugai et al., 2000). Check-In, Check-Out (CICO) is a targeted group behavioral intervention that is utilized within this framework in schools to prevent severe problem behavior in students that are beginning to exhibit externalizing and/or internalizing behavioral needs; thus, preventing the use of exclusionary discipline practices (Crone et al., 2010; Hawken & Horner, 2003). As the use of CICO in schools continues to grow, so too does the need for an instrument measuring its fidelity of implementation. The purpose of this study was to investigate the reliability and validity of the Check-In, Check-Out Fidelity of Implementation Measure (Crone et al., 2010), an instrument created to measure the fidelity of implementation of CICO intervention. This study assessed the psychometric properties of the instrument utilizing an archival data set collected by the statewide PBIS initiative in a western state in the U.S. The results demonstrated promising content validity, construct validity, internal consistency, and interrater reliability. A unidimensional structure was determined to be the best structure for the instrument based on parsimony and the strong results obtained from the item loadings, internal consistency, and interrater reliability. Implications for use and future research are discussed.