The Comprehensive Assessment of Leadership for Learning: A Next Generation Formative Evaluation and Feedback System

Recent developments in federal policy have prompted the creation of state evaluation frameworks for principals and teachers that hold educators accountable for effective practices and student outcomes. These changes have created a demand for formative evaluation instruments that reflect current accountability pressures and can be used by schools to focus school improvement and leadership development efforts. The Comprehensive Assessment of Leadership for Learning (CALL) is a next generation, 360-degree on-line assessment and feedback system that reflect best practices in feedback design. Some unique characteristics of CALL include a focus on: leadership distributed throughout the school rather than as carried out by an individual leader; assessment of leadership tasks rather than perceptions of leadership practice; a focus on larger complex systems of middle and high school; and transparency of assessment design. This paper describes research contributing to the design and validation of the CALL survey instrument.

Comparing Campus Discipline Rates: A Multivariate Approach for Identifying Schools with Significantly Different than Expected Exclusionary Discipline Rates

Campus behavior management is important for ensuring classroom order and promoting positive academic outcomes. Previous studies have shown the importance of individual student and campus personnel characteristics and campus context for explaining campus discipline rates (e.g., rates of suspension and expulsion). Assessing campus discipline rates, while controlling for these individual and campus characteristics, is important for the monitoring, evaluation, and intervention role of policymakers as well as state and federal level education agencies. Systems or metrics exist that measure other student outcomes (i.e., academic performance) with controls for individual and campus characteristics, but none exist that monitor these differences for discipline rates across campuses. In this paper, we use a multivariate model to analyze a longitudinal, statewide dataset for all secondary students in Texas from 2000 to 2008 in order to examine how campus discipline rates differ across schools with statistically similar students, teachers, and campus characteristics. The findings are important for understanding that some schools with similar characteristics have significantly different exclusionary discipline rates, and they are important for informing policy and agency level decision-making. The methodology described can easily be used by monitoring agencies as well as local school districts.

When Policy Opportunity is not Enough: College Access and Enrollment Patterns among Texas Percent Plan Eligible Students

In 1998, Texas initiated a bold new statewide university admission policy aimed at increasing college access for traditionally underserved students in the state. House Bill 588 (known as the Texas Top 10 Percent Plan (TTPP)) guaranteed automatic admission to the college or university of their choice for all top performing students in Texas public high schools. Fourteen years after the plan’s implementation, we see great strides and complexities in understanding student outcomes as a result of the percent plan. However, the legal controversy over the percent plan both in Texas and other states incorporating similar yet distinctly motivated alternative admissions plans continues to play out from institutional decision boards to the highest court in the nation. This study seeks to add to that discussion by exploring two questions. Descriptively, what are the admission and enrollment patterns within racial/ethnic groups of percent plan eligible students, over time, for Texas elite, emergent elite, and remaining public institutions? Given that all eligible percent plan students may enter the institution of choice in Texas, does which type of institution a TTPP student chooses relate to their race/ethnicity? The descriptive story told by the admission and enrollment distributions of equally eligible TTPP students is a complex but compelling one. Fundamentally, it identifies that statistically different application and enrollment patterns exist for Hispanic and especially African American TTPP beneficiaries relative to their White and Asian American counterparts.

The Renaissance of Performance Funding for Higher Education: Policy Adoption, Implementation, and Impacts

Thesis (Ph.D.)--University of Washington, 2016-06

Adult education directors' perceptions of performance indicators for adult education programs in the state of Florida

Current demands for accountability in education emphasize outcome-based program evaluation and tie program funding to individual student performance. As has been the case for elementary and secondary programs, demands for accountability have increased pressure on adult educators to show evidence of the benefits of their programs in order to justify their financial support. In Florida, recent legislation fundamentally changes the delivery of adult education in the state by establishing a performance-based funding system that is based on outcomes related to the retention, completion, and employment of program participants.^ A performance-based funding system requires an evaluation process that stresses outcome indicators over indicators that focus on program context or process. Although the state has adopted indicators of program quality to evaluate its adult education programs, these indicators focus mostly on program processes rather than student outcomes. In addition, the indicators are not specifically tied to workforce development outcomes, a priority to federal and local funding agents.^ Improving the accountability of adult education programs and defining the role of these programs in Florida's Workforce Development System has become a priority to policy makers across the state. Another priority has been to involve adult education practitioners in every step of this process.^ This study was conducted in order to determine what performance indicators, as judged by the directors and supervisors of adult education programs in the state of Florida, are important and feasible in measuring the quality and effectiveness of these programs. The results of the study indicated that, both statewide and by region, the respondents consistently gave the highest ratings on both importance and feasibility to the indicators of Program Context, which reflect the needs, composition, and structure of the programs, and to the indicators of Educational Gain, which reflect learner progress in the attainment of basic skills and competencies. In turn, the respondents gave the lowest ratings on both importance and feasibility to the indicators in the areas of Return on State's Investment, Efficiency, Retention, and Workforce Training. In general, the indicators that received high ratings for importance also received high ratings for feasibility. ^

Essays on the Economics of Higher Education

At least since the seminal works of Jacob Mincer, labor economists have sought to understand how students make higher education investment decisions. Mincer’s original work seeks to understand how students decide how much education to accrue; subsequent work by various authors seeks to understand how students choose where to attend college, what field to major in, and whether to drop out of college.

Broadly speaking, this rich sub-field of literature contributes to society in two ways: First, it provides a better understanding of important social behaviors. Second, it helps policymakers anticipate the responses of students when evaluating various policy reforms.

While research on the higher education investment decisions of students has had an enormous impact on our understanding of society and has shaped countless education policies, students are only one interested party in the higher education landscape. In the jargon of economists, students represent only the `demand side’ of higher education---customers who are choosing options from a set of available alternatives. Opposite students are instructors and administrators who represent the `supply side’ of higher education---those who decide which options are available to students.

For similar reasons, it is also important to understand how individuals on the supply side of education make decisions: First, this provides a deeper understanding of the behaviors of important social institutions. Second, it helps policymakers anticipate the responses of instructors and administrators when evaluating various reforms. However, while there is substantial literature understanding decisions made on the demand side of education, there is far less attention paid to decisions on the supply side of education.

This dissertation uses empirical evidence to better understand how instructors and administrators make decisions and the implications of these decisions for students.

In the first chapter, I use data from Duke University and a Bayesian model of correlated learning to measure the signal quality of grades across academic fields. The correlated feature of the model allows grades in one academic field to signal ability in all other fields allowing me to measure both ‘own category' signal quality and ‘spillover' signal quality. Estimates reveal a clear division between information rich Science, Engineering, and Economics grades and less informative Humanities and Social Science grades. In many specifications, information spillovers are so powerful that precise Science, Engineering, and Economics grades are more informative about Humanities and Social Science abilities than Humanities and Social Science grades. This suggests students who take engineering courses during their Freshman year make more informed specialization decisions later in college.

In the second chapter, I use data from the University of Central Arkansas to understand how universities decide which courses to offer and how much to spend on instructors for these courses. Course offerings and instructor characteristics directly affect the courses students choose and the value they receive from these choices. This chapter reveals the university preferences over these student outcomes which best explain observed course offerings and instructors. This allows me to assess whether university incentives are aligned with students, to determine what alternative university choices would be preferred by students, and to illustrate how a revenue neutral tax/subsidy policy can induce a university to make these student-best decisions.

In the third chapter, co-authored with Thomas Ahn, Peter Arcidiacono, and Amy Hopson, we use data from the University of Kentucky to understand how instructors choose grading policies. In this chapter, we estimate an equilibrium model in which instructors choose grading policies and students choose courses and study effort given grading policies. In this model, instructors set both a grading intercept and a return on ability and effort. This builds a rich link between the grading policy decisions of instructors and the course choices of students. We use estimates of this model to infer what preference parameters best explain why instructors chose estimated grading policies. To illustrate the importance of these supply side decisions, we show changing grading policies can substantially reduce the gender gap in STEM enrollment.

Can a Broader Education Narrow the Gap? Evidence on Non-Academic Features of Schooling

Empirical studies of education programs and systems, by nature, rely upon use of student outcomes that are measurable. Often, these come in the form of test scores. However, in light of growing evidence about the long-run importance of other student skills and behaviors, the time has come for a broader approach to evaluating education. This dissertation undertakes experimental, quasi-experimental, and descriptive analyses to examine social, behavioral, and health-related mechanisms of the educational process. My overarching research question is simply, which inside- and outside-the-classroom features of schools and educational interventions are most beneficial to students in the long term? Furthermore, how can we apply this evidence toward informing policy that could effectively reduce stark social, educational, and economic inequalities?

The first study of three assesses mechanisms by which the Fast Track project, a randomized intervention in the early 1990s for high-risk children in four communities (Durham, NC; Nashville, TN; rural PA; and Seattle, WA), reduced delinquency, arrests, and health and mental health service utilization in adolescence through young adulthood (ages 12-20). A decomposition of treatment effects indicates that about a third of Fast Track’s impact on later crime outcomes can be accounted for by improvements in social and self-regulation skills during childhood (ages 6-11), such as prosocial behavior, emotion regulation and problem solving. These skills proved less valuable for the prevention of mental and physical health problems.

The second study contributes new evidence on how non-instructional investments – such as increased spending on school social workers, guidance counselors, and health services – affect multiple aspects of student performance and well-being. Merging several administrative data sources spanning the 1996-2013 school years in North Carolina, I use an instrumental variables approach to estimate the extent to which local expenditure shifts affect students’ academic and behavioral outcomes. My findings indicate that exogenous increases in spending on non-instructional services not only reduce student absenteeism and disciplinary problems (important predictors of long-term outcomes) but also significantly raise student achievement, in similar magnitude to corresponding increases in instructional spending. Furthermore, subgroup analyses suggest that investments in student support personnel such as social workers, health services, and guidance counselors, in schools with concentrated low-income student populations could go a long way toward closing socioeconomic achievement gaps.

The third study examines individual pathways that lead to high school graduation or dropout. It employs a variety of machine learning techniques, including decision trees, random forests with bagging and boosting, and support vector machines, to predict student dropout using longitudinal administrative data from North Carolina. I consider a large set of predictor measures from grades three through eight including academic achievement, behavioral indicators, and background characteristics. My findings indicate that the most important predictors include eighth grade absences, math scores, and age-for-grade as well as early reading scores. Support vector classification (with a high cost parameter and low gamma parameter) predicts high school dropout with the highest overall validity in the testing dataset at 90.1 percent followed by decision trees with boosting and interaction terms at 89.5 percent.

Closing the Gap: Young People's Views on Learning at School

The UK has been criticised for its inequitable education system, as student outcomes are strongly linked to parental socio-economic status. Children and young people experiencing poverty are less likely than their better off peers to leave school with good grades, which can perpetuate disadvantage in later life. The attainment gap between children and young people experiencing poverty and their better off peers in the UK is widening, despite an increasing media and policy focus in this area. Poverty-related educational inequality is a complex area and there is no conclusive evidence in what works to reduce its effects. While there is a plethora of research on the impact of poverty on education, very little of it includes the voice of children and young people and/or the psychological impact of poverty on learning. The importance of hearing the views of children and young people is central to educational psychology, as is social justice and facilitating access to the curriculum for all students. The barriers presented by the experience of poverty to learning are thus vital for educational psychologists to address. This study used qualitative methods to explore the learning journey of Key Stage 3 (age 12-13) young people experiencing poverty in an English coastal borough. Questions from the Little Box of Big Questions 2 were used as a tool in semi-structured interviews, in addition to questions devised by the researcher. Young people discussed aspects of their lives that enabled them to learn at school, and aspects that presented barriers to learning. The research used Positive Psychology, taking a strengths based approach to explore the skills young people thought they brought to education, skills they would like to develop, and how they could be supported in this. The study has highlighted themes that, if addressed, could potentially raise the attainment of children and young people experiencing poverty.

Prioritizing Purposes: Two American History Teachers' Choices Among Subject Matter and Classroom-Related Purposes

This study explores how two American history teachers - one novice and one experienced – make in-the-moment choices among their history subject matter and classroom-related purposes during the teaching of an American history unit. Using classroom observations, lesson artifacts, student work products, and deep, retrospective interviews with the teachers as they watched videos of their teaching, this study maps out in detail the teachers’ purposes, both within and across different lesson activity structures. This study finds that the novice and the experienced teacher navigated among their purposes differently from each other, and that the characteristics of each teacher’s purposes navigation aligned with student outcomes in that teacher’s class. The novice teacher acted more like a juggler, with visible, reactive navigation among each purpose operational throughout his teaching; student outcomes in his class were similarly fragmented and discrete. The experienced teacher presented more like an orchestra conductor, interweaving his purposes and anticipating the navigation decisions that would create a more seamless whole; student outcomes in his class were aligned with his holistic navigation of purposes. Findings from this study have important implications for education research and teacher practice, including the relationship between teachers’ navigation among purposes and desired student outcomes, the integral role of classroom-related purposes interwoven with history subject matter purposes in teachers’ decision-making, and the differences in purposes navigation between a novice and an experienced history teacher.

Investigating and Understanding Student Learning Outcomes in an Online and Face-to-Face Graduate Level Legal Administration Course: An Embedded Mixed Methods Design

Online education is a new teaching and learning medium with few current guidelines for faculty, administrators or students. Its rapid growth over the last decade has challenged academic institutions to keep up with the demand, while also providing a quality education. Our understanding of the factors that determine quality and effective online learning experiences that lead to student learning outcomes is still evolving. There is a lack of consensus on the effectiveness of online versus face-to-face education in the current research. The U.S. Department of Education conducted a meta-analysis in 2009 and concluded that student-learning outcomes in online courses were equal to and, often times, better than face-to-face traditional courses. Subsequent research has found contradictory findings, and further inquiry is necessary. The purpose of this embedded mixed methods design research study is to further our understanding of the factors that create quality and successful educational outcomes in an online course. To achieve this, the first phase of this study measured and compared learning outcomes in an online and in class graduate-level legal administration course. The second phase of the study entailed interviews with those students in both the online and face-to-face sections to understand their perspectives on the factors contributing to learning outcomes. Six themes emerged from the qualitative findings: convenience, higher order thinking, discussions, professor engagement, professor and student interaction, and face-to-face interaction. Findings from this study indicate the factors students perceive as contributing to learning outcomes in an online course are consistent among all students and are supported in the existing literature. Higher order thinking, however, emerged as a stronger theme than indicated in the current research, and the face-to-face nature of the traditional classroom may be more an issue of familiarity than a factor contributing to learning outcomes. As education continues to reach new heights and developments in technology advance, the factors found to contribute to student learning outcomes will be refined and enhanced. These developments will continue to transform the ways in which we deliver and receive knowledge in both traditional and online classrooms. While there is a growing body of research on online education, the field’s evolution has unsettled earlier findings and posed new areas to investigate.

Volunteering for success? An Analysis of Student Perceptions of the Pedagogical and Social Outcomes of Peer Mentoring in Higher Education

Previous studies into student volunteering have shown how formally organized volunteering activities have social, economic and practical benefits for student volunteers and the recipients of their volunteerism (Egerton, 2002; Vernon & Foster, 2002); moreover student volunteering provides the means by which undergraduates are able to acquire and hone transferable skills sought by employers following graduation (Eldridge & Wilson, 2003; Norris et al, 2006). Although much is known about the benefits of student volunteering, few previous studies have focused on the pedagogical value of student mentoring from the perspectives of both student mentee and mentor. Utilising grounded theory methodology this paper provides a critical analysis of an exploratory study analysing students’ perceptions of the pedagogical and social outcomes of student mentoring. It looks at students’ perceptions of mentoring, and being mentored, in terms of the learning experience and development of knowledge and skills. In doing so the paper considers how volunteering in a mentoring capacity adds ‘value’ to students’ experiences of higher education. From a public policy perspective, the economic, educational, vocational and social outcomes of student volunteering in general, and student mentoring in particular, make this an important subject meriting investigation. In terms of employability, the role of mentoring in equipping mentors and mentees with transferable, employability competencies has not been investigated. By critiquing the mentoring experiences of undergraduates within a single institution, this paper will make an important contribution to policy debates with regards to the pedagogical and employability related outcomes of student volunteering and mentoring.

How remote labs impact on course outcomes: various practices using VISIR

As technology is increasingly being seen as a facilitator to learning, open remote laboratories are increasingly available and in widespread use around the world. They provide some advantages over traditional hands-on labs or simulations. This paper presents the results of integrating the open remote laboratory VISIR into several courses, in various contexts and using various methodologies. These integrations, all related to higher education engineering, were designed by teachers with different perspectives to achieve a range of learning outcomes. The degree to which these VISIR-related outcomes were accomplished is discussed. The results reflect the levels of student engagement and learning and of teacher involvement. From the analysis, a connection between these two aspects was traced, although only related to the user profiles. VISIR is shown to be always of benefit for more motivated students, but this benefit can be maximized under particular conditions and characteristics.

Interoperability frameworks for competencies and learning outcomes: databases vs ontologies

The changes introduced into the European Higher Education Area (EHEA) by the Bologna Process, together with renewed pedagogical and methodological practices, have created a new teaching-learning paradigm: Student-Centred Learning. In addition, the last few years have been characterized by the application of Information Technologies, especially the Semantic Web, not only to the teaching-learning process, but also to administrative processes within learning institutions. On one hand, the aim of this study was to present a model for identifying and classifying Competencies and Learning Outcomes and, on the other hand, the computer applications of the information management model were developed, namely a relational Database and an Ontology.

Same wording, distinct concepts? Testing differences between expectancies and motives in a mediation model of alcohol outcomes.

Per definition, alcohol expectancies (after alcohol I expect X), and drinking motives (I drink to achieve X) are conceptually distinct constructs. Theorists have argued that motives mediate the association between expectancies and drinking outcomes. Yet, given the use of different instruments, do these constructs remain distinct when assessment items are matched? The present study tested to what extent motives mediated the link between expectancies and alcohol outcomes when identical items were used, first as expectancies and then as motives. A linear structural equation model was estimated based on a national representative sample of 5,779 alcohol-using students in Switzerland (mean age = 15.2 years). The results showed that expectancies explained up to 38% of the variance in motives. Together with motives, they explained up to 48% of the variance in alcohol outcomes (volume, 5+ drinking, and problems). In 10 of 12 outcomes, there was a significant mediated effect that was often higher than the direct expectancy effect. For coping, the expectancy effect was close to zero, indicating the strongest form of mediation. In only one case (conformity and 5+ drinking), there was a direct expectancy effect but no mediation. To conclude, the study demonstrates that motives are distinct from expectancies even when identical items are used. Motives are more proximally related to different alcohol outcomes, often mediating the effects of expectancies. Consequently, the effectiveness of interventions, particularly those aimed at coping drinkers, should be improved through a shift in focus from expectancies to drinking motives.