Program evaluation primer: A review of three evaluations

Program evaluation—a “tool used to assess the implementation and outcomes of a program, to increase a program’s efficiency and impact over time, and to demonstrate accountability” (MacDonald et. al, 2001, p. 1)—is an essential process to program assessment and improvement. This paper overviews three published program evaluations and considers important aspects of program evaluation more broadly.

Tobacco use prevention education (TUPE) programs in the state of Florida: Teachers' perceptions of program structure and effectiveness

This study described teacher perceptions of TUPE program effectiveness in Florida in an attempt to improve programs by identifying factors that might influence teacher motivation and performance. Very little work has been done to examine how teachers' perceptions are related to the effectiveness of TUPE programs. A statewide survey provided information about how teachers' perceptions of program effectiveness are affected by variables such as: program structure, barriers, tobacco use norms, and training variables. Data were obtained from a telephone survey conducted in Florida as part of the Tobacco Pilot Project (TPP). The sample included 296 middle school teachers and 282 high school teachers as well as 193 middle school principals and 190 high school principals. Correlational and hierarchical regression analyses identified correlates and predictors of teachers' ratings of effectiveness. Results suggest that use of peer leaders, more frequent evaluations, a higher degree of parent involvement, fewer barriers, greater student interest, and lower tolerance for tobacco use were correlated with higher ratings of program effectiveness. Furthermore, student interest, peer, staff, and community tolerance norms, peer leaders, program evaluation, and parent involvement predicted middle school teachers' perceptions. Parent tolerance, student interest, number of barriers, and more frequent program evaluation predicted high school teachers' perceptions. In addition, middle school teachers who reported a lower number of factors negatively associated with teacher receptivity were more likely to view TUPE programs more favorably than teachers who reported a greater number of these risk factors. This relationship was not as robust among the high school teacher sample. Differences between the middle and high school sample were found in the magnitude and number of significant correlations, the proportion of variance accounted for by predictor variables, and the strength of the relationship between the number of factors negatively associated with teacher receptivity and teachers' perceptions of TUPE effectiveness. These findings highlighted the importance of the timing, program features, and the external environment for enhancing or minimizing teachers' ratings of TUPE program effectiveness. In conclusion, significant increases in TUPE teachers' self-efficacy will occur through the participation of peers, parents, staff, and community leaders in different aspects of TUPE programs. ^

Self-esteem and job satisfaction of adults with mild mental retardation in sheltered workshops and supported employment programs

A sample of 200 adults with mild mental retardation were assessed on overall job satisfaction and self-esteem using the Vocational Program Evaluation Profile and the Coopersmith Self-esteem Inventory. The subjects worked either in a sheltered workshop or in a supported employment setting. Results indicated that there was a significant relationship between self-esteem and job satisfaction for both groups of subjects. In addition, subjects who worked in supported employment reported significantly higher levels of job satisfaction also. There was also an interaction between place of residence and place of employment when looking at self-esteem; those who live in a semi-independent home and work in supported employment reported the highest levels of self-esteem. These results are discussed in terms of the social validity of supported-employment for persons with mild mental retardation. ^

The effect of an enriched learning community on success and retention in chemistry courses

Since the mid-1990s, the United States has experienced a shortage of scientists and engineers, declining numbers of students choosing these fields as majors, and low student success and retention rates in these disciplines. Learning theorists, educational researchers, and practitioners believe that learning environments can be created so that an improvement in the numbers of students who complete courses successfully could be attained (Astin, 1993; Magolda & Terenzini, n.d.; O'Banion, 1997). Learning communities do this by providing high expectations, academic and social support, feedback during the entire educational process, and involvement with faculty, other students, and the institution (Ketcheson & Levine, 1999). ^ A program evaluation of an existing learning community of science, mathematics, and engineering majors was conducted to determine the extent to which the program met its goals and was effective from faculty and student perspectives. The program provided laptop computers, peer tutors, supplemental instruction with and without computer software, small class size, opportunities for contact with specialists in selected career fields, a resource library, and Peer-Led Team Learning. During the two years the project has existed, success, retention, and next-course continuation rates were higher than in traditional courses. Faculty and student interviews indicated there were many affective accomplishments as well. ^ Success and retention rates for one learning community class ( n = 27) and one traditional class (n = 61) in chemistry were collected and compared using Pearson chi square procedures ( p = .05). No statistically significant difference was found between the two groups. Data from an open-ended student survey about how specific elements of their course experiences contributed to success and persistence were analyzed by coding the responses and comparing the learning community and traditional classes. Substantial differences were found in their perceptions about the lecture, the lab, other supports used for the course, contact with other students, helping them reach their potential, and their recommendation about the course to others. Because of the limitation of small sample size, these differences are reported in descriptive terms. ^

Goma curriculum, a character education paradigm: Composing a text for shaping classroom character culture

The written text, and approaches to reading it, serves well as an analogy for the classroom space as a "text" that teachers are able to compose; and students are able to read, interpret meaning(s) of, and make responses to and about (Rosenblatt, 1988). Researchers point to ways in which the classroom can be conceptualized as a text to be evoked, experienced, and read (Freire & Macedo, 1987; Powell, 2009; Rosenblatt, 1988; Spears-Bunton & Powell, 2009).^ The present study analyzed secondary data including: 10 transcripts of teacher talks and six self-reports retrieved from the program evaluation archives of DOR Foundation. The data described six teachers' classroom experiences subsequent to professional development centered on Goma character education curriculum that was used during a summer youth program located in South Georgia. Goma, an acronym that stands for Goal, Objective, Method, and Attitude, is a character education paradigm derived from The Inclusive Community Building Ellison Model, the theoretical framework used for this study. The Model identifies conflict resolution as one of its five foci (Hunt, Howard, & Rice, 1998). Hunt (2006) conceived Goma as part of a 7-Step unitary process, also named the 7-Step pathway, to demonstrate how conflict resolution is accomplished within a variety of contexts.^ Analysis of the data involved: (a) a priori coding of teacher talks transcripts using the components of the Goma 7-Step pathway as coding categories, (b) emergent coding of teacher talks transcripts for the types of experiences teachers evidenced, and (c) emergent coding of teachers' self-reports for categories of teachers' instructional activities. Results of the study showed positive influence of Goma curriculum on participating teachers and their instructional practices. Teachers were shown to have had cognitive, instructional, emotional, and social experiences that were most evident when they reported changes in their attitudes toward their students, themselves, and their instructional practices. The present study provided implications for classroom teachers wherein all aspects of teachers' instructional practices can be guided by principles of positive character; and can be used to help compose the kinds of "texts" that may likely contribute to a classroom character culture.^

A Study of the Relational Component in an Academic Advisor Professional Development Program

The purpose of the study was to investigate the significance of the relational component of academic advisor training and development in the learning opportunities of the professional development program and the advisors’ evaluation score at Florida International University.

Application of an Education Research Lab (ERL) Model to a Community Sailing Program

This presentation showcases the application of a university-based education research lab (ERL) model to the evaluation of a community sailing program for individuals with disabilities. Presenters conceptualize the ERL model as a mutually beneficial relationship between universities and community education agencies.

Evaluating the Sandcastles Program: A Group Intervention for Children of Divorce

The Sandcastles program has been utilized nationwide as a one-time group intervention to assist children of divorcing parents. For several years Miami-Dade family court services mandated participation in the program for divorcing or separating families. Currently, there is a paucity of research and evaluation to ascertain the efficacy of the program. This symposium will provide details and discussion regarding the planning and process used to establish an evaluation plan to assess the effectiveness of the Sandcastles program for families in MiamiDade County. Any preliminary outcomes available at the time of the symposium will also be shared.

Goma Curriculum, A Character Education Paradigm: Composing a Text for Shaping Classroom Character Culture

The written text, and approaches to reading it, serves well as an analogy for the classroom space as a “text” that teachers are able to compose; and students are able to read, interpret meaning(s) of, and make responses to and about (Rosenblatt, 1988). Researchers point to ways in which the classroom can be conceptualized as a text to be evoked, experienced, and read (Freire & Macedo, 1987; Powell, 2009; Rosenblatt, 1988; Spears-Bunton & Powell, 2009). The present study analyzed secondary data including: 10 transcripts of teacher talks and six self-reports retrieved from the program evaluation archives of DOR Foundation. The data described six teachers’ classroom experiences subsequent to professional development centered on Goma character education curriculum that was used during a summer youth program located in South Georgia. Goma, an acronym that stands for Goal, Objective, Method, and Attitude, is a character education paradigm derived from The Inclusive Community Building Ellison Model, the theoretical framework used for this study. The Model identifies conflict resolution as one of its five foci (Hunt, Howard, & Rice, 1998). Hunt (2006) conceived Goma as part of a 7-Step unitary process, also named the 7-Step pathway, to demonstrate how conflict resolution is accomplished within a variety of contexts. Analysis of the data involved: (a) a priori coding of teacher talks transcripts using the components of the Goma 7-Step pathway as coding categories, (b) emergent coding of teacher talks transcripts for the types of experiences teachers evidenced, and (c) emergent coding of teachers’ self-reports for categories of teachers’ instructional activities. Results of the study showed positive influence of Goma curriculum on participating teachers and their instructional practices. Teachers were shown to have had cognitive, instructional, emotional, and social experiences that were most evident when they reported changes in their attitudes toward their students, themselves, and their instructional practices. The present study provided implications for classroom teachers wherein all aspects of teachers’ instructional practices can be guided by principles of positive character; and can be used to help compose the kinds of “texts” that may likely contribute to a classroom character culture.

Adult education directors' perceptions of performance indicators for adult education programs in the state of Florida

Current demands for accountability in education emphasize outcome-based program evaluation and tie program funding to individual student performance. As has been the case for elementary and secondary programs, demands for accountability have increased pressure on adult educators to show evidence of the benefits of their programs in order to justify their financial support. In Florida, recent legislation fundamentally changes the delivery of adult education in the state by establishing a performance-based funding system that is based on outcomes related to the retention, completion, and employment of program participants.^ A performance-based funding system requires an evaluation process that stresses outcome indicators over indicators that focus on program context or process. Although the state has adopted indicators of program quality to evaluate its adult education programs, these indicators focus mostly on program processes rather than student outcomes. In addition, the indicators are not specifically tied to workforce development outcomes, a priority to federal and local funding agents.^ Improving the accountability of adult education programs and defining the role of these programs in Florida's Workforce Development System has become a priority to policy makers across the state. Another priority has been to involve adult education practitioners in every step of this process.^ This study was conducted in order to determine what performance indicators, as judged by the directors and supervisors of adult education programs in the state of Florida, are important and feasible in measuring the quality and effectiveness of these programs. The results of the study indicated that, both statewide and by region, the respondents consistently gave the highest ratings on both importance and feasibility to the indicators of Program Context, which reflect the needs, composition, and structure of the programs, and to the indicators of Educational Gain, which reflect learner progress in the attainment of basic skills and competencies. In turn, the respondents gave the lowest ratings on both importance and feasibility to the indicators in the areas of Return on State's Investment, Efficiency, Retention, and Workforce Training. In general, the indicators that received high ratings for importance also received high ratings for feasibility. ^

Application of a PERT-Type System and "Crashing" in a Food Service Operation

A pert-type system, a combination of the program evaluation and review technique (PERT) and the critical path method (CPM), might be used by the hospitality industry to improve planning and control of complex functions. The author discusses this management science technique and how it can assist.

A determination of the effects of a ropes course on the self -esteem of undergraduate and graduate students

The purpose of this study was to evaluate the impact of the ropes course on the self-esteem of undergraduate and graduate students. The ropes course provides challenging experiential activities that facilitate personal confidence and group teamwork. The study relates to adult education, experiential education, and program evaluation within the context of hospitality and tourism management education. Quantitative data were based on the assessment of self-esteem through the completion of the Coopersmith Self-Esteem Inventory (SEI-C) modified for this study. Qualitative data were based on assessment of participants' reflective papers stating their perceptions of the ropes course. ^ The study compared a treatment group (31 undergraduate and 25 graduate students) which participated in the ropes course, and a control group (31 undergraduate and 25 graduate students) which did not participate. Both groups completed the pre- and post-treatment SEI-C at the same time intervals. The quantitative data were analyzed using a two-way repeated-measures analysis of variance (ANOVA). The qualitative data, comprised of reflective papers voluntarily written by 44 (79%) of the treatment group, were coded using four major themes reflecting the students' perceptions about the ropes course. ^ Scores on the pretest and posttest of the SEI-C were not significantly different for the two groups. The qualitative data showed a favorable impact of the ropes course. The discrepancy in the outcomes based on these two measures suggests that the SEI-C self-report paper-and-pencil instrument may not be sufficiently refined to evaluate the complex issue of self-esteem. The SEI-C, if used, should be supplemented by other evaluation measures, since the two measures may be evaluating different components of self-esteem. They may be also differentially affected by bias in scoring, or by statistical characteristics of reliability and validity. ^

A model of self -transformative identity development in troubled adolescent youth

Within the Stage II program evaluation of the Miami Youth Development Project's (YDP) Changing Lives Program (CLP), this study evaluated CLP intervention effectiveness in promoting positive change in emotion-focused identity exploration (i.e. feelings of personal expressiveness; PE) and a "negative" symptom of identity development (i.e. identity distress; ID) as a first step toward the investigation of a self-transformative model of identity development in adolescent youth. Using structural equation modeling techniques, this study found that participation in the CLP is associated with positive changes in PE (path = .841, p < .002), but not changes in ID. Increase in ID scores was found to be associated with increases in PE (path = .229, p < .002), as well. Intervention effects were not moderated by age/stage, gender, or ethnicity, though differences were found in the degree to which participating subgroups (African-American/Hispanic, male/female, 14-16 years old/17-19 years old) experience change in PE and ID. Findings also suggest that moderate levels of ID may not be deleterious to identity exploration and may be associated with active exploration. ^

A model of self-transformative identity development in troubled adolescent youth

Within the Stage II program evaluation of the Miami Youth Development Project's (YDP) Changing Lives Program (CLP), this study evaluated CLP intervention effectiveness in promoting positive change in emotion-focused identity exploration (i.e. feelings of personal expressiveness; PE) and a "negative" symptom of identity development (i.e. identity distress; ID) as a first step toward the investigation of a self-transformative model of identity development in adolescent youth. Using structural equation modeling techniques, this study found that participation in the CLP is associated with positive changes in PE (path = .841, p < .002), but not changes in ID. Increase in ID scores was found to be associated with increases in PE (path = .229, p < .002), as well. Intervention effects were not moderated by age/stage, gender, or ethnicity, though differences were found in the degree to which participating subgroups (African- American/Hispanic, male/female, 14-16 years old/17-19 years old) experience change in PE and ID. Findings also suggest that moderate levels of ID may not be deleterious to identity exploration and may be associated with active exploration.

The relative influence of social support, rehabilitation program participation, and inmate preincarceration characteristics on incarcerated substance abusers' perceptions of adjustment to community reintegration

This dissertation identifies, examines, and assesses the relative influence of identified empirically and conceptually relevant variables on incarcerated substance abusers' expectations of postrelease adjustment. A purposive sampling procedure was used to recruit 101 male and female substance-abusing offenders participating in prison- and jail-based drug treatment programs in south Florida. A 92-item survey questionnaire was used to collect basic demographic data; measure inmate preincarceration characteristics, social support, and rehabilitation program participation; and record archival data. Regression equations were developed utilizing ten different measures of the participants' expectations of their postrelease adjustment. Two equations yielded statistically significant F ratios; maintaining a stable living and maintaining abstinence. Twenty-two percent of the variance in respondents' expectations of maintaining a stable living was explained by preincarceration characteristics, social support, and rehabilitation program participation (F = 1.89; df = 13,87; p $<$.05). The only significant predictor variable was perception of social support (b = $-$.05; t = $-$3.6; p $<$.001). Twenty-three percent of the variance in respondents' expectations of maintaining abstinence from substances was explained by preincarceration characteristics, social support, and rehabilitation program participation (F = 2; df = 13,87; p $<$.05). Once again, the only significant predictor variable was perception of social support. The results of the analyses indicate that social support was the only important variable for understanding these respondents' efficacy expectations of postrelease abstinence and stable living. The results of this investigation demonstrate the complexity of the social support variable for prisoners, and identify social support as a potential rehabilitative resource for substance-abusing inmates. The results of this investigation underscore the importance of continued, detailed empirical study in order to understand and clarify how social support, efficacy expectations, and actual postrelease performance interrelate for this population of offenders.