Program evaluation primer: A review of three evaluations

Program evaluation—a “tool used to assess the implementation and outcomes of a program, to increase a program’s efficiency and impact over time, and to demonstrate accountability” (MacDonald et. al, 2001, p. 1)—is an essential process to program assessment and improvement. This paper overviews three published program evaluations and considers important aspects of program evaluation more broadly.

Tobacco use prevention education (TUPE) programs in the state of Florida: Teachers' perceptions of program structure and effectiveness

This study described teacher perceptions of TUPE program effectiveness in Florida in an attempt to improve programs by identifying factors that might influence teacher motivation and performance. Very little work has been done to examine how teachers' perceptions are related to the effectiveness of TUPE programs. A statewide survey provided information about how teachers' perceptions of program effectiveness are affected by variables such as: program structure, barriers, tobacco use norms, and training variables. Data were obtained from a telephone survey conducted in Florida as part of the Tobacco Pilot Project (TPP). The sample included 296 middle school teachers and 282 high school teachers as well as 193 middle school principals and 190 high school principals. Correlational and hierarchical regression analyses identified correlates and predictors of teachers' ratings of effectiveness. Results suggest that use of peer leaders, more frequent evaluations, a higher degree of parent involvement, fewer barriers, greater student interest, and lower tolerance for tobacco use were correlated with higher ratings of program effectiveness. Furthermore, student interest, peer, staff, and community tolerance norms, peer leaders, program evaluation, and parent involvement predicted middle school teachers' perceptions. Parent tolerance, student interest, number of barriers, and more frequent program evaluation predicted high school teachers' perceptions. In addition, middle school teachers who reported a lower number of factors negatively associated with teacher receptivity were more likely to view TUPE programs more favorably than teachers who reported a greater number of these risk factors. This relationship was not as robust among the high school teacher sample. Differences between the middle and high school sample were found in the magnitude and number of significant correlations, the proportion of variance accounted for by predictor variables, and the strength of the relationship between the number of factors negatively associated with teacher receptivity and teachers' perceptions of TUPE effectiveness. These findings highlighted the importance of the timing, program features, and the external environment for enhancing or minimizing teachers' ratings of TUPE program effectiveness. In conclusion, significant increases in TUPE teachers' self-efficacy will occur through the participation of peers, parents, staff, and community leaders in different aspects of TUPE programs. ^

A Study of the Relational Component in an Academic Advisor Professional Development Program

The purpose of the study was to investigate the significance of the relational component of academic advisor training and development in the learning opportunities of the professional development program and the advisors’ evaluation score at Florida International University.

Application of an Education Research Lab (ERL) Model to a Community Sailing Program

This presentation showcases the application of a university-based education research lab (ERL) model to the evaluation of a community sailing program for individuals with disabilities. Presenters conceptualize the ERL model as a mutually beneficial relationship between universities and community education agencies.

Evaluating the Sandcastles Program: A Group Intervention for Children of Divorce

The Sandcastles program has been utilized nationwide as a one-time group intervention to assist children of divorcing parents. For several years Miami-Dade family court services mandated participation in the program for divorcing or separating families. Currently, there is a paucity of research and evaluation to ascertain the efficacy of the program. This symposium will provide details and discussion regarding the planning and process used to establish an evaluation plan to assess the effectiveness of the Sandcastles program for families in MiamiDade County. Any preliminary outcomes available at the time of the symposium will also be shared.

Self-esteem and job satisfaction of adults with mild mental retardation in sheltered workshops and supported employment programs

A sample of 200 adults with mild mental retardation were assessed on overall job satisfaction and self-esteem using the Vocational Program Evaluation Profile and the Coopersmith Self-esteem Inventory. The subjects worked either in a sheltered workshop or in a supported employment setting. Results indicated that there was a significant relationship between self-esteem and job satisfaction for both groups of subjects. In addition, subjects who worked in supported employment reported significantly higher levels of job satisfaction also. There was also an interaction between place of residence and place of employment when looking at self-esteem; those who live in a semi-independent home and work in supported employment reported the highest levels of self-esteem. These results are discussed in terms of the social validity of supported-employment for persons with mild mental retardation. ^

Adult education directors' perceptions of performance indicators for adult education programs in the state of Florida

Current demands for accountability in education emphasize outcome-based program evaluation and tie program funding to individual student performance. As has been the case for elementary and secondary programs, demands for accountability have increased pressure on adult educators to show evidence of the benefits of their programs in order to justify their financial support. In Florida, recent legislation fundamentally changes the delivery of adult education in the state by establishing a performance-based funding system that is based on outcomes related to the retention, completion, and employment of program participants.^ A performance-based funding system requires an evaluation process that stresses outcome indicators over indicators that focus on program context or process. Although the state has adopted indicators of program quality to evaluate its adult education programs, these indicators focus mostly on program processes rather than student outcomes. In addition, the indicators are not specifically tied to workforce development outcomes, a priority to federal and local funding agents.^ Improving the accountability of adult education programs and defining the role of these programs in Florida's Workforce Development System has become a priority to policy makers across the state. Another priority has been to involve adult education practitioners in every step of this process.^ This study was conducted in order to determine what performance indicators, as judged by the directors and supervisors of adult education programs in the state of Florida, are important and feasible in measuring the quality and effectiveness of these programs. The results of the study indicated that, both statewide and by region, the respondents consistently gave the highest ratings on both importance and feasibility to the indicators of Program Context, which reflect the needs, composition, and structure of the programs, and to the indicators of Educational Gain, which reflect learner progress in the attainment of basic skills and competencies. In turn, the respondents gave the lowest ratings on both importance and feasibility to the indicators in the areas of Return on State's Investment, Efficiency, Retention, and Workforce Training. In general, the indicators that received high ratings for importance also received high ratings for feasibility. ^

The effect of an enriched learning community on success and retention in chemistry courses

Since the mid-1990s, the United States has experienced a shortage of scientists and engineers, declining numbers of students choosing these fields as majors, and low student success and retention rates in these disciplines. Learning theorists, educational researchers, and practitioners believe that learning environments can be created so that an improvement in the numbers of students who complete courses successfully could be attained (Astin, 1993; Magolda & Terenzini, n.d.; O'Banion, 1997). Learning communities do this by providing high expectations, academic and social support, feedback during the entire educational process, and involvement with faculty, other students, and the institution (Ketcheson & Levine, 1999). ^ A program evaluation of an existing learning community of science, mathematics, and engineering majors was conducted to determine the extent to which the program met its goals and was effective from faculty and student perspectives. The program provided laptop computers, peer tutors, supplemental instruction with and without computer software, small class size, opportunities for contact with specialists in selected career fields, a resource library, and Peer-Led Team Learning. During the two years the project has existed, success, retention, and next-course continuation rates were higher than in traditional courses. Faculty and student interviews indicated there were many affective accomplishments as well. ^ Success and retention rates for one learning community class ( n = 27) and one traditional class (n = 61) in chemistry were collected and compared using Pearson chi square procedures ( p = .05). No statistically significant difference was found between the two groups. Data from an open-ended student survey about how specific elements of their course experiences contributed to success and persistence were analyzed by coding the responses and comparing the learning community and traditional classes. Substantial differences were found in their perceptions about the lecture, the lab, other supports used for the course, contact with other students, helping them reach their potential, and their recommendation about the course to others. Because of the limitation of small sample size, these differences are reported in descriptive terms. ^

Goma curriculum, a character education paradigm: Composing a text for shaping classroom character culture

The written text, and approaches to reading it, serves well as an analogy for the classroom space as a "text" that teachers are able to compose; and students are able to read, interpret meaning(s) of, and make responses to and about (Rosenblatt, 1988). Researchers point to ways in which the classroom can be conceptualized as a text to be evoked, experienced, and read (Freire & Macedo, 1987; Powell, 2009; Rosenblatt, 1988; Spears-Bunton & Powell, 2009).^ The present study analyzed secondary data including: 10 transcripts of teacher talks and six self-reports retrieved from the program evaluation archives of DOR Foundation. The data described six teachers' classroom experiences subsequent to professional development centered on Goma character education curriculum that was used during a summer youth program located in South Georgia. Goma, an acronym that stands for Goal, Objective, Method, and Attitude, is a character education paradigm derived from The Inclusive Community Building Ellison Model, the theoretical framework used for this study. The Model identifies conflict resolution as one of its five foci (Hunt, Howard, & Rice, 1998). Hunt (2006) conceived Goma as part of a 7-Step unitary process, also named the 7-Step pathway, to demonstrate how conflict resolution is accomplished within a variety of contexts.^ Analysis of the data involved: (a) a priori coding of teacher talks transcripts using the components of the Goma 7-Step pathway as coding categories, (b) emergent coding of teacher talks transcripts for the types of experiences teachers evidenced, and (c) emergent coding of teachers' self-reports for categories of teachers' instructional activities. Results of the study showed positive influence of Goma curriculum on participating teachers and their instructional practices. Teachers were shown to have had cognitive, instructional, emotional, and social experiences that were most evident when they reported changes in their attitudes toward their students, themselves, and their instructional practices. The present study provided implications for classroom teachers wherein all aspects of teachers' instructional practices can be guided by principles of positive character; and can be used to help compose the kinds of "texts" that may likely contribute to a classroom character culture.^

Application of a PERT-Type System and "Crashing" in a Food Service Operation

A pert-type system, a combination of the program evaluation and review technique (PERT) and the critical path method (CPM), might be used by the hospitality industry to improve planning and control of complex functions. The author discusses this management science technique and how it can assist.

A determination of the effects of a ropes course on the self -esteem of undergraduate and graduate students

The purpose of this study was to evaluate the impact of the ropes course on the self-esteem of undergraduate and graduate students. The ropes course provides challenging experiential activities that facilitate personal confidence and group teamwork. The study relates to adult education, experiential education, and program evaluation within the context of hospitality and tourism management education. Quantitative data were based on the assessment of self-esteem through the completion of the Coopersmith Self-Esteem Inventory (SEI-C) modified for this study. Qualitative data were based on assessment of participants' reflective papers stating their perceptions of the ropes course. ^ The study compared a treatment group (31 undergraduate and 25 graduate students) which participated in the ropes course, and a control group (31 undergraduate and 25 graduate students) which did not participate. Both groups completed the pre- and post-treatment SEI-C at the same time intervals. The quantitative data were analyzed using a two-way repeated-measures analysis of variance (ANOVA). The qualitative data, comprised of reflective papers voluntarily written by 44 (79%) of the treatment group, were coded using four major themes reflecting the students' perceptions about the ropes course. ^ Scores on the pretest and posttest of the SEI-C were not significantly different for the two groups. The qualitative data showed a favorable impact of the ropes course. The discrepancy in the outcomes based on these two measures suggests that the SEI-C self-report paper-and-pencil instrument may not be sufficiently refined to evaluate the complex issue of self-esteem. The SEI-C, if used, should be supplemented by other evaluation measures, since the two measures may be evaluating different components of self-esteem. They may be also differentially affected by bias in scoring, or by statistical characteristics of reliability and validity. ^

A model of self -transformative identity development in troubled adolescent youth

Within the Stage II program evaluation of the Miami Youth Development Project's (YDP) Changing Lives Program (CLP), this study evaluated CLP intervention effectiveness in promoting positive change in emotion-focused identity exploration (i.e. feelings of personal expressiveness; PE) and a "negative" symptom of identity development (i.e. identity distress; ID) as a first step toward the investigation of a self-transformative model of identity development in adolescent youth. Using structural equation modeling techniques, this study found that participation in the CLP is associated with positive changes in PE (path = .841, p < .002), but not changes in ID. Increase in ID scores was found to be associated with increases in PE (path = .229, p < .002), as well. Intervention effects were not moderated by age/stage, gender, or ethnicity, though differences were found in the degree to which participating subgroups (African-American/Hispanic, male/female, 14-16 years old/17-19 years old) experience change in PE and ID. Findings also suggest that moderate levels of ID may not be deleterious to identity exploration and may be associated with active exploration. ^

A model of self-transformative identity development in troubled adolescent youth

Within the Stage II program evaluation of the Miami Youth Development Project's (YDP) Changing Lives Program (CLP), this study evaluated CLP intervention effectiveness in promoting positive change in emotion-focused identity exploration (i.e. feelings of personal expressiveness; PE) and a "negative" symptom of identity development (i.e. identity distress; ID) as a first step toward the investigation of a self-transformative model of identity development in adolescent youth. Using structural equation modeling techniques, this study found that participation in the CLP is associated with positive changes in PE (path = .841, p < .002), but not changes in ID. Increase in ID scores was found to be associated with increases in PE (path = .229, p < .002), as well. Intervention effects were not moderated by age/stage, gender, or ethnicity, though differences were found in the degree to which participating subgroups (African- American/Hispanic, male/female, 14-16 years old/17-19 years old) experience change in PE and ID. Findings also suggest that moderate levels of ID may not be deleterious to identity exploration and may be associated with active exploration.

Goma Curriculum, A Character Education Paradigm: Composing a Text for Shaping Classroom Character Culture

The written text, and approaches to reading it, serves well as an analogy for the classroom space as a “text” that teachers are able to compose; and students are able to read, interpret meaning(s) of, and make responses to and about (Rosenblatt, 1988). Researchers point to ways in which the classroom can be conceptualized as a text to be evoked, experienced, and read (Freire & Macedo, 1987; Powell, 2009; Rosenblatt, 1988; Spears-Bunton & Powell, 2009). The present study analyzed secondary data including: 10 transcripts of teacher talks and six self-reports retrieved from the program evaluation archives of DOR Foundation. The data described six teachers’ classroom experiences subsequent to professional development centered on Goma character education curriculum that was used during a summer youth program located in South Georgia. Goma, an acronym that stands for Goal, Objective, Method, and Attitude, is a character education paradigm derived from The Inclusive Community Building Ellison Model, the theoretical framework used for this study. The Model identifies conflict resolution as one of its five foci (Hunt, Howard, & Rice, 1998). Hunt (2006) conceived Goma as part of a 7-Step unitary process, also named the 7-Step pathway, to demonstrate how conflict resolution is accomplished within a variety of contexts. Analysis of the data involved: (a) a priori coding of teacher talks transcripts using the components of the Goma 7-Step pathway as coding categories, (b) emergent coding of teacher talks transcripts for the types of experiences teachers evidenced, and (c) emergent coding of teachers’ self-reports for categories of teachers’ instructional activities. Results of the study showed positive influence of Goma curriculum on participating teachers and their instructional practices. Teachers were shown to have had cognitive, instructional, emotional, and social experiences that were most evident when they reported changes in their attitudes toward their students, themselves, and their instructional practices. The present study provided implications for classroom teachers wherein all aspects of teachers’ instructional practices can be guided by principles of positive character; and can be used to help compose the kinds of “texts” that may likely contribute to a classroom character culture.

An evaluation of an intervention strategy for retention and determination of predictors of success for high -risk students in an associate degree nursing program

The purpose of this study was twofold: (1) to evaluate the effect of a specific instructional Intervention, a Nursing Theory Laboratory, on increasing the retention of high risk students in the associate degree nursing program at Miami-Dade Community College in Miami, Florida; and (2) to identify predictors of success of high risk nursing students in this associate degree nursing program.^ Data were collected from the 195 nursing students enrolled in Nursing Fundamentals during the 1985-1987 academic years, and identified as high risk students. Control and experimental groups were selected based on enrollment in the Nursing Theory Laboratory.^ Results were determined by analyzing several cross-tabulations of selected variables and yielding chi square values, t-tests, and two discriminant analyses. There was no significant relationship between age or ethnic background and enrollment in the Nursing Theory Laboratory. There was no significant relationship between enrollment in the Nursing Theory Laboratory and success in Nursing 1 (Nursing Fundamentals). There was a significant relationship between enrollment in the Nursing Theory Laboratory and success in Nursing 3 (Medical-Surgical Nursing). Writing assessment test scores in two entrance tests and high risk categories, based on the number of enrollments in required science courses, were identified as predictors of success in this program.^ The conclusion was that the Nursing Theory Laboratory does not significantly improve retention of high risk associate degree nursing students if they are enrolled in this intervention at the same time they are enrolled in Nursing Fundamentals. Since those students who were enrolled in the Nursing Theory Laboratory had a significantly higher success rate in Nursing 3, than those students who were not enrolled in the Nursing Theory Laboratory, a recommendation of this study was to offer the Nursing Theory Laboratory to high risk students prior to the beginning of nursing courses. Another recommendation was that students deficient in reading and writing skills should be required to enroll in developmental courses prior to enrollment in the nursing course. ^