Addressing Multilevel Program Complexity by Evaluation Design

In this paper, we present the evaluation design for a complex multilevel program recently introduced in Switzerland. The evaluation embraces the federal level, the cantonal program level, and the project level where target groups are directly addressed. We employ Pawson and Tilley’s realist evaluation approach, in order to do justice to the varying context factors that impact the cantonal programs leading to varying effectiveness of the implemented activities. The application of the model to the canton of Uri shows that the numerous vertical and horizontal relations play a crucial role for the program’s effectiveness. As a general learning for the evaluation of complex programs, we state that there is a need to consider all affected levels of a program and that no monocausal effects can be singled out in programs where multiple interventions address the same problem. Moreover, considering all affected levels of a program can mean going beyond the borders of the actual program organization and including factors that do not directly interfere with the policy delivery as such. In particular, we found that the relationship between the cantonal and the federal level was a crucial organizational factor influencing the effectiveness of the cantonal program.

Evaluation of Medical Students’ Attitudes and Performance of Basic Surgery Skills in a Training Program Using Fresh Human skin, Excised During Body Contouring Surgeries

BACKGROUND: Learning surgical skills in the operating room may be a challenge for medical students. Therefore, more approaches using simulation to enable students to develop their practical skills are required. OBJECTIVES: We hypothesized that (1) there would be a need for additional surgical training for medical students in the pre-final year, and (2) our basic surgery skills training program using fresh human skin would improve medical students' surgical skills. DESIGN: We conducted a preliminary survey of medical students to clarify the need for further training in basic surgery procedures. A new approach using simulation to teach surgical skills on human skin was set up. The procedural skills of 15 randomly selected students were assessed in the operating room before and after participation in the simulation, using Objective Structured Assessment of Technical Skills. Furthermore, subjective assessment was performed based on students' self-evaluation. The data were analyzed using SPSS, version 21 (SPSS, Inc., Chicago, IL). SETTING: The study took place at the Inselspital, Bern University Hospital. PARTICIPANTS: A total of 186 pre-final-year medical students were enrolled into the preliminary survey; 15 randomly selected medical students participated in the basic surgical skills training course on the fresh human skin operating room. RESULTS: The preliminary survey revealed the need for a surgical skills curriculum. The simulation approach we developed showed significant (p < 0.001) improvement for all 12 surgical skills, with mean cumulative precourse and postcourse values of 31.25 ± 5.013 and 45.38 ± 3.557, respectively. The self-evaluation contained positive feedback as well. CONCLUSION: Simulation of surgery using human tissue samples could help medical students become more proficient in handling surgical instruments before stepping into a real surgical situation. We suggest further studies evaluating our proposed teaching method and the possibility of integrating this simulation approach into the medical school curriculum.

Evaluation of the March of Dimes Family Support Program with Focus on Parent to Parent Support

The March of Dimes NICU Family Support Program provides families with support specialists, educational materials, and community resources to aide in the emotional and physical adaptation to a new life with a premature infant. Parent-to-parent support has been shown to more effective than group support because the new NICU parent is able to connect on a more personal level with an experienced NICU parent. The purpose of the research was to develop and implement an evaluation instrument to assess the effectiveness of the March of Dimes NICU Family Support Program (MODFSP) in the NICU at the UConn Health Center, specifically parent-to-parent support. Steps involved in the process included determining areas of focus for the instrument, developing items based on MODFSP materials and literature review, piloting the materials on parents with infants in the NICU (with IRB approval), and utilizing descriptive statistics through SPSS Version 14 to assess the results from the instrument content. Qualitative items were also included in the evaluation, and descriptive qualitative methods, as appropriate, were used to analyze those items. The findings have supported the literature in that all parents that participated in parent to parent support agreed the program was beneficial in assisting them with their transition to life with a premature infant. In addition to evaluating the new NICU parent’s opinions of the program, researchers evaluated the staff and volunteer parents that were involved in the program. The results also revealed that the new NICU parent’s fear, anxiety, and stress decreased after the parent-to-parent interactions. The preliminary results were encouraging that the MODFSP has implemented an effective parent-to-parent support program to support parents through their time of crisis.

Na 1 Nachlass Max Horkheimer, 659 - Berichte über die Arbeit des Instituts für Sozialforschung u.a. (p. IX 64-IX 65b)

"The International Institute of Social Research. Its History, Program and Needs" (1943?), als Typoskript vervielfältigt, 18 Blatt, mit 6 Photographien und einem Anhang: Rezensionen von "Autorität und Familie" und "Abstracts" von Aufsätzen aus der Zeitschrift für Sozialforschung, Photokopien, 12 Blatt; "Über Geschichte, Organisation und Tätigkeiten des Instituts für Sozialforschung. Aus einem Bericht" (1943), Teilstück, Typoskript mit handschriftlichen Korrekturen, 10 Blatt; "Annual Report on the Activities of the Social Studies Association, Inc." (6.5.1944), Typoskript, 6 Blatt; "Ten Years on Morningside Hights. A Report on the Institute's History 1934 to 1944 (Dezember 1944): 1. Report, a) als Typoskript vervielfältigt, 36 Blatt, b) Typoskript, 48 Blatt, c) Entwurf des Teils "Philosophical Studies", Typoskript, 4 Blatt; 2. "Letters Regarding Ten Years Report", 1 Blatt; 3. Liste der Empfänger des Reports. 1 Blatt; 4. Max Horkheimer: 1 Brief an Nicholas Murray Butler, New York, 17.4.1945 , 1 Blatt; 5. Briefentwürfe zur Versendung des Reports, 2 Blatt; Theodore Abel: "Evaluation of the Institute's Work" (1944?): 1. Würdigung, Typoskript, englisch, 4 Blatt; 2. Theodore Abel: 1 Brief mit Unterschrift an Leo Löwenthal, New York, 6.3.1946; 3. Abschrift von 1. und 2., Typoskript, 5 Blatt; Über Organisation und Tätigkeiten des Instituts für Sozialforschung (Mai 1945), Typoskript, 3 Blatt;

An evaluation of the impact of Inner City Development's Cooperative Home School: An educational alternative program to increase the quality and level of education in San Antonio's West Side community

The research study was intended to evaluate the effectiveness of Inner City Development's (I.C.D.) Cooperative Home School, an educational alternative program to the Title I public schools of San Antonio's West Side community. The study investigated students', parents' and tutors' perception of parental involvement and educational resources. The study also investigated each student's academic achievement. ^ The study found that students progressed toward expected math proficiency at a faster rate than they did in reading proficiency. However, because the target population size was small and a comparison group was not used, the results of this study are only suggestive. This research also indicated that study subjects believed students' quality and level of education increased substantially since program exposure. Study subjects mainly attributed the students' strides in academic performance to the increased amount of individualized attention students received in the small twelve-student class size. Study subjects were more satisfied with the home school's educational resources than those of the Title I public schools. Study subjects also perceived that parental involvement both at home and at school increased since enrollment in the home school program because: (1) there were more opportunities for involvement in the home school; and (2) parents felt closer to the tutors than the teachers in public school. ^ This evaluation also suggested improvements to program operations. With the help of additional volunteers, I.C.D. program operators could improve collection and organization of academic records. Furthermore, as suggested by program participants, science could be added to the curriculum. Lastly, a formal tutor orientation could be implemented to familiarize and train tutors on classroom management procedures. ^

Evaluation of the city of Houston HIV/STD prevention program in Houston Independent School District (HISD) high schools

This cross-sectional study was undertaken to evaluate the impact in terms of HIV/STD knowledge and sexual behavior that the City of Houston HIV/STD prevention program in HISD high schools has had on students who have participated in it by comparing them with their peers who have not, based on self reports. The study further evaluated the program cost-effectiveness for averting future HIV infections by computing Cost-Utility Ratios based on reported sexual behavior. ^ Mixed results were obtained, indicating a statistically significant difference in knowledge with the intervention group having scored higher (p-value 0.001) but not for any of the behaviors assessed. The knowledge score outcome's overall p-value after adjusting for each stratifying variable (age, grade, gender and ethnicity) was statistically significant. The Odds Ratio of intervention group participants aged 15 years or more scoring 70% or higher was 1.86 times; that of intervention group female participants was 2.29 times; and that of intervention group Black/African American participants was 2.47 times relative to their comparison group counterparts. The knowledge score results remained statistically significant in the logistic regression model, which controlled for age, grade level, gender and ethnicity. The Odds Ratio in this case was 1.74. ^ Three scenarios based on the difference in the risk of HIV infection between the intervention and comparison group were used for computation of Cost-Utility Ratios: Base, worst and best-case scenario. The best-case scenario yielded cost-effective results for male participants and cost-saving results for female participants when using ethnicity-adjusted HIV prevalence. The scenario remained cost-effective for female participants when using the unadjusted HIV prevalence. ^ The challenge to the program is to devise approaches that can enhance benefits for male participants. If it is a threshold problem implying that male participants require more intensive programs for behavioral change, then programs should first be piloted among boys before being implemented across the board. If it is a reflection of gender differences, then we might have to go back to the drawing board and engage boys in focus group discussions that will help formulate more effective programs. Gender-blind approaches currently in vogue do not seem to be working. ^

Evaluation of the effectiveness of the Fresh and Healthy Program

Background. Diets high in fat and calories are promoted by the toxic food environment in which high fat, high calorie foods are readily accessible, thus contributing to high rates of overweight and obesity.^ Hypothesis. Changing the food environment to make low-fat, low-calorie foods readily identifiable and accessible while simultaneously offering incentives for choosing those foods will result in increased consumption of targeted foods, thus decreasing caloric and fat intake and ultimately decreasing obesity rates.^ Objective. To conduct an outcome evaluation study on the effectiveness of The Fresh & Healthy Program, a health promotion project designed to promote healthy eating among The Methodist Hospital employees by labeling and promoting low calorie, low fat items in the hospital cafeteria. ^ Program. By promoting healthy eating, this program seeks to address unhealthy dietary behaviors, one of the most widely known and influential behavioral causes of obesity. Food items that are included in the program meet nutritional criteria for calories and fat and are labeled with a special logo. Program participants receive incentives for purchasing Fresh & Healthy items. The program was designed and implemented by a team of registered dietitians, two health education specialists, and retail foodservice managers at The Methodist Hospital in the Texas Medical Center in Houston and has been in existence since April 2006.^ Methods. The evaluation uses a non-randomized, one-group, time series design to evaluate the effect of the program on sales of targeted food items.^ Key words. point-of-purchase, menu labeling, environmental obesity interventions, food pricing interventions ^

A program manual to assist with implementation of a volunteer health and wellness program

Based on the success of a community health and wellness program, Wellness in the City, delivered in Dallas County by trained volunteers, Texas Cooperative Extension (TCE) decided to pilot the program in 16 counties in North and East Texas. Before implementing the program, TCE wanted the Dallas County program to be reviewed and revised as needed to meet the diverse community needs in the pilot counties. TCE also asked for an implementation manual to be developed for the county agents to utilize. ^ To achieve these objectives, I first reviewed literature on other volunteer-implemented health interventions in the U.S. to see how they were planned, disseminated, implemented, and evaluated. Next, I reviewed the Wellness in the City program and materials. I applied all the information I gathered up to that point to the program development committee meetings (committee included seven TCE county agents, a TCE regional program director, and me). The program structure and training materials were revised based on our research and program implementation experience. These changes were made to ensure adequate training for the volunteers and to create a program that is applicable in the communities it will be piloted in. ^ With the program structure and training presentations developed, next I focused on compiling the implementation manual, which includes program details and volunteer recruitment, training, and management materials. The goal was to create a manual with everything the county agents will need to implement the program, so they can focus their efforts on putting the manual to use and recruiting and managing the volunteers. The final step was developing a program evaluation form for the agents to complete. It includes questions to assess the agents' thoughts about the training content, the feasibility of implementing the program using the manual, and the challenges of the program. ^

The School Sealant Program: An evaluation of the oral health education component

Dental caries is a common preventable childhood disease leading to severe physical, mental and economic repercussions for children and their families if left untreated. A needs assessment in Harris County reported that 45.9% of second graders had untreated dental caries. In order to address this growing problem, the School Sealant Program (SSP), a primary preventive initiative, was launched by the Houston Department of Health and Human Services (HDHHS) to provide oral health education, and underutilized dental preventive services to second grade children from participating Local School Districts (LSDs). ^ To determine the effectiveness and efficiency of the SSP, a program evaluation was conducted by the HDHHS between September 2007 and June 2008 for the Oral Health Education (OHE) component of the SSP. The objective of the evaluation was to assess short term changes in oral health knowledge of the participants and determine if these changes, if any, were due to the OHE sessions. An 8-item multiple choice pre/post test was developed for this purpose and administered to the participants before and immediately after the OHE sessions. ^ The present project analyzed pre and post test data of 1,088 second graders from 22 participating schools. Changes in overall and topic-specific knowledge of the program participants before and after the OHE sessions were analyzed using the Wilcoxon's signed rank test. ^ Results. The overall knowledge assessment showed a statistically significant (p <0.001) increase in the dental health knowledge of the participants after the oral health education sessions. Participants in the higher scoring category (7-8 correct responses) increased from 9.5% at baseline to 60.8% after the education sessions. Overall knowledge increased in all school regions with the highest knowledge gains seen in the Central and South regions. Males and females had similar knowledge gains. Significant knowledge differences were also found for each of the topic specific categories (functions of teeth, healthy diet, healthy habits, dental sealants; p<0.001) indicating an increase in topic specific knowledge of the participants post-health education sessions. ^ Conclusions. The OHE sessions were successful in increasing the short term oral health knowledge of the participants. ^