Program evaluation primer: A review of three evaluations

Program evaluation—a “tool used to assess the implementation and outcomes of a program, to increase a program’s efficiency and impact over time, and to demonstrate accountability” (MacDonald et. al, 2001, p. 1)—is an essential process to program assessment and improvement. This paper overviews three published program evaluations and considers important aspects of program evaluation more broadly.

Evaluation of a program to reduce back pain in nursing personnel

OBJECTIVE: To evaluate the effectiveness of a program designed to reduce back pain in nursing aides. METHODS: Female nursing aides from a university hospital who had suffered episodes of back pain for at least six months were included in the study. Participants were randomly divided into a control group and an intervention group. The intervention program involved a set of exercises and an educational component stressing the ergonomic aspect, administered twice a week during working hours for four months. All subjects answered a structured questionnaire and the intensity of pain was assessed before and after the program using a visual analogue scale (VAS). Student's t-test or the Wilcoxon Rank Sum Test for independent samples, and Chi-square test or the Exact Fisher test for categorical analysis, were used. The McNemar test and the Wilcoxon matched pairs test were used to compare the periods before and after the program. RESULTS: There was a statistically significant decrease in the frequency of cervical pain in the last two months and in the last seven days in the intervention group. There was also a reduction in cervical pain intensity in the two periods (2 months, 7 days) and lumbar pain intensity in the last 7 days. CONCLUSIONS: The results suggest that a program of regular exercise with an emphasis on ergonomics can reduce musculoskeletal symptoms in nursing personnel.

Evaluation of the Chagas Disease Control Program in Açucena Municipality, Rio Doce Valley, State of Minas Gerais, Brazil

Introduction Açucena Municipality, Rio Doce Valley, State of Minas Gerais, Brazil temporarily (2001-2005) interrupted epidemiological surveillance for Chagas disease. The objective of this work was to evaluate the Chagas Disease Control Program (CDCP) in Açucena and to offer suggestions for improving local epidemiological surveillance. Methods This study was conducted in three phases: I) a serological investigation of schoolchildren aged 5 to 15 years using an enzyme-linked immunosorbent assay (ELISA) test performed on blood collected on filter paper followed by ELISA, indirect immunofluorescence (IIF) and indirect hemaglutination (IHA) on venous blood for borderline cases and those in the gray zone of reactivity; II) vector evaluation using the data obtained by local health agents during 2006-2010; and III) examination by ELISA, IIF and IHA of serum samples from the inhabitants of houses where infected Triatoma vitticeps was found and evaluation of their knowledge about Chagas disease. Results Five individuals had inconclusive results in the ELISA screening but were seronegative for Chagas disease. The triatomine evaluation revealed the presence of three species: Triatoma vitticeps, Panstrongylus megistus and Panstrongylus diasi. Triatoma vitticeps was the most prevalent and widespread, with a higher (67%) index of Trypanosoma cruzi flagellates and evidence of colonization. Most of the inhabitants of the infested houses recognized triatomines and had basic knowledge about Chagas disease. Conclusions Although T. vitticeps is not clearly associated with Chagas disease transmission, these results highlight the importance of maintaining CDCP in endemic areas and the need for greater emphasis on epidemiological surveillance, especially in areas with important vectorial changes or that have been modified by human intervention.

Development and evaluation of a shaken baby syndrome prevention program.

OBJECTIVE: To evaluate parents' and nurses' opinions regarding the adequacy of an educational program on shaken baby syndrome: the Perinatal Shaken Baby Syndrome Prevention Program (PSBSPP). DESIGN: Qualitative and quantitative assessments in the form of interviews and questionnaires administered in French. SETTING: Two birthing institutions in Montréal, QC, Canada: a university hospital and a regional center. PARTICIPANTS: Two hundred and sixty-three parents (73.8% mothers, 26.2% fathers) received the intervention after the birth of their child, and 69 nurses administered it. METHODS: Parents' and nurses' assessments of the adequacy and relevance of the program and nurses' assessments of the training they received to administer the program were evaluated. RESULTS: Both parents and nurses supported this initiative. Most parents appreciated the usefulness of the information. Nurses believed the program was adequate, and their training to deliver the program was satisfactory. All participants reported that the program was highly relevant, especially for new parents. CONCLUSION: The Perinatal Shaken Baby Syndrome Prevention Program achieves the goals of (a) increasing parents' knowledge about infant crying, anger, and shaken baby syndrome and (b) helping parents identify coping strategies. The relevance of introducing the PSBSPP in all birthing institutions is supported. Future studies should focus on vulnerable and culturally diverse populations, and longitudinal follow-up could help determine if the PSBSPP reduces the incidence of shaken baby syndrome.

Evaluation of a multicomponent worksite health promotion program for cardiovascular risk factors-correcting for the regression towards the mean effect.

BACKGROUND: We assessed the impact of a multicomponent worksite health promotion program for0 reducing cardiovascular risk factors (CVRF) with short intervention, adjusting for regression towards the mean (RTM) affecting such nonexperimental study without control group. METHODS: A cohort of 4,198 workers (aged 42 +/- 10 years, range 16-76 years, 27% women) were analyzed at 3.7-year interval and stratified by each CVRF risk category (low/medium/high blood pressure [BP], total cholesterol [TC], body mass index [BMI], and smoking) with RTM and secular trend adjustments. Intervention consisted of 15 min CVRF screening and individualized counseling by health professionals to medium- and high-risk individuals, with eventual physician referral. RESULTS: High-risk groups participants improved diastolic BP (-3.4 mm Hg [95%CI: -5.1, -1.7]) in 190 hypertensive patients, TC (-0.58 mmol/l [-0.71, -0.44]) in 693 hypercholesterolemic patients, and smoking (-3.1 cig/day [-3.9, -2.3]) in 808 smokers, while systolic BP changes reflected RTM. Low-risk individuals without counseling deteriorated TC and BMI. Body weight increased uniformly in all risk groups (+0.35 kg/year). CONCLUSIONS: In real-world conditions, short intervention program participants in high-risk groups for diastolic BP, TC, and smoking improved their CVRF, whereas low-risk TC and BMI groups deteriorated. Future programs may include specific advises to low-risk groups to maintain a favorable CVRF profile.

The development, assessment, and implementation of an evaluation of the EMPOWER Program at shelters for abused women /

The development, assessment, and implementation of a program evaluation instrument was carried out to evaluate the impact and efficacy of the EMPOWER Program. This intervention was created to educate residents at a shelter for abused women with an anticipated outcome of prevention. Participants included the staff and residents at 2 shelters in Southern Ontario. Client pre, post and follow-up measures were obtained and analyzed statistically and using keyword content analysis. A single staff measure was obtained and summarized using keyword content analysis. Qualitative results were suggestive of important change in participants. All women in the post and follow-up measures believed their participation in the EMPOWER Program provided them with the knowledge, skills, and confidence to avoid abusive relationships in the fliture. This transformational impact was repeatedly expressed in both resident and staff feedback. Limitations of this research, as well as suggestions for future study were discussed.

HIDING IN PLAIN SIGHT- Sustainable Physical Activity Program Development and Evaluation for Youth with Special Needs: An Evaluative Case Study

The purpose of my research was to contribute to the improvement and sustainability of the Special Needs Activity Program, and develop program implementation strategies that had practical outcomes. I conducted an evaluative case study of S.N.A.P in order to determine what a quality adapted physical activity (APA) program is, why S.N.A.P is considered a quality APA program, and what institutional policies and practices exist to support it. Data was collected via interviews, questionnaires, and observations. Data analysis involved inductive and deductive methods, and a SWOTAR evaluation. Results indicate that quality APA programs include: ‘people’, ‘environment’, and ‘expectations’; there are benefits of experiential learning; activity stations that promote creativity are valuable; several stakeholders do not know the details about S.N.A.P but recognize its value; the institution values what S.N.A.P provides, yet, there is nothing being done to sustain it. Future research should investigate the feasibility of implementing S.N.A.P in various contexts.

Evaluation of the March of Dimes Family Support Program with Focus on Parent to Parent Support

The March of Dimes NICU Family Support Program provides families with support specialists, educational materials, and community resources to aide in the emotional and physical adaptation to a new life with a premature infant. Parent-to-parent support has been shown to more effective than group support because the new NICU parent is able to connect on a more personal level with an experienced NICU parent. The purpose of the research was to develop and implement an evaluation instrument to assess the effectiveness of the March of Dimes NICU Family Support Program (MODFSP) in the NICU at the UConn Health Center, specifically parent-to-parent support. Steps involved in the process included determining areas of focus for the instrument, developing items based on MODFSP materials and literature review, piloting the materials on parents with infants in the NICU (with IRB approval), and utilizing descriptive statistics through SPSS Version 14 to assess the results from the instrument content. Qualitative items were also included in the evaluation, and descriptive qualitative methods, as appropriate, were used to analyze those items. The findings have supported the literature in that all parents that participated in parent to parent support agreed the program was beneficial in assisting them with their transition to life with a premature infant. In addition to evaluating the new NICU parent’s opinions of the program, researchers evaluated the staff and volunteer parents that were involved in the program. The results also revealed that the new NICU parent’s fear, anxiety, and stress decreased after the parent-to-parent interactions. The preliminary results were encouraging that the MODFSP has implemented an effective parent-to-parent support program to support parents through their time of crisis.

The School Sealant Program: An evaluation of the oral health education component

Dental caries is a common preventable childhood disease leading to severe physical, mental and economic repercussions for children and their families if left untreated. A needs assessment in Harris County reported that 45.9% of second graders had untreated dental caries. In order to address this growing problem, the School Sealant Program (SSP), a primary preventive initiative, was launched by the Houston Department of Health and Human Services (HDHHS) to provide oral health education, and underutilized dental preventive services to second grade children from participating Local School Districts (LSDs). ^ To determine the effectiveness and efficiency of the SSP, a program evaluation was conducted by the HDHHS between September 2007 and June 2008 for the Oral Health Education (OHE) component of the SSP. The objective of the evaluation was to assess short term changes in oral health knowledge of the participants and determine if these changes, if any, were due to the OHE sessions. An 8-item multiple choice pre/post test was developed for this purpose and administered to the participants before and immediately after the OHE sessions. ^ The present project analyzed pre and post test data of 1,088 second graders from 22 participating schools. Changes in overall and topic-specific knowledge of the program participants before and after the OHE sessions were analyzed using the Wilcoxon's signed rank test. ^ Results. The overall knowledge assessment showed a statistically significant (p <0.001) increase in the dental health knowledge of the participants after the oral health education sessions. Participants in the higher scoring category (7-8 correct responses) increased from 9.5% at baseline to 60.8% after the education sessions. Overall knowledge increased in all school regions with the highest knowledge gains seen in the Central and South regions. Males and females had similar knowledge gains. Significant knowledge differences were also found for each of the topic specific categories (functions of teeth, healthy diet, healthy habits, dental sealants; p<0.001) indicating an increase in topic specific knowledge of the participants post-health education sessions. ^ Conclusions. The OHE sessions were successful in increasing the short term oral health knowledge of the participants. ^

Use of empowerment evaluation to determine a sustainability plan for a community-based preconception care program

Women With IMPACT (WWI) is a community-based preconception care educational intervention. WWI is being implemented by the Impacting Maternal and Prenatal Care Together (IMPACT) Collaborative and targets zip codes in Harris County, Texas at high risk for infant mortality, low birthweight, and preterm birth. WWI started March 2012 and continues through August 2013. Three workshop series are planned. This study was conducted with participants and facilitators from the first workshop series. This study aimed to 1) evaluate the WWI program using empowerment evaluation, 2) engage all WWI stakeholders in an empowerment evaluation so the method could be adopted as a participatory evaluation process for future IMPACT activities, and 3) develop recommendations for sustainability of the WWI intervention, based on empowerment evaluation findings and results from the pre/post program evaluation completed by WWI participants. Study participants included WWI participants and facilitators and IMPACT Collaborative Steering Committee members. WWI participants were female, 18-35 year-old, non-pregnant residents of zip codes at high risk of adverse birth outcomes. All other study participants were 18 years or older. A two-phased empowerment evaluation (EE) was utilized in this study. Sessions 1-4 were conducted independently of one another – 3 with participants at different sites and one with the facilitators. The fifth session included WWI participant and facilitator representatives, and IMPACT Steering Committee members. Session 5 built upon the work of the other sessions. Observation notes were recorded during each session. Thematic content analysis was conducted on all EE tables and observation notes. Mission statements drafted by each group focused on improvement of physical and mental health through behavior change and empowerment of all participants. The top 5 overall program components were: physical activity, nutrition, self-worth, in-class communication, and stress. Goals for program improvement were set by EE participants for each of these components. Through thematic content analysis of the tables and observation notes, social support emerged as an important theme of the program among all participant groups. Change to a healthy lifestyle emerged as an important theme in terms of program improvement. Two-phased EE provided an opportunity for all program stakeholders to provide feedback regarding important program components and provide suggestions for program improvement. EE, thematic content analysis, pre/post evaluation results, and inherent program knowledge were triangulated to make recommendations to sustain the program once the initial funding ends. ^

Evaluation of the economic impact of the Job Corps program : special report : an examination of Job Corps participation /

"PB 288167."

Evaluation of the economic impact of the Job Corps program : first follow-up report /

"PB 292410."

Evaluation of the Concentrated Employment Program in six selected areas : final report /

"Prepared under a contract with the Office of Manpower Policy, Evaluation, and Research, U. S. Dept. of Labor."

Evaluation use within a humanitarian non-governmental organization’s health care user-fee exemption program in West Africa

L’évaluation de l’action humanitaire (ÉAH) est un outil valorisé pour soutenir l’imputabilité, la transparence et l’efficience de programmes humanitaires contribuant à diminuer les inéquités et à promouvoir la santé mondiale. L’EAH est incontournable pour les parties prenantes de programme, les bailleurs de fonds, décideurs et intervenants souhaitant intégrer les données probantes aux pratiques et à la prise de décisions. Cependant, l’utilisation de l’évaluation (UÉ) reste incertaine, l’ÉAH étant fréquemment menée, mais inutilisé. Aussi, les conditions influençant l’UÉ varient selon les contextes et leur présence et applicabilité au sein d’organisations non-gouvernementales (ONG) humanitaires restent peu documentées. Les évaluateurs, parties prenantes et décideurs en contexte humanitaire souhaitant assurer l’UÉ pérenne détiennent peu de repères puisque rares sont les études examinant l’UÉ et ses conditions à long terme. La présente thèse tend à clarifier ces enjeux en documentant sur une période de deux ans l’UÉ et les conditions qui la détermine, au sein d’une stratégie d’évaluation intégrée au programme d’exemption de paiement des soins de santé d’une ONG humanitaire. L’objectif de ce programme est de faciliter l’accès à la santé aux mères, aux enfants de moins de cinq ans et aux indigents de districts sanitaires au Niger et au Burkina Faso, régions du Sahel où des crises alimentaires et économiques ont engendré des taux élevés de malnutrition, de morbidité et de mortalité. Une première évaluation du programme d’exemption au Niger a mené au développement de la stratégie d’évaluation intégrée à ce même programme au Burkina Faso. La thèse se compose de trois articles. Le premier présente une étude d’évaluabilité, étape préliminaire à la thèse et permettant de juger de sa faisabilité. Les résultats démontrent une logique cohérente et plausible de la stratégie d’évaluation, l’accessibilité de données et l’utilité d’étudier l’UÉ par l’ONG. Le second article documente l’UÉ des parties prenantes de la stratégie et comment celle-ci servit le programme d’exemption. L’utilisation des résultats fut instrumentale, conceptuelle et persuasive, alors que l’utilisation des processus ne fut qu’instrumentale et conceptuelle. Le troisième article documente les conditions qui, selon les parties prenantes, ont progressivement influencé l’UÉ. L’attitude des utilisateurs, les relations et communications interpersonnelles et l’habileté des évaluateurs à mener et à partager les connaissances adaptées aux besoins des utilisateurs furent les conditions clés liées à l’UÉ. La thèse contribue à l’avancement des connaissances sur l’UÉ en milieu humanitaire et apporte des recommandations aux parties prenantes de l’ONG.

Evaluation use within a humanitarian non-governmental organization’s health care user-fee exemption program in West Africa

L’évaluation de l’action humanitaire (ÉAH) est un outil valorisé pour soutenir l’imputabilité, la transparence et l’efficience de programmes humanitaires contribuant à diminuer les inéquités et à promouvoir la santé mondiale. L’EAH est incontournable pour les parties prenantes de programme, les bailleurs de fonds, décideurs et intervenants souhaitant intégrer les données probantes aux pratiques et à la prise de décisions. Cependant, l’utilisation de l’évaluation (UÉ) reste incertaine, l’ÉAH étant fréquemment menée, mais inutilisé. Aussi, les conditions influençant l’UÉ varient selon les contextes et leur présence et applicabilité au sein d’organisations non-gouvernementales (ONG) humanitaires restent peu documentées. Les évaluateurs, parties prenantes et décideurs en contexte humanitaire souhaitant assurer l’UÉ pérenne détiennent peu de repères puisque rares sont les études examinant l’UÉ et ses conditions à long terme. La présente thèse tend à clarifier ces enjeux en documentant sur une période de deux ans l’UÉ et les conditions qui la détermine, au sein d’une stratégie d’évaluation intégrée au programme d’exemption de paiement des soins de santé d’une ONG humanitaire. L’objectif de ce programme est de faciliter l’accès à la santé aux mères, aux enfants de moins de cinq ans et aux indigents de districts sanitaires au Niger et au Burkina Faso, régions du Sahel où des crises alimentaires et économiques ont engendré des taux élevés de malnutrition, de morbidité et de mortalité. Une première évaluation du programme d’exemption au Niger a mené au développement de la stratégie d’évaluation intégrée à ce même programme au Burkina Faso. La thèse se compose de trois articles. Le premier présente une étude d’évaluabilité, étape préliminaire à la thèse et permettant de juger de sa faisabilité. Les résultats démontrent une logique cohérente et plausible de la stratégie d’évaluation, l’accessibilité de données et l’utilité d’étudier l’UÉ par l’ONG. Le second article documente l’UÉ des parties prenantes de la stratégie et comment celle-ci servit le programme d’exemption. L’utilisation des résultats fut instrumentale, conceptuelle et persuasive, alors que l’utilisation des processus ne fut qu’instrumentale et conceptuelle. Le troisième article documente les conditions qui, selon les parties prenantes, ont progressivement influencé l’UÉ. L’attitude des utilisateurs, les relations et communications interpersonnelles et l’habileté des évaluateurs à mener et à partager les connaissances adaptées aux besoins des utilisateurs furent les conditions clés liées à l’UÉ. La thèse contribue à l’avancement des connaissances sur l’UÉ en milieu humanitaire et apporte des recommandations aux parties prenantes de l’ONG.