A Multi- Faceted, Intensive Family Preservation Program Evaluation

This evaluation of a county intensive family preservation services (ifps) program makes several important methodological contributions to assessing post-treatment placement patterns of ifps clients. It is the first published ifps evaluation that utilizes an interval-level, overall measure of restrictiveness of placement, and one of the few that has followed placement patterns for a full two-years after treatment. The study is also a good example of complementing placement data with measures of family health and stability, and with qualitative feedback from former ifps clients. Finally, this study demonstrates the potential for doing methodologically sound evaluations of local ifps programs.

Program evaluation for a summer science internship: Short and long term benefits as reported by interns

Undergraduate research programs have been used as a tool to attract and retain student interest in science careers. This study evaluates the short and long-term benefits of a Summer Science Internship (SSI) at the University of Texas Health Science Center at Houston– School of Public Health – in Brownsville, Texas, by analyzing survey data from alumni. Questions assessing short-term program impact were aimed at three main topics, student: satisfaction with program, self-efficacy for science after completing the program, and perceived benefits. Long-term program impact was assessed by looking at student school attendance and college majors along with perceived links between SSI and future college plans. Students reported high program satisfaction, a significant increase in science self-efficacy and high perceived benefits. At the time data were collected for the study, one-hundred percent of alumni were enrolled in school (high school or college). The majority of students indicated they were interested in completing a science major/career, heavily influenced by their participation in the program.^

The Colic Clinic---a comprehensive program evaluation

Child abuse correlated with excessive infant crying affects millions of families each year, with consequences of the abuse lasting a lifetime. The University Of Texas School Of Medicine's Colic Clinic is currently in the early stages of testing Dr. Harvey Karp's combinatorial soothing technique for infants called "The Happiest Baby on the Block". In order to gauge the program's potential effectiveness, the Colic Clinic Protocol was examined in order to assess the applicability of the intervention to known causal factors of child abuse associated with excessive infant crying. ^ This evaluation also carried out an anticipated cost-benefit breakout analysis for the implementation of the intervention for 100 children and compared the cost of the program implementation to the cost associated with a single instance of child abuse. The analysis revealed that while accounting for materials, advertising, salaried personnel and other overhead expenses, the cost to implement the intervention was less than half the cost of the medical treatment associated with a single victim of whiplash-shaken infant syndrome. ^ Although the program is still in its early evaluative phase, the future implications of this work are extensive. If this intervention is revealed to be relevant and cost effective, it will precipitate sweeping changes in medical education and training, public health detection and prevention programs, and law enforcement.^

Use of empowerment evaluation to determine a sustainability plan for a community-based preconception care program

Women With IMPACT (WWI) is a community-based preconception care educational intervention. WWI is being implemented by the Impacting Maternal and Prenatal Care Together (IMPACT) Collaborative and targets zip codes in Harris County, Texas at high risk for infant mortality, low birthweight, and preterm birth. WWI started March 2012 and continues through August 2013. Three workshop series are planned. This study was conducted with participants and facilitators from the first workshop series. This study aimed to 1) evaluate the WWI program using empowerment evaluation, 2) engage all WWI stakeholders in an empowerment evaluation so the method could be adopted as a participatory evaluation process for future IMPACT activities, and 3) develop recommendations for sustainability of the WWI intervention, based on empowerment evaluation findings and results from the pre/post program evaluation completed by WWI participants. Study participants included WWI participants and facilitators and IMPACT Collaborative Steering Committee members. WWI participants were female, 18-35 year-old, non-pregnant residents of zip codes at high risk of adverse birth outcomes. All other study participants were 18 years or older. A two-phased empowerment evaluation (EE) was utilized in this study. Sessions 1-4 were conducted independently of one another – 3 with participants at different sites and one with the facilitators. The fifth session included WWI participant and facilitator representatives, and IMPACT Steering Committee members. Session 5 built upon the work of the other sessions. Observation notes were recorded during each session. Thematic content analysis was conducted on all EE tables and observation notes. Mission statements drafted by each group focused on improvement of physical and mental health through behavior change and empowerment of all participants. The top 5 overall program components were: physical activity, nutrition, self-worth, in-class communication, and stress. Goals for program improvement were set by EE participants for each of these components. Through thematic content analysis of the tables and observation notes, social support emerged as an important theme of the program among all participant groups. Change to a healthy lifestyle emerged as an important theme in terms of program improvement. Two-phased EE provided an opportunity for all program stakeholders to provide feedback regarding important program components and provide suggestions for program improvement. EE, thematic content analysis, pre/post evaluation results, and inherent program knowledge were triangulated to make recommendations to sustain the program once the initial funding ends. ^

The School Sealant Program: An evaluation of the oral health education component

Dental caries is a common preventable childhood disease leading to severe physical, mental and economic repercussions for children and their families if left untreated. A needs assessment in Harris County reported that 45.9% of second graders had untreated dental caries. In order to address this growing problem, the School Sealant Program (SSP), a primary preventive initiative, was launched by the Houston Department of Health and Human Services (HDHHS) to provide oral health education, and underutilized dental preventive services to second grade children from participating Local School Districts (LSDs). ^ To determine the effectiveness and efficiency of the SSP, a program evaluation was conducted by the HDHHS between September 2007 and June 2008 for the Oral Health Education (OHE) component of the SSP. The objective of the evaluation was to assess short term changes in oral health knowledge of the participants and determine if these changes, if any, were due to the OHE sessions. An 8-item multiple choice pre/post test was developed for this purpose and administered to the participants before and immediately after the OHE sessions. ^ The present project analyzed pre and post test data of 1,088 second graders from 22 participating schools. Changes in overall and topic-specific knowledge of the program participants before and after the OHE sessions were analyzed using the Wilcoxon's signed rank test. ^ Results. The overall knowledge assessment showed a statistically significant (p <0.001) increase in the dental health knowledge of the participants after the oral health education sessions. Participants in the higher scoring category (7-8 correct responses) increased from 9.5% at baseline to 60.8% after the education sessions. Overall knowledge increased in all school regions with the highest knowledge gains seen in the Central and South regions. Males and females had similar knowledge gains. Significant knowledge differences were also found for each of the topic specific categories (functions of teeth, healthy diet, healthy habits, dental sealants; p<0.001) indicating an increase in topic specific knowledge of the participants post-health education sessions. ^ Conclusions. The OHE sessions were successful in increasing the short term oral health knowledge of the participants. ^

A program manual to assist with implementation of a volunteer health and wellness program

Based on the success of a community health and wellness program, Wellness in the City, delivered in Dallas County by trained volunteers, Texas Cooperative Extension (TCE) decided to pilot the program in 16 counties in North and East Texas. Before implementing the program, TCE wanted the Dallas County program to be reviewed and revised as needed to meet the diverse community needs in the pilot counties. TCE also asked for an implementation manual to be developed for the county agents to utilize. ^ To achieve these objectives, I first reviewed literature on other volunteer-implemented health interventions in the U.S. to see how they were planned, disseminated, implemented, and evaluated. Next, I reviewed the Wellness in the City program and materials. I applied all the information I gathered up to that point to the program development committee meetings (committee included seven TCE county agents, a TCE regional program director, and me). The program structure and training materials were revised based on our research and program implementation experience. These changes were made to ensure adequate training for the volunteers and to create a program that is applicable in the communities it will be piloted in. ^ With the program structure and training presentations developed, next I focused on compiling the implementation manual, which includes program details and volunteer recruitment, training, and management materials. The goal was to create a manual with everything the county agents will need to implement the program, so they can focus their efforts on putting the manual to use and recruiting and managing the volunteers. The final step was developing a program evaluation form for the agents to complete. It includes questions to assess the agents' thoughts about the training content, the feasibility of implementing the program using the manual, and the challenges of the program. ^

Critical indicators associated with negative performance in a selectively manned unit

Introduction. Selectively manned units have a long, international history, both military and civilian. Some examples include SWAT teams, firefighters, the FBI, the DEA, the CIA, and military Special Operations. These special duty operators are individuals who perform a highly skilled and dangerous job in a unique environment. A significant amount of money is spent by the Department of Defense (DoD) and other federal agencies to recruit, select, train, equip and support these operators. When a critical incident or significant life event occurs, that jeopardizes an operator's performance; there can be heavy losses in terms of training, time, money, and potentially, lives. In order to limit the number of critical incidents, selection processes have been developed over time to “select out” those individuals most likely to perform below desired performance standards under pressure or stress and to "select in" those with the "right stuff". This study is part of a larger program evaluation to assess markers that identify whether a person will fail under the stresses in a selectively manned unit. The primary question of the study is whether there are indicators in the selection process that signify potential negative performance at a later date. ^ Methods. The population being studied included applicants to a selectively manned DoD organization between 1993 and 2001 as part of a unit assessment and selection process (A&S). Approximately 1900 A&S records were included in the analysis. Over this nine year period, seventy-two individuals were determined to have had a critical incident. A critical incident can come in the form of problems with the law, personal, behavioral or family problems, integrity issues, and skills deficit. Of the seventy-two individuals, fifty-four of these had full assessment data and subsequent supervisor performance ratings which assessed how an individual performed while on the job. This group was compared across a variety of variables including demographics and psychometric testing with a group of 178 individuals who did not have a critical incident and had been determined to be good performers with positive ratings by their supervisors.^ Results. In approximately 2004, an online pre-screen survey was developed in the hopes of preselecting out those individuals with items that would potentially make them ineligible for selection to this organization. This survey has aided the organization to increase its selection rates and save resources in the process. (Patterson, Howard Smith, & Fisher, Unit Assessment and Selection Project, 2008) When the same prescreen was used on the critical incident individuals, it was found that over 60% of the individuals would have been flagged as unacceptable. This would have saved the organization valuable resources and heartache.^ There were some subtle demographic differences between the two groups (i.e. those with critical incidents were almost twice as likely to be divorced compared with the positive performers). Upon comparison of Psychometric testing several items were noted to be different. The two groups were similar when their IQ levels were compared using the Multidimensional Aptitude Battery (MAB). When looking at the Minnesota Multiphasic Personality Inventory (MMPI), there appeared to be a difference on the MMPI Social Introversion; the Critical Incidence group scored somewhat higher. When analysis was done, the number of MMPI Critical Items between the two groups was similar as well. When scores on the NEO Personality Inventory (NEO) were compared, the critical incident individuals tended to score higher on Openness and on its subscales (Ideas, Actions, and Feelings). There was a positive correlation between Total Neuroticism T Score and number of MMPI critical items.^ Conclusions. This study shows that the current pre-screening process is working and would have saved the organization significant resources. ^ If one was to develop a profile of a candidate who potentially could suffer a critical incident and subsequently jeopardize the unit, mission and the safety of the public they would look like the following: either divorced or never married, score high on the MMPI in Social Introversion, score low on MMPI with an "excessive" amount of MMPI critical items; and finally scores high on the NEO Openness and subscales Ideas, Feelings, and Actions.^ Based on the results gleaned from the analysis in this study there seems to be several factors, within psychometric testing, that when taken together, will aid the evaluators in selecting only the highest quality operators in order to save resources and to help protect the public from unfortunate critical incidents which may adversely affect our health and safety.^

Process evaluation of the Texas occupational safety & health surveillance system

Introduction: The Texas Occupational Safety & Health Surveillance System (TOSHSS) was created to collect, analyze and interpret occupational injury and illness data in order to decrease the impact of occupational injuries within the state of Texas. This process evaluation was performed midway through the 4-year grant to assess the efficiency and effectiveness of the surveillance system’s planning and implementation activities1. ^ Methods: Two evaluation guidelines published by the Centers for Disease Control and Prevention (CDC) were used as the theoretical models for this process evaluation. The Framework for Program Evaluation in Public Health was used to examine the planning and design of TOSHSS using logic models. The Framework for Evaluating Public Health Surveillance Systems was used to examine the implementation of approximately 60 surveillance activities, including uses of the data obtained from the surveillance system. ^ Results/Discussion: TOSHSS planning activities omitted the creation of a scientific advisory committee and specific activities designed to maintain contacts with stakeholders; and proposed activities should be reassessed and aligned with ongoing performance measurement criteria, including the role of collaborators in helping the surveillance system achieve each proposed activity. TOSHSS implementation activities are substantially meeting expectations and received an overall score of 61% for all activities being performed. TOSHSS is considered a surveillance system that is simple, flexible, acceptable, fairly stable, timely, moderately useful, with good data quality and a PVP of 86%. ^ Conclusions: Through the third year of TOSHSS implementation, the surveillance system is has made a considerable contribution to the collection of occupational injury and illness information within the state of Texas. Implementation of the nine recommendations provided under this process evaluation is expected to increase the overall usefulness of the surveillance system and assist TDSHS in reducing occupational fatalities, injuries, and diseases within the state of Texas. ^ 1 Disclaimer: The Texas Occupational Safety and Health Surveillance System is supported by Grant/Cooperative Agreement Number (U60 OH008473-01A1). The content of the current evaluation are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health.^

Analysis of participation in the federally mandated coal workers' health surveillance program

The Federal Coal Mine Health and Safety Act of 1969 required that periodic chest radiographs be offered to underground coal miners to protect the miners from the development of Coal Workers' Pneumoconiosis (CWP) and progression of the disease to progressive massive fibrosis (PMF). These examinations are administered by the National Institute for Occupational Safety and Health (NIOSH) through the Coal Workers' Health Surveillance Program (CWHSP). The mine operator is required to provide each miner with the opportunity to have the chest radiograph at no cost to the miner.^ Three rounds of examinations have been conducted since 1969 and the fourth is underway. The decrease in participation over rounds is of great concern if the incidence and progression of CWP are to be understood and controlled.^ This study developed rates of participation for each of 558 West Virginia underground coal mines who submitted or had NIOSH assigned plans for making chest radiographs available during the third round, July 1978 through December 1980. These rates were analyzed in relation to desired levels of participation and to reinforcing, predisposing and enabling factors presumed to affect rates of participation in disease prevention and surveillance programs.^ Two reinforcing factors, size of mine and inclusion of the mine in the National Coal Study (NCS) epidemiology research program, and the enabling factor, use of an on-site radiograph facility, demonstrated highly significant relationships to participation rates.^ The major findings of the study were: (1) Participation in the CWHSP is even lower than previously estimated; (2) CWHSP program evaluation is not systematic and program data base is not complete and comprehensive; and (3) NIOSH program policy is not clear and administration of the CWHSP is fragmented and lacks adequate fiscal and personnel resources. ^

Parents Too Can Evaluate Family Preservation Services - or –Involving Parents In Evaluation of Family Preservation Services

In line with the long-standing emphasis in the human services on involving clients in program evaluation, in this essay I focus on principles and strategies for involving parents in evaluation of family preservation services. In particular, I delineate the crucial roles that parents can play as partners in the helping process within a family-centered context.

Parental Employment and Home Visiting Program Service Delivery

Home visiting programs, which provide in-home services to disadvantaged families with young children, rest on the assumption that poor parents can be reached at home. Increased levels of maternal employment raise questions about this assumption. In this study, longitudinal data collected for a home visiting program evaluation were analyzed to assess whether employment patterns of parents who receive home visiting services reflect employment patterns of other poor mothers between 1995 and 2000. The study also addresses the relationship between maternal employment and home visiting service intensity. To effectively reach home visiting participants, service providers may need to modify service delivery practices.

Health outcomes, quality improvement and cost-effectiveness of the Community Diabetes Education (CoDE) program: Results of a randomized controlled trial

The objectives of this dissertation were to evaluate health outcomes, quality improvement measures, and the long-term cost-effectiveness and impact on diabetes-related microvascular and macrovascular complications of a community health worker-led culturally tailored diabetes education and management intervention provided to uninsured Mexican Americans in an urban faith-based clinic. A prospective, randomized controlled repeated measures design was employed to compare the intervention effects between: (1) an intervention group (n=90) that participated in the Community Diabetes Education (CoDE) program along with usual medical care; and (2) a wait-listed comparison group (n=90) that received only usual medical care. Changes in hemoglobin A1c (HbA1c) and secondary outcomes (lipid status, blood pressure and body mass index) were assessed using linear mixed-models and an intention-to-treat approach. The CoDE group experienced greater reduction in HbA1c (-1.6%, p<.001) than the control group (-.9%, p<.001) over the 12 month study period. After adjusting for group-by-time interaction, antidiabetic medication use at baseline, changes made to the antidiabetic regime over the study period, duration of diabetes and baseline HbA1c, a statistically significant intervention effect on HbA1c (-.7%, p=.02) was observed for CoDE participants. Process and outcome quality measures were evaluated using multiple mixed-effects logistic regression models. Assessment of quality indicators revealed that the CoDE intervention group was significantly more likely to have received a dilated retinal examination than the control group, and 53% achieved a HbA1c below 7% compared with 38% of control group subjects. Long-term cost-effectiveness and impact on diabetes-related health outcomes were estimated through simulation modeling using the rigorously validated Archimedes Model. Over a 20 year time horizon, CoDE participants were forecasted to have less proliferative diabetic retinopathy, fewer foot ulcers, and reduced numbers of foot amputations than control group subjects who received usual medical care. An incremental cost-effectiveness ratio of $355 per quality-adjusted life-year gained was estimated for CoDE intervention participants over the same time period. The results from the three areas of program evaluation: impact on short-term health outcomes, quantification of improvement in quality of diabetes care, and projection of long-term cost-effectiveness and impact on diabetes-related health outcomes provide evidence that a community health worker can be a valuable resource to reduce diabetes disparities for uninsured Mexican Americans. This evidence supports formal integration of community health workers as members of the diabetes care team.^

Randomized comparisons among health informatics students identify hypertutorial features as improving web-based instruction.

Hypertutorials optimize five features - presentation, learner control, practice, feedback, and elaborative learning resources. Previous research showed graduate students significantly and overwhelmingly preferred Web-based hypertutorials to conventional "Book-on-the-Web" statistics or research design lessons. The current report shows that the source of hypertutorials' superiority in student evaluations of instruction lies in their hypertutorial features. Randomized comparisons between the two methodologies were conducted in two successive iterations of a graduate level health informatics research design and evaluation course. The two versions contained the same text and graphics, but differed in the presence or absence of hypertutorial features: Elaborative learning resources, practice, feedback, and amount of learner control. Students gave high evaluations to both Web-based methodologies, but consistently rated the hypertutorial lessons as superior. Significant differences localized in the hypertutorial subscale that measured student responses to hypertutorial features.

Commentary on “Connecting the Dots: Families and Children with Special Needs in a Rural Community”

Starks et al. discuss an evaluation of educational initiatives to improve rural systems of care for children with special needs. Akin to the medical home neighborhood concept, building a holistic medical "village" entails improving collaboration, sharing accountability, and helping families to navigate the health care system. The article suggests that policymakers should include patient experiences of care in program evaluation, and they should build a community infrastructure for shared information and efficient communication.

Family Preservation Journal, 1997, Volume 2, Issue 2. (Entire issue)

Entire issue (large pdf file) Articles include: A Model for Family Preservation Case Assessment. Kam-fong Monit Cheung, Patrick Leung and Sharon Alpert Behavioral Outcomes of Home-Based Services for Children and Adolescents with Serious Emotional Disorders. Edwin Morris, Lourdes Suarez, John C. Reid A Multi-Faceted, Intensive Family Preservation Program Evaluation. Michael Raschick Targeting Families to Receive Intensive Family Preservation Services: Assessing the Use of Imminent Risk of Placement as a Service Criterion. Elaine Walton and Ramona W. Denby