Controlled Software Evaluation Methodology: Case Simulation and Cognitive User Analysis

Software for use with patient records is challenging to design and difficult to evaluate because of the tremendous variability of patient circumstances. A method was devised by the authors to overcome a number of difficulties. The method evaluates and compares objectively various software products for use in emergency departments and compares software to conventional methods like dictation and templated chart forms. The technique utilizes oral case simulation and video recording for analysis. The methodology and experiences of executing a study using this case simulation are discussed in this presentation.

Can prospective usability evaluation predict data errors?

Increasing amounts of clinical research data are collected by manual data entry into electronic source systems and directly from research subjects. For this manual entered source data, common methods of data cleaning such as post-entry identification and resolution of discrepancies and double data entry are not feasible. However data accuracy rates achieved without these mechanisms may be higher than desired for a particular research use. We evaluated a heuristic usability method for utility as a tool to independently and prospectively identify data collection form questions associated with data errors. The method evaluated had a promising sensitivity of 64% and a specificity of 67%. The method was used as described in the literature for usability with no further adaptations or specialization for predicting data errors. We conclude that usability evaluation methodology should be further investigated for use in data quality assurance.

A process evaluation of the childhood lead poisoning prevention program in Houston, Texas, 1992--1995

A process evaluation of the Houston Childhood Lead Poisoning Prevention Program, 1992-1995, was conducted. The Program's goal is to reduce lead poisoning prevalence. The study proposed to determine to what extent the Program was implemented as planned by measuring how well Program services were actually: (1) received by the intended target population; (2) delivered to children with elevated blood lead levels; (3) delivered in compliance with the Centers for Disease Control and Prevention and Program guidelines and timetables; and (4) able to reduce lead poisoning prevalence among those rescreened. Utilizing a program monitoring design, the Program's pre-collected computer records were reviewed. The study sample consisted of 820 children whose blood lead levels were above 15 micrograms per deciLiter, representing approximately 2.9% of the 28,406 screened over this period. Three blood lead levels from each participant were examined: the initial elevated result; the confirmatory result; and the next rescreen result, after the elevated confirmatory level. Results showed that the Program screened approximately 18% (28,406 of 161,569) of Houston's children under age 6 years for lead poisoning. Based on Chi-square tests of significance, results also showed that lead-poisoned participants were more likely to be younger than 3 years, male and Hispanic, compared to those not lead poisoned. The age, gender and ethnic differences observed were statistically significant (p =.01, p =.00, p =.00). Four of the six Program services: medical evaluations, rescreening, environmental inspections and confirmation, had satisfactory delivery completion rates of 71%-98%. Delivery timetable compliance rates for three of the six services examined: outreach contacts, home visits and environmental inspections were below 32%. However, dangerously elevated blood lead levels fell and lead poisoning prevalence dropped from 3.3% at initial screening to 1.2% among those rescreened, after intervention. From a public health perspective, reductions in lead poisoning prevalence are very meaningful. Based on these findings, the following are recommendations for future research: (1) integrate Program database files by utilizing a computer database management program; (2) target services at Hispanic male children under age 3 years living in the highest risk neighborhoods; (3) increase resources to: improve tracking and documentation of service delivery and provide more non-medical case management and environmental services; and (4) share the evaluation methodology/findings with the Centers for Disease Control and Prevention administrators; the implications may be relevant to other program managers conducting such assessments. ^

Use of empowerment evaluation to determine a sustainability plan for a community-based preconception care program

Women With IMPACT (WWI) is a community-based preconception care educational intervention. WWI is being implemented by the Impacting Maternal and Prenatal Care Together (IMPACT) Collaborative and targets zip codes in Harris County, Texas at high risk for infant mortality, low birthweight, and preterm birth. WWI started March 2012 and continues through August 2013. Three workshop series are planned. This study was conducted with participants and facilitators from the first workshop series. This study aimed to 1) evaluate the WWI program using empowerment evaluation, 2) engage all WWI stakeholders in an empowerment evaluation so the method could be adopted as a participatory evaluation process for future IMPACT activities, and 3) develop recommendations for sustainability of the WWI intervention, based on empowerment evaluation findings and results from the pre/post program evaluation completed by WWI participants. Study participants included WWI participants and facilitators and IMPACT Collaborative Steering Committee members. WWI participants were female, 18-35 year-old, non-pregnant residents of zip codes at high risk of adverse birth outcomes. All other study participants were 18 years or older. A two-phased empowerment evaluation (EE) was utilized in this study. Sessions 1-4 were conducted independently of one another – 3 with participants at different sites and one with the facilitators. The fifth session included WWI participant and facilitator representatives, and IMPACT Steering Committee members. Session 5 built upon the work of the other sessions. Observation notes were recorded during each session. Thematic content analysis was conducted on all EE tables and observation notes. Mission statements drafted by each group focused on improvement of physical and mental health through behavior change and empowerment of all participants. The top 5 overall program components were: physical activity, nutrition, self-worth, in-class communication, and stress. Goals for program improvement were set by EE participants for each of these components. Through thematic content analysis of the tables and observation notes, social support emerged as an important theme of the program among all participant groups. Change to a healthy lifestyle emerged as an important theme in terms of program improvement. Two-phased EE provided an opportunity for all program stakeholders to provide feedback regarding important program components and provide suggestions for program improvement. EE, thematic content analysis, pre/post evaluation results, and inherent program knowledge were triangulated to make recommendations to sustain the program once the initial funding ends. ^

Youth-focused empowerment evaluation using Photovoice

The Carrera-Children's Aid Society of New York model is a proven model of teen pregnancy prevention. There is a need to evaluate the effect that the model has had on the participants as individuals. Brighter Futures is a replication of the Carrera-Children's Aid Society of New York youth development model operated by Planned Parenthood of Houston and Southeast Texas. A qualitative youth-focused Empowerment Evaluation was initiated to determine the individual impact of the achieved program outcomes. The Empowerment Evaluation assessed what impact the Brighter Futures program has had on the students. The youth used Community-based Participatory Research (CBPR) approaches, such as Photovoice, in conjunction with guided writing and participatory groups to conduct this evaluation with in the framework of Empowerment Evaluation. Additionally, a semi-structured CBPR exercise was used to evaluate the effectiveness of the feasibility of Photovoice methodology with urban youth. ^

Evaluation of deformable image registration for improved 4D CT-derived ventilation for image guided radiotherapy

Recent treatment planning studies have demonstrated the use of physiologic images in radiation therapy treatment planning to identify regions for functional avoidance. This image-guided radiotherapy (IGRT) strategy may reduce the injury and/or functional loss following thoracic radiotherapy. 4D computed tomography (CT), developed for radiotherapy treatment planning, is a relatively new imaging technique that allows the acquisition of a time-varying sequence of 3D CT images of the patient's lungs through the respiratory cycle. Guerrero et al. developed a method to calculate ventilation imaging from 4D CT, which is potentially better suited and more broadly available for IGRT than the current standard imaging methods. The key to extracting function information from 4D CT is the construction of a volumetric deformation field that accurately tracks the motion of the patient's lungs during the respiratory cycle. The spatial accuracy of the displacement field directly impacts the ventilation images; higher spatial registration accuracy will result in less ventilation image artifacts and physiologic inaccuracies. Presently, a consistent methodology for spatial accuracy evaluation of the DIR transformation is lacking. Evaluation of the 4D CT-derived ventilation images will be performed to assess correlation with global measurements of lung ventilation, as well as regional correlation of the distribution of ventilation with the current clinical standard SPECT. This requires a novel framework for both the detailed assessment of an image registration algorithm's performance characteristics as well as quality assurance for spatial accuracy assessment in routine application. Finally, we hypothesize that hypo-ventilated regions, identified on 4D CT ventilation images, will correlate with hypo-perfused regions in lung cancer patients who have obstructive lesions. A prospective imaging trial of patients with locally advanced non-small-cell lung cancer will allow this hypothesis to be tested. These advances are intended to contribute to the validation and clinical implementation of CT-based ventilation imaging in prospective clinical trials, in which the impact of this imaging method on patient outcomes may be tested.

Youth empowerment evaluation to improve afterschool programming

Community-based participatory research necessitates that community members act as partners in decision making and mutual learning and discovery. In the same light, for programs/issues involving youth, youth should be partners in knowledge sharing and evaluation (Checkoway & Richards-Schuster, 2004). This study is a youth-focused empowerment evaluation for the Successful Youth program. Successful Youth is a multi-component youth development after-school program for Latino middle school youth, created with the goal of reducing teen pregnancy. An empowerment evaluation is collaborative and participatory (Balcazar and Harper 2003). The three steps of an empowerment evaluation are: (1) defining mission, (2) taking stock, and (3) planning for the future (Fetterman 2001).^ In a program where youth are developing leadership skills, making choices, and learning how to self reflect and evaluate, the empowerment evaluation could not be more aligned with promoting and enhancing these skills. In addition, an empowerment evaluation is designed to "foster improvement and self-determination" and "build capacity" (Fetterman 2001). Four empowerment groups were conducted with approximately 6-9 Latino 7th grade students per group. All participants were enrolled in the Successful Youth program. Results indicate points where students' perceptions of the program were aligned with the program's mission and where gaps were identified. Students offered recommendations for program improvements. Additionally, students enjoyed expressing their feelings about the program and appreciated that their opinions were valued. Youth recommendations will be brought to program staff; and, where possible, gaps will be addressed. Empowerment evaluations with youth will continue during the duration of the program so that youth involvement and input remains integral in the evaluation and to ascertain whether the program's goals are being met. ^

Process evaluation of intervention program for prevention of early sexual activity and alcohol use among Hispanic teenagers

Hispanic teens are a high-risk population for initiation of early sexual activity and alcohol use which in turn has numerous social and health consequences. One strategy to address prevention of these behaviors is to implement a capacity building intervention that promotes parent child communication, encompasses their cultural values and community participation. This study describes the process evaluation of a pilot intervention program amongst Hispanic teens and their families living along the Texas-Mexico border. “Girls Lets Talk” is a small group intervention with 10-14 year old teens and their female adult family members that involves education regarding effects of alcohol use and sexual activity as well as activities for monitoring and refusal skills to prevent risky behaviors. Two waves of the program each consisting of at least seven mother daughter dyads were conducted. During the designing process, community advisory board meetings and focus groups were held to review course materials and ensure they were appropriate to the Mexican American culture. Parent and adolescent surveys were administered at the beginning and end of the intervention to assess for psychosocial outcome variables. All sessions received high mean satisfactory scores (mean of 4.00 or better on a five point scale) for both adult and adolescent participants. Qualitative feedback was obtained via debriefing sessions to evaluate experience as well as alter recruitment strategies. A Wilcoxon Sign Rank analysis of the pre and post intervention surveys was done that showed significant changes in some outcome variables such as intentions and confidence for monitoring behaviors for adults and beliefs regarding sexual activity. “Girls Lets Talk” is a promising example of how a process evaluation plan can help develop a theory based health promotion program using the community based participatory research approach. The intervention may also be effective in altering intentions and enhancing self-efficacy among parents and teens in order to decrease risky behaviors such as early sexual activity and alcohol use.^

Sacred vocation program follow up evaluation at the Baylor healthcare system

Background: The Sacred Vocation Program (SVP) (Amick B, Karff S., 2003) helps workers find meaning, spirituality, and see their job as a sacred vocation. The SVP is based on Participatory Action Research (PAR) (Minkler & Wallerstein, 1997; Parker & Wall, 1998). This study aims to evaluate the SVP implemented at the Baylor Healthcare System, Dallas-Fort Worth. ^ Methods: The study design is a qualitative design. We used data from study participants who have participated in focus groups. During these focus groups specific questions and probes regarding the effectiveness of the SVP have been asked. We analyzed the focus groups and derived themes. ^ Results: Results of this study demonstrate SVP helps graduates feel valued and important. The SVP has improved meaningful work for employees and improved a sense of belonging for participants. The program has also increased participant spirituality. The coping techniques developed during a SVP class helps participants deal with stressful situations. The SVP faces challenges of implementation fidelity, poor communication, program viability in tough economic times and implementation of phase II. Another sustainability challenge for SVP is the perception of the program being a religious one versus a spiritual program. ^ Conclusion: Several aspects of the SVP work. The phase I of SVP is successful in improving meaningful work and a sense of belonging for participants. The coping techniques help participants deal with difficult work situations. The SVP can increase effectiveness through improvements in implementation fidelity, communication and leadership commitment. ^

Perceptions of institutional support of community -based participatory research within prevention research centers

The Institute of Medicine (IOM) report on the future of health care states that the focus on health needs to shift to the management and prevention of chronic illnesses and that academic health centers (AHCs) should play an active role in this process through community partnerships (IOM, 2002). Grant funding from the National Institutes of Health and the creation of the Centers for Disease Control and Prevention (CDC) Prevention Research Centers (PRC) across the county represent a transition toward more proactively seeking out community partnerships to better design and disseminate health promotion programs (Green, 2001). ^ The focus of the PRCs is to conduct rigorous, community-based, prevention research, to seek outcomes applicable to public health programs and policies. The PRCs work is to create and foster partnerships among public health and community organizations, to address health promotion and disease prevention issues (CDC, 2003). ^ The W.K. Kellogg Foundation defines CBPR as "a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBPR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health." ^ In 1995, CDC asked the IOM to review the PRC program to examine the extent to which the program is providing the public health community with strategies to address public health problems in disease prevention and health promotion (IOM, 1997). No comprehensive evaluation n of the individual PRCs had ever been done (IOM, 1997). ^ The CDC was interested in understanding how it could better support the PRC program through improved management and oversight to influence the program's success. The CDC only represents one of the entities that influence the success of a PRC. Another key entity to consider is the support of and influence of the Schools of Public Health in which the PRCs reside. Using evaluation criteria similar to those that were developed by the IOM, this study examined how aspects of structural capacity of the Schools of Public Health in which the PRCs reside are perceived to influence PRC community-based research activities. ^