Preliminary Evaluation of a Guideline Classification System

In a previous paper, we presented a proposed expansion of the National Guideline Clearing-house (NGC) classification1. We performed a preliminary evaluation of the classification based on 100 guidelines randomly selected from the NGC collection. We found that 89 of the 100 guidelines could be assigned to a single guideline category. To test inter-observer agreement, twenty guidelines were also categorized by a second investigator. Agreement was found to be 40-90% depending on the axis, which compares favorably with agreement among MeSH indexers (30-60%)2. We conclude that categorization is feasible. Further research is needed to clarify axes with poor inter-observer agreement.

An evaluation of the personal response system program for the elderly in Harris County, Texas

The Personal Response System Program at Huffington Center on Aging, Baylor College of Medicine, provides emergency call systems for elderly people living independently in Houston, Texas. The goal of the project was to complete a formative evaluation of the Personal Response System Program. The specific aims of the evaluation were three-fold. One aim was to evaluate participant health status and level of disability. The second aim was to develop a health care cost estimation strategy. Finally, a preliminary cost-effectiveness analysis was completed to evaluate the program's impact on health care costs and health status among the elderly target population. ^ The evaluation was a longitudinal, randomized experimental design. After the screening of 120 volunteers for eligibility, clients were asked to complete a written questionnaire and a monthly health service contact diary. Volunteers were contacted by telephone interviewers to collect health status information from 100 eligible clients (83%) on three occasions during the six months of follow-up. ^ Initially, volunteers were randomized to two experimental groups. The two groups were found to be comparable at the beginning of the study. No significant differences were detected related to health status, level of disability, or history of physician visits at baseline. However, the group with the Personal Response System (PRS) device had more adverse health events, higher IADL scores, more frequent use of walkers, lower average health status scores, and fewer community volunteers hours than the usual care comparison group. ^ The health care costs were estimated based on an algorithm adapted from the American Medical Association guidelines. Average total health care costs for the group with the PRS device ($912) were greater than the usual care group ($464). However, median health care values for the PRS group ($263) were similar to the usual care comparison group ($234). The preliminary findings indicated that the use of the PRS device was not associated with health care cost savings. ^ In the preliminary cost-effectiveness analysis, use of the personal response system was found to be associated with increased mental health status among elderly clients. The cost-effectiveness evaluation indicated that the associated cost for six months was $710 per unit increase in mental component score when the $150 PRS subscription was included. However, clients with the PRS device were found to have a greater decline in physical health status during the six-month follow-up. The beneficial effect on mental health status was found to be in contrast to negative findings associated with changes in physical health status. The implications for future research relate to the need to identify risk factors among geriatric populations to better target groups that would most likely benefit from PRS Program enrollment. ^

A MANAGEMENT INFORMATION SYSTEM AND EVALUATION SCHEME FOR THE UNIVERSITY OF TEXAS EMPLOYEE ASSISTANCE PROGRAMS

Early Employee Assistance Programs (EAPs) had their origin in humanitarian motives, and there was little concern for their cost/benefit ratios; however, as some programs began accumulating data and analyzing it over time, even with single variables such as absenteeism, it became apparent that the humanitarian reasons for a program could be reinforced by cost savings particularly when the existence of the program was subject to justification.^ Today there is general agreement that cost/benefit analyses of EAPs are desirable, but the specific models for such analyses, particularly those making use of sophisticated but simple computer based data management systems, are few.^ The purpose of this research and development project was to develop a method, a design, and a prototype for gathering managing and presenting information about EAPS. This scheme provides information retrieval and analyses relevant to such aspects of EAP operations as: (1) EAP personnel activities, (2) Supervisory training effectiveness, (3) Client population demographics, (4) Assessment and Referral Effectiveness, (5) Treatment network efficacy, (6) Economic worth of the EAP.^ This scheme has been implemented and made operational at The University of Texas Employee Assistance Programs for more than three years.^ Application of the scheme in the various programs has defined certain variables which remained necessary in all programs. Depending on the degree of aggressiveness for data acquisition maintained by program personnel, other program specific variables are also defined. ^

The design and evaluation of a 2D dose verification system for intensity modulated radiotherapy

The usage of intensity modulated radiotherapy (IMRT) treatments necessitates a significant amount of patient-specific quality assurance (QA). This research has investigated the precision and accuracy of Kodak EDR2 film measurements for IMRT verifications, the use of comparisons between 2D dose calculations and measurements to improve treatment plan beam models, and the dosimetric impact of delivery errors. New measurement techniques and software were developed and used clinically at M. D. Anderson Cancer Center. The software implemented two new dose comparison parameters, the 2D normalized agreement test (NAT) and the scalar NAT index. A single-film calibration technique using multileaf collimator (MLC) delivery was developed. EDR2 film's optical density response was found to be sensitive to several factors: radiation time, length of time between exposure and processing, and phantom material. Precision of EDR2 film measurements was found to be better than 1%. For IMRT verification, EDR2 film measurements agreed with ion chamber results to 2%/2mm accuracy for single-beam fluence map verifications and to 5%/2mm for transverse plane measurements of complete plan dose distributions. The same system was used to quantitatively optimize the radiation field offset and MLC transmission beam modeling parameters for Varian MLCs. While scalar dose comparison metrics can work well for optimization purposes, the influence of external parameters on the dose discrepancies must be minimized. The ability of 2D verifications to detect delivery errors was tested with simulated data. The dosimetric characteristics of delivery errors were compared to patient-specific clinical IMRT verifications. For the clinical verifications, the NAT index and percent of pixels failing the gamma index were exponentially distributed and dependent upon the measurement phantom but not the treatment site. Delivery errors affecting all beams in the treatment plan were flagged by the NAT index, although delivery errors impacting only one beam could not be differentiated from routine clinical verification discrepancies. Clinical use of this system will flag outliers, allow physicists to examine their causes, and perhaps improve the level of agreement between radiation dose distribution measurements and calculations. The principles used to design and evaluate this system are extensible to future multidimensional dose measurements and comparisons. ^

Process evaluation of the Texas occupational safety & health surveillance system

Introduction: The Texas Occupational Safety & Health Surveillance System (TOSHSS) was created to collect, analyze and interpret occupational injury and illness data in order to decrease the impact of occupational injuries within the state of Texas. This process evaluation was performed midway through the 4-year grant to assess the efficiency and effectiveness of the surveillance system’s planning and implementation activities1. ^ Methods: Two evaluation guidelines published by the Centers for Disease Control and Prevention (CDC) were used as the theoretical models for this process evaluation. The Framework for Program Evaluation in Public Health was used to examine the planning and design of TOSHSS using logic models. The Framework for Evaluating Public Health Surveillance Systems was used to examine the implementation of approximately 60 surveillance activities, including uses of the data obtained from the surveillance system. ^ Results/Discussion: TOSHSS planning activities omitted the creation of a scientific advisory committee and specific activities designed to maintain contacts with stakeholders; and proposed activities should be reassessed and aligned with ongoing performance measurement criteria, including the role of collaborators in helping the surveillance system achieve each proposed activity. TOSHSS implementation activities are substantially meeting expectations and received an overall score of 61% for all activities being performed. TOSHSS is considered a surveillance system that is simple, flexible, acceptable, fairly stable, timely, moderately useful, with good data quality and a PVP of 86%. ^ Conclusions: Through the third year of TOSHSS implementation, the surveillance system is has made a considerable contribution to the collection of occupational injury and illness information within the state of Texas. Implementation of the nine recommendations provided under this process evaluation is expected to increase the overall usefulness of the surveillance system and assist TDSHS in reducing occupational fatalities, injuries, and diseases within the state of Texas. ^ 1 Disclaimer: The Texas Occupational Safety and Health Surveillance System is supported by Grant/Cooperative Agreement Number (U60 OH008473-01A1). The content of the current evaluation are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health.^

EVALUATION OF AN AUTOMATED AMBULATORY MEDICAL RECORD SYSTEM

This research was intended to evaluate an automated ambulatory medical record and chart review system. Chart review as conceptualized in this research is a series of statements that are made by the computer after reviewing the patients entire computer medical record. The actual chart review st

APPLICATION OF THE BRAINSTEM EVOKED RESPONSE (BER) TO NEUROTOXICITY EVALUATION OF THE AUDITORY SYSTEM: ASSESSMENT OF INORGANIC LEAD-INDUCED OTOTOXICITY

An experimental procedure was developed using the Brainstem Evoked Response (BER) electrophysiological technique to assess the effect of neurotoxic substances on the auditory system. The procedure utilizes Sprague-Dawley albino rats who have had dural electrodes implanted in their skulls, allowing neuroelectric evoked potentials to be recorded from their brainstems. Latency and amplitude parameters derived from the evoked potentials help assess the neuroanatomical integrity of the auditory pathway in the brainstem. Moreover, since frequency-specific auditory stimuli are used to evoke the neural responses, additional audiometric information is obtainable. An investigation on non-exposed control animals shows the BER threshold curve obtained by tests at various frequencies very closely approximates that obtained by behavioral audibility tests. Thus, the BER appears to be a valid measure of both functional and neuroanatomical integrity of the afferent auditory neural pathway.^ To determine the usefulness of the BER technique in neurobehavioral toxicology research, a known neurotoxic agent, Pb, was studied. Female Sprague-Dawley rats were dosed for 45 days with low levels of Pb acetate in their drinking water, after which BER recordings were obtained. The Pb dosages were determined from the findings of an earlier pilot study. One group of 6 rats received normal tap water, one group of 7 rats received a solution of 0.1% Pb, and another group of 7 rats received a solution of 0.2% Pb. After 45 days, the three groups exhibited blood Pb levels of 4.5 (+OR-) 0.43 (mu)g/100 ml, 37.8 (+OR-) 4.8 (mu)g/100 ml and 47.3 (+OR-) 2.7 (mu)g/100 ml, respectively.^ The results of the BER recording indicated evoked response waveform latency abnormalities in both the Pb-treated groups when midrange frequency (8 kHz to 32 kHz) stimuli were used. For the most part, waveform amplitudes did not vary significantly from control values. BER recordings obtained after a 30-day recovery period indicated the effects seen in the 0.1% Pb group had disappeared. However, those anomalies exhibited by the 0.2% Pb group either remained or increased in number. This outcome indicates a longer lasting or possibly irreversible effect on the auditory system from the higher dose of Pb. The auditory pathway effect appears to be in the periphery, at the level of the cochlea or the auditory (VIII) nerve. The results of this research indicate the BER technique is a valuable and sensitive indicator of low-level toxic effects on the auditory system.^

THE ALCOHOL TREATMENT PROFILE SYSTEM: ITS DEVELOPMENT AND AN EVALUATION OF ITS CAPABILITIES

The development of the Alcohol Treatment Profile System (ATPS) was described and an evaluation of its perceived value by various States was undertaken, The ATPS is a treatment needs assessment tool based on the unification of several large national epidemiologic and treatment data sets. It was developed by the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and responsibility for its creation was given to the NIAAA's Alcohol Epidemiologic Data System (AEDS). The ATPS merges county-level measures of alcohol problem prevalence (the specially constructed AEDS Alcohol Problem Indicators), indicating "need" for treatment, and treatment utilization measures (the National Drug and Alcohol Treatment Utilization Survey), indicating treatment "demand." The capabilities of the ATPS in the unique planning and policy-making settings of several States were evaluated.^

Sacred vocation program follow up evaluation at the Baylor healthcare system

Background: The Sacred Vocation Program (SVP) (Amick B, Karff S., 2003) helps workers find meaning, spirituality, and see their job as a sacred vocation. The SVP is based on Participatory Action Research (PAR) (Minkler & Wallerstein, 1997; Parker & Wall, 1998). This study aims to evaluate the SVP implemented at the Baylor Healthcare System, Dallas-Fort Worth. ^ Methods: The study design is a qualitative design. We used data from study participants who have participated in focus groups. During these focus groups specific questions and probes regarding the effectiveness of the SVP have been asked. We analyzed the focus groups and derived themes. ^ Results: Results of this study demonstrate SVP helps graduates feel valued and important. The SVP has improved meaningful work for employees and improved a sense of belonging for participants. The program has also increased participant spirituality. The coping techniques developed during a SVP class helps participants deal with stressful situations. The SVP faces challenges of implementation fidelity, poor communication, program viability in tough economic times and implementation of phase II. Another sustainability challenge for SVP is the perception of the program being a religious one versus a spiritual program. ^ Conclusion: Several aspects of the SVP work. The phase I of SVP is successful in improving meaningful work and a sense of belonging for participants. The coping techniques help participants deal with difficult work situations. The SVP can increase effectiveness through improvements in implementation fidelity, communication and leadership commitment. ^

The Comprehensive Assessment of Leadership for Learning: A Next Generation Formative Evaluation and Feedback System

Recent developments in federal policy have prompted the creation of state evaluation frameworks for principals and teachers that hold educators accountable for effective practices and student outcomes. These changes have created a demand for formative evaluation instruments that reflect current accountability pressures and can be used by schools to focus school improvement and leadership development efforts. The Comprehensive Assessment of Leadership for Learning (CALL) is a next generation, 360-degree on-line assessment and feedback system that reflect best practices in feedback design. Some unique characteristics of CALL include a focus on: leadership distributed throughout the school rather than as carried out by an individual leader; assessment of leadership tasks rather than perceptions of leadership practice; a focus on larger complex systems of middle and high school; and transparency of assessment design. This paper describes research contributing to the design and validation of the CALL survey instrument.