A comprehensive framework for evaluating funder's performance

Evaluation methods for assessing the performance of non-profit funders are lacking. The purpose of the research was to create a comprehensive framework that systematically assesses the goals and objectives of a funder, how these relate to the funder's allocation of resources, and the potential impact of programs and services selected by the funder for resource allocation to address organizational goals and objectives. The Houston Affiliate of Susan G. Komen for the Cure, a local chapter of a national breast cancer awareness advocacy organization, was selected as the funding agency whose performance assessment was to assist in the creation of this framework. Evaluation approaches from the government sector were adapted and incorporated into the research to guide the methods used to answer the three research questions corresponding to the three phases of research within the study: (1) what are the funding goals and objectives of the Affiliate?; (2) what allocation scheme does the organization use to address these goals and objectives and select programs for funding?; and, (3) to what extent do the programs funded by the Affiliate have potential long-term impact? ^ Within the first stage of the research, document reviews of the Affiliate's mission-based documents and bylaws and interviews with organizational and community informants revealed a highly latent constellation of broad objectives that were not formalized into one guiding document, thus creating gaps in management and governance. Within the second phase of the research, reviews of grant applications from the 2008-2009 funding cycle and interviews with employees and volunteers familiar with the funding process revealed competing ideas regarding resource allocation in light of vague organizational documents describing funding goals and objectives. Within the final stage of the research, these findings translated to the Affiliate selecting programs with highly varying potential long-term impact with regards to addressing goals and objectives relating to breast cancer education, screening, diagnostics, treatment, and support. The resulting performance assessment framework, consisting of three phases of research utilizing organizational documents and key informant interviews, demonstrated the importance of clearly defined funding goals and objectives, reference documents and committee participation within the funding process, and regular reviews of potential long-term impact for selected programs, all supported by the active participation and governance of a funder's Board of Directors.^

An evaluation of sleep hypnotic and stimulant effect on human performance using a computerized psychological test battery

This study evaluated the administration-time-dependent effects of a stimulant (Dexedrine 5-mg), a sleep-inducer (Halcion 0.25-mg) and placebo (control) on human performance. The investigation was conducted on 12 diurnally active (0700-2300) male adults (23-38 yrs) using a double-blind, randomized sixway-crossover three-treatment, two-timepoint (0830 vs 2030) design. Performance tests were conducted hourly during sleepless 13-hour studies using a computer generated, controlled and scored multi-task cognitive performance assessment battery (PAB) developed at the Walter Reed Army Institute of Research. Specific tests were Simple and Choice Reaction Time, Serial Addition/Subtraction, Spatial Orientation, Logical Reasoning, Time Estimation, Response Timing and the Stanford Sleepiness Scale. The major index of performance was "Throughput", a combined measure of speed and accuracy.^ For the Placebo condition, Single and Group Cosinor Analysis documented circadian rhythms in cognitive performance for the majority of tests, both for individuals and for the group. Performance was best around 1830-2030 and most variable around 0530-0700 when sleepiness was greatest (0300).^ Morning Dexedrine dosing marginally enhanced performance an average of 3% with reference to the corresponding in time control level. Dexedrine AM also increased alertness by 10% over the AM control. Dexedrine PM failed to improve performance with reference to the corresponding PM control baseline. With regard to AM and PM Dexedrine administrations, AM performance was 6% better with subjects 25% more alert.^ Morning Halcion administration caused a 7% performance decrement and 16% increase in sleepiness and a 13% decrement and 10% increase in sleepiness when administered in the evening compared to corresponding in time control data. Performance was 9% worse and sleepiness 24% greater after evening versus morning Halcion administration.^ These results suggest that for evening Halcion dosing, the overnight sleep deprivation occurring in coincidence with the nadir in performance due to circadian rhythmicity together with the CNS depressant effects combine to produce performance degradation. For Dexedrine, morning administration resulted in only marginal performance enhancement; Dexedrine in the evening was less effective, suggesting the 5-mg dose level may be too low to counteract the partial sleep deprivation and nocturnal nadir in performance. ^

Validation of an instrument for assessing the medicolegal death investigation system: A public health approach

Background. This study validated the content of an instrument designed to assess the performance of the medicolegal death investigation system. The instrument was modified from Version 2.0 of the Local Public Health System Performance Assessment Instrument (CDC) and is based on the 10 Essential Public Health Services. ^ Aims. The aims were to employ a cognitive testing process to interview a randomized sample of medicolegal death investigation office leaders, qualitatively describe the results, and revise the instrument accordingly. ^ Methods. A cognitive testing process was used to validate the survey instrument's content in terms of the how well participants could respond to and interpret the questions. Twelve randomly selected medicolegal death investigation chiefs (or equivalent) that represented the seven types of medicolegal death investigation systems and six different state mandates were interviewed by telephone. The respondents also were representative of the educational diversity within medicolegal death investigation leadership. Based on respondent comments, themes were identified that permitted improvement of the instrument toward collecting valid and reliable information when ultimately used in a field survey format. ^ Results. Responses were coded and classified, which permitted the identification of themes related to Comprehension/Interpretation, Retrieval, Estimate/Judgment, and Response. The majority of respondent comments related to Comprehension/Interpretation of the questions. Respondents identified 67 questions and 6 section explanations that merited rephrasing, adding, or deleting examples or words. In addition, five questions were added based on respondent comments. ^ Conclusion. The content of the instrument was validated by cognitive testing method design. The respondents agreed that the instrument would be a useful and relevant tool for assessing system performance. ^

An ecologic comparison study of the impact of economic disadvantage on Texas Assessment of Knowledge and Skills performance, graduation rates, and readiness for higher education for students attending public and charter schools in Texas from 2004 to 2006

Purpose. No Child Left Behind aimed to "improve the academic achievement of the disadvantaged." The primary research question considered how academic achievement of those from economic disadvantage compared to those not from disadvantage? ^ Economically disadvantaged students can potentially have added academic disadvantage. Research shows low academic achievement can potentially result in drug abuse, youth violence, and teen pregnancy. ^ Methods. To compare the student populations, measures included TAKS results and academic indicator data collected by the Texas Education Agency. ^ Results. T-test analyses showed a significant difference between the economically and non-economically disadvantaged student populations in meeting the TAKS passing standard, graduation, and preparation for higher education.^ Conclusions. The achievement gap between students remained as indicated by the Texas testing program. More research and time are needed to observe if the desired impact on those from economic disadvantage will be reflected by academic achievement data.^

Association of self-reported physical activity and performance in the physical fitness assessment among Airmen

The U.S. Air Force, as with the other branches of military services, has physical fitness standards imposed on their personnel. These standards ensure a healthy and fit combat force. To meet these standards, Airmen have to maintain a certain level of physical activity in their lifestyle. Objective. This was a cross sectional (prevalence) study to evaluate the association of Airmen's self-reported physical activity and their performance in the Air Force Physical Fitness Assessment in 2007. Methods. The self-reported physical activity data were obtained from the Air Force Web Health Assessment (AF WEB HA), a web-based health questionnaire completed by the Airmen during their annual Preventive Health Assessment. The physical activity levels were categorized as having met or not having met the Centers for Disease Control and Prevention (CDC) and the American College of Sports Medicine (ACSM) physical activity recommendations. Physical Fitness scores were collected from the Air Force Fitness Management System (AFFMS), a repository of physical fitness test data. Results. There were 49,029 Airmen who answered the AF WEB HA in 2007 and also took their physical fitness test. 94.4% (n = 46,304) of Airmen met the recommended physical activity guidelines and 79.9% (n = 39,178) passed the fitness test. Total Airmen who both met the physical activity recommendations and passed the fitness test was 75.6% (n = 37,088). Airmen who did not meet the activity recommendations and also failed the fitness test totaled 635 or 1.3% of the study group. The Mantel-Haenszel Chi-Square analysis of the data on the activity levels and the physical fitness test relationship was the following χ2 = 18.52, df 1, and p = <0.0001. The Odds Ratio (OR) was 1.22 (95% CI 1.12, 1.34). Conclusion. The study determined that there was a positive association between Airmen's self-reported physical activity and their performance in the physical fitness assessment.^

Assessment of the performance of 3m 3500 organic vapor monitors over extended sampling durations

The purpose of this study was to assess the accuracy and precision of airborne volatile organic compound (VOC) concentrations measured using passive air samplers (3M 3500 organic vapor monitors) over extended sampling durations (9 and 15 days). A total of forty-five organic vapor monitor samples were collected at a State of Texas air monitoring site during two different sampling periods (July/August and November 2008). The results of this study indicate that for most of the tested compounds, there was no significant difference between long-term (9 or 15 days) sample concentrations and the means of parallel consecutive short-term (3 days) sample concentrations. Biases of 9 or 15-day measurements vs. consecutive 3-day measurements showed considerable variability. Those compounds that had percent bias values of <10% are suggested as acceptable for long-term sampling (9 and 15 days). Of the twenty-one compounds examined, 10 compounds are classified as acceptable for long-term sampling; these include m,p-xylene, 1,2,4-trimethylbenzene, n-hexane, ethylbenzene, benzene, toluene, o-xylene, d-limonene, dimethylpentane and methyl tertbutyl ether. The ratio of sampling procedure variability relative to variability within days was approximately 1.89 for both sampling periods for the 3-day vs. 9-day comparisons and approximately 2.19 for both sampling periods for the 3-day vs. 15-day comparisons. Considerably higher concentrations of most VOCs were measured during the November sampling period compared to the July/August period. These differences may be a result of varying meteorological conditions during these two time periods, e.g., the differences in wind direction, and wind speed. Further studies are suggested to further evaluate the accuracy and precision of 3M 3500 organic vapor monitors over extended sampling durations. ^

THE RELATIONSHIP BETWEEN ATTITUDE ASSESSMENT/ROLE PERCEPTION AND THE FINANCIAL PERFORMANCE OF HOSPITAL DEPARTMENT MANAGERS

Much of the current healthcare financial literature addresses the concern of government officials, the public, and healthcare providers regarding the need for control of health care costs. The literature suggests that attitudes of hospital department managers toward their role in financial management affects their ability to effect favorable financial results.^ There were several objectives of the dissertation: (1) To identify whether or not there exists a relationship between the attitude/role perception of hospital managers and the financial performance of their departments. (2) To compile a descriptive survey data base of key factors identified in the financial literature from individual hospitals. (3) To compile a brief descriptive survey of hospital managers' financial management background and training (both formal and informal). (4) To conduct an attitude assessment/role perception survey regarding the importance or relevance of a suggested financial management role set (i.e., issues discussed in the current literature) as viewed by the selected hospital managers and their matched administrators. (5) To propose plausible theoretical models and statistical tests of seven proposed hypotheses.^ The statistical results of a variety of methods generally suggested, for the sample population, that the null hypothesis should not be rejected concerning the relationships between a department manager's financial attitudes and role perceptions and the resultant financial performance.^ The fact that the results of this study did not suggest that there was a significant relationship which existed between role perception and financial performance does not necessarily indicate that the theories supporting such a relationship in literature are false, not that such a relationship does not exist. Several alternative theories were postulated to explain the apparent lack of statistical relationship, and suggestions for refinement and/or improvement of further research were discussed. ^

An assessment of the relative influence of a vapor recovery system in reducing occupational exposure to volatile organic compounds in high performance liquid chromatography

Occupational exposures to organic solvents, specifically acetonitrile and methanol, have the potential to cause serious long-term health effects. In the laboratory, these solvents are used extensively in protocols involving the use of high performance liquid chromatography (HPLC). Operators of HPLC equipment may be potentially exposed to these organic solvents when local exhaust ventilation is not employed properly or is not available, which can be the case in many settings. The objective of this research was to characterize the various sites of vapor release in the HPLC process and then to determine the relative influence of a novel vapor recovery system on the overall exposure to laboratory personnel. The effectiveness of steps to reduce environmental solvent vapor concentrations was assessed by measuring exposure levels of acetonitrile and methanol before and after installation of the vapor recovery system. With respect to acetonitrile, the concentration was not statistically significant with p=0.938; moreover, exposure after the intervention was actually higher than prior to intervention. With respect to methanol, the concentration was not statistically significant with p=0.278. This indicates that the exposure to methanol after the intervention was not statistically significantly higher or lower than prior to intervention. Thus, installation of the vapor recovery device did not result in statistically significant reduction in exposures in the settings encountered, and acetonitrile actually increased significantly.^

Noninvasive assessment of diffusion hypoxia following administration of nitrous oxide-oxygen.

The phenomenon of diffusion hypoxia is commonly believed to occur unless nitrous oxide-oxygen inhalation sedation is followed by "washout" with 100% oxygen for 5 minutes upon termination of the flow of nitrous oxide. When systematically studied, however, this phenomenon generally appears to be unfounded. The present study evaluated the effect of breathing room air instead of 100% oxygen in healthy (ASA 1) human volunteers following administration of sedative concentrations of nitrous oxide. The occurrence of hypoxia was determined objectively, using pulse oximetry and a standardized psychomotor skills test (Trieger test). Diffusion hypoxia was not observed using these criteria.

Emergency health care assessment in the Houston-Galveston region 2002--2006

Background. Previous studies show emergency rooms to be over crowded nation wide. With growing attention to this problem, the Houston-Galveston Area Council (H-GAC) initiated a study in 2005 to assess their region's emergency health care system, and continued this effort in 2007. The purpose of this study was to examine recent changes in volume, capacity and performance in the Houston-Galveston region's emergency health care system and determine whether the system has been able to effectively respond to the residents' demands. Methods. Data were collected by the Houston-Galveston Area Council and The Abaris Group using a self-administered 2002-2006 survey completed by administrators of the region's hospitals, EMS providers, and select fire departments that provide EMS services. Data from both studies were combined and matched to examine trends. Results. Volume increased among the reporting hospitals within the Houston-Galveston region from 2002 to 2006; however, capacity remained relatively unchanged. EMS providers reported higher average off load times in 2007 compared to 2005, but the increases were not statistically significant. Hospitals reported transferring a statistically significant greater percentage of patients in 2006 than 2004. There was no statistically significant change in any of the other measures. Conclusion. These findings indicate an increase in demand for the Houston-Galveston region's emergency healthcare services with no change in supply. Additional studies within the area are needed to fully assess and evaluate the impact of these changes on system performance. ^

Assessment of US Air Force student pilots with intermittent monofixation syndrome on a non-stereoptic dependent flight maneuver in pilot training

The United States Air Force School of Aerospace Medicine (USAFSAM) and Aeromedical Consult Service (ACS) have developed waiver criteria for pilots with subtle substandard depth perception. This is to allow United States Air Force (USAF) pilots with mild depth perception deficiency to continue flying duties while limiting the risk to flight safety and ensuring the availability of costly human resources. From 1999 to 2005, 166 aviators were given waivers for intermittent monofixation syndrome (IMFS). Of these, 96 were student pilots who performed slightly worse at stereoptic dependent flight maneuvers than student pilots (8,907) with normal depth perception (Lowry, 2006).^ This study's purpose is to evaluate the performance of the extended-trail maneuver, a non-stereoptic dependent flying maneuver, as executed by a cohort of 12 United States Air Force student pilots with intermittent monofixation syndrome versus the cohort of 100 student pilots with normal depth perception. These subjects are extracted from the cohorts examined by Lowry (2006) and the null hypothesis predicts no statistical difference in the performance of the non-stereoptic dependant flight maneuver extended-trail between student pilots with intermittent monofixation syndrome and those without the condition. ^

Critical indicators associated with negative performance in a selectively manned unit

Introduction. Selectively manned units have a long, international history, both military and civilian. Some examples include SWAT teams, firefighters, the FBI, the DEA, the CIA, and military Special Operations. These special duty operators are individuals who perform a highly skilled and dangerous job in a unique environment. A significant amount of money is spent by the Department of Defense (DoD) and other federal agencies to recruit, select, train, equip and support these operators. When a critical incident or significant life event occurs, that jeopardizes an operator's performance; there can be heavy losses in terms of training, time, money, and potentially, lives. In order to limit the number of critical incidents, selection processes have been developed over time to “select out” those individuals most likely to perform below desired performance standards under pressure or stress and to "select in" those with the "right stuff". This study is part of a larger program evaluation to assess markers that identify whether a person will fail under the stresses in a selectively manned unit. The primary question of the study is whether there are indicators in the selection process that signify potential negative performance at a later date. ^ Methods. The population being studied included applicants to a selectively manned DoD organization between 1993 and 2001 as part of a unit assessment and selection process (A&S). Approximately 1900 A&S records were included in the analysis. Over this nine year period, seventy-two individuals were determined to have had a critical incident. A critical incident can come in the form of problems with the law, personal, behavioral or family problems, integrity issues, and skills deficit. Of the seventy-two individuals, fifty-four of these had full assessment data and subsequent supervisor performance ratings which assessed how an individual performed while on the job. This group was compared across a variety of variables including demographics and psychometric testing with a group of 178 individuals who did not have a critical incident and had been determined to be good performers with positive ratings by their supervisors.^ Results. In approximately 2004, an online pre-screen survey was developed in the hopes of preselecting out those individuals with items that would potentially make them ineligible for selection to this organization. This survey has aided the organization to increase its selection rates and save resources in the process. (Patterson, Howard Smith, & Fisher, Unit Assessment and Selection Project, 2008) When the same prescreen was used on the critical incident individuals, it was found that over 60% of the individuals would have been flagged as unacceptable. This would have saved the organization valuable resources and heartache.^ There were some subtle demographic differences between the two groups (i.e. those with critical incidents were almost twice as likely to be divorced compared with the positive performers). Upon comparison of Psychometric testing several items were noted to be different. The two groups were similar when their IQ levels were compared using the Multidimensional Aptitude Battery (MAB). When looking at the Minnesota Multiphasic Personality Inventory (MMPI), there appeared to be a difference on the MMPI Social Introversion; the Critical Incidence group scored somewhat higher. When analysis was done, the number of MMPI Critical Items between the two groups was similar as well. When scores on the NEO Personality Inventory (NEO) were compared, the critical incident individuals tended to score higher on Openness and on its subscales (Ideas, Actions, and Feelings). There was a positive correlation between Total Neuroticism T Score and number of MMPI critical items.^ Conclusions. This study shows that the current pre-screening process is working and would have saved the organization significant resources. ^ If one was to develop a profile of a candidate who potentially could suffer a critical incident and subsequently jeopardize the unit, mission and the safety of the public they would look like the following: either divorced or never married, score high on the MMPI in Social Introversion, score low on MMPI with an "excessive" amount of MMPI critical items; and finally scores high on the NEO Openness and subscales Ideas, Feelings, and Actions.^ Based on the results gleaned from the analysis in this study there seems to be several factors, within psychometric testing, that when taken together, will aid the evaluators in selecting only the highest quality operators in order to save resources and to help protect the public from unfortunate critical incidents which may adversely affect our health and safety.^