A comprehensive framework for evaluating funder's performance

Evaluation methods for assessing the performance of non-profit funders are lacking. The purpose of the research was to create a comprehensive framework that systematically assesses the goals and objectives of a funder, how these relate to the funder's allocation of resources, and the potential impact of programs and services selected by the funder for resource allocation to address organizational goals and objectives. The Houston Affiliate of Susan G. Komen for the Cure, a local chapter of a national breast cancer awareness advocacy organization, was selected as the funding agency whose performance assessment was to assist in the creation of this framework. Evaluation approaches from the government sector were adapted and incorporated into the research to guide the methods used to answer the three research questions corresponding to the three phases of research within the study: (1) what are the funding goals and objectives of the Affiliate?; (2) what allocation scheme does the organization use to address these goals and objectives and select programs for funding?; and, (3) to what extent do the programs funded by the Affiliate have potential long-term impact? ^ Within the first stage of the research, document reviews of the Affiliate's mission-based documents and bylaws and interviews with organizational and community informants revealed a highly latent constellation of broad objectives that were not formalized into one guiding document, thus creating gaps in management and governance. Within the second phase of the research, reviews of grant applications from the 2008-2009 funding cycle and interviews with employees and volunteers familiar with the funding process revealed competing ideas regarding resource allocation in light of vague organizational documents describing funding goals and objectives. Within the final stage of the research, these findings translated to the Affiliate selecting programs with highly varying potential long-term impact with regards to addressing goals and objectives relating to breast cancer education, screening, diagnostics, treatment, and support. The resulting performance assessment framework, consisting of three phases of research utilizing organizational documents and key informant interviews, demonstrated the importance of clearly defined funding goals and objectives, reference documents and committee participation within the funding process, and regular reviews of potential long-term impact for selected programs, all supported by the active participation and governance of a funder's Board of Directors.^

Evaluating the U.S. Global Health Policy Initiative

In understanding that the efforts made in improving global health affects the health of U.S. citizens, a policy analysis of President Barak Obama's Global Health Initiative was conducted. Using materials gathered from experts in the field of health and their findings and recommendations, paired with the current policies of other G8 countries that pledged to support the efforts of improving global health, the analysis was conducted using four specifically defined criteria. The set criteria determine the appropriateness, responsiveness, effectiveness and equity of Obama's GHI in comparison to other G8 country health policies and overall global health priorities. G8 countries without a specific global health policy, or with a policy that was not in English were excluded from this study and Switzerland, headquarters of the World Health Organization, was added due to its membership in the OECD, and the fact that it has a specific foreign health policy. In evaluating the U.S. Global Health Initiative it is clear that in terms of implementing foreign policy specific to health, the United States is on the forefront alongside the United Kingdom and Switzerland. Other G8 Countries have pledged monies and in order to Millennium Development Health Goals by 2015. The U.S. Global Health Policy does not address issues necessary to meet Millennium Development Goals in Health. Instead the Global Health Initiative is focused narrowly on Fighting and rolling back the HIV/Aids Epidemic based on President Bush's PEPFAR policy. Policy recommendations for a more effective and efficient Global Health Initiative include building upon the PEPFAR policy foundation in order to strengthen health systems worldwide, allowing individuals and communities to combat unnecessary death and disease through research, education, and other preventative methods.^

THE USE OF MULTIVARIATE ANALYSIS OF COVARIANCE IN EVALUATING THE COMPARABILITY OF LABORATORY DETERMINATIONS IN COOPERATIVE STUDIES

The role of clinical chemistry has traditionally been to evaluate acutely ill or hospitalized patients. Traditional statistical methods have serious drawbacks in that they use univariate techniques. To demonstrate alternative methodology, a multivariate analysis of covariance model was developed and applied to the data from the Cooperative Study of Sickle Cell Disease.^ The purpose of developing the model for the laboratory data from the CSSCD was to evaluate the comparability of the results from the different clinics. Several variables were incorporated into the model in order to control for possible differences among the clinics that might confound any real laboratory differences.^ Differences for LDH, alkaline phosphatase and SGOT were identified which will necessitate adjustments by clinic whenever these data are used. In addition, aberrant clinic values for LDH, creatinine and BUN were also identified.^ The use of any statistical technique including multivariate analysis without thoughtful consideration may lead to spurious conclusions that may not be corrected for some time, if ever. However, the advantages of multivariate analysis far outweigh its potential problems. If its use increases as it should, the applicability to the analysis of laboratory data in prospective patient monitoring, quality control programs, and interpretation of data from cooperative studies could well have a major impact on the health and well being of a large number of individuals. ^

Evaluating the Utility of Clinical Criteria for the Identification of Lynch Syndrome among Endometrial Cancer Patients

Background: Lynch Syndrome (LS) is a familial cancer syndrome with a high prevalence of colorectal and endometrial carcinomas among affected family members. Clinical criteria, developed from information obtained from familial colorectal cancer registries, have been generated to identify individuals at elevated risk for having LS. In 2007, the Society of Gynecologic Oncology (SGO) codified criteria to assist in identifying women presenting with gynecologic cancers at elevated risk for having LS. These criteria have not been validated in a population-based setting. Materials and Methods: We retrospectively identified 412, unselected endometrial cancer cases. Clinical and pathologic information were obtained from the electronic medical record, and all tumors were tested for expression of the DNA mismatch repair proteins through immunohistochemistry. Tumors exhibiting loss of MSH2, MSH6 and PMS2 were designated as probable Lynch Syndrome (PLS). For tumors exhibiting immunohistochemical loss of MLH1, we used the PCR-based MLH1 methylation assay to delineate PLS tumors from sporadic tumors. Samples lacking methylation of the MLH1 promoter were also designated as PLS. The sensitivity and specificity for SGO criteria for detecting PLS tumors was calculated. We compared clinical and pathologic features of sporadic tumors and PLS tumors. A simplified cost-effectiveness analysis was also performed comparing the direct costs of utilizing SGO criteria vs. universal tumor testing. Results: In our cohort, 43/408 (10.5%) of endometrial carcinomas were designated as PLS. The sensitivity and specificity of SGO criteria to identify PLS cases were 32.7 and 77%, respectively. Multivariate analysis of clinical and pathologic parameters failed to identify statistically significant differences between sporadic and PLS tumors with the exception of tumors arising from the lower uterine segment. These tumors were more likely to occur in PLS tumors. Cost-effectiveness analysis showed clinical criteria and universal testing strategies cost $6,235.27/PLS case identified and $5,970.38/PLS case identified, respectively. Conclusions: SGO 5-10% criteria successfully identify PLS cases among women who are young or have significant family history of LS related tumors. However, a larger proportion of PLS cases occurring at older ages with less significant family history are not detected by this screening strategy. Compared to SGO clinical criteria, universal tumor testing is a cost effective strategy to identify women presenting with endometrial cancer who are at elevated risk for having LS.

Evaluating the achievements and impacts of EC framework programme transport projects

Purpose The purpose of this paper is to present what kind of elements and evaluation methods should be included into a framework for evaluating the achievements and impacts of transport projects supported in EC Framework Programmes (FP). Further, the paper discusses the possibilities of such an evaluation framework in producing recommendations regarding future transport research and policy objectives as well as mutual learning for the basis of strategic long term planning. Methods The paper describes the two-dimensional evaluation methodology developed in the course of the FP7 METRONOME project. The dimensions are: (1) achievement of project objectives and targets in different levels and (2) research project impacts according to four impact groups. The methodology uses four complementary approaches in evaluation, namely evaluation matrices, coordinator questionnaires, lead user interviews and workshops. Results Based on the methodology testing, with a sample of FP5 and FP6 projects, the main results relating to the rationale, implementation and achievements of FP projects is presented. In general, achievement of objectives in both FPs was good. Strongest impacts were identified within the impact group of management and co-ordination. Also scientific and end-user impacts of the projects were adequate, but wider societal impacts quite modest. The paper concludes with a discussion both on the theoretical and practical implications of the proposed methodology and by presenting some relevant future research needs.

Comparison of methods for downscaling runoff from regional climate models in Spanish basins

At present there is much literature that refers to the advantages and disadvantages of different methods of statistical and dynamical downscaling of climate variables projected by climate models. Less attention has been paid to other indirect variables, like runoff, which play a significant role in evaluating the impact of climate change on hydrological systems. Runoff presents a much greater bias in climate models than other climate variables, like temperature or precipitation. It is very important to identify the methods that minimize bias while downscaling runoff from the gridded results of climate models to the basin scale

Evaluating social choice techniques into intelligent environments by agent based social simulation

The primary hypothesis stated by this paper is that the use of social choice theory in Ambient Intelligence systems can improve significantly users satisfaction when accessing shared resources. A research methodology based on agent based social simulations is employed to support this hypothesis and to evaluate these benefits. The result is a six-fold contribution summarized as follows. Firstly, several considerable differences between this application case and the most prominent social choice application, political elections, have been found and described. Secondly, given these differences, a number of metrics to evaluate different voting systems in this scope have been proposed and formalized. Thirdly, given the presented application and the metrics proposed, the performance of a number of well known electoral systems is compared. Fourthly, as a result of the performance study, a novel voting algorithm capable of obtaining the best balance between the metrics reviewed is introduced. Fifthly, to improve the social welfare in the experiments, the voting methods are combined with cluster analysis techniques. Finally, the article is complemented by a free and open-source tool, VoteSim, which ensures not only the reproducibility of the experimental results presented, but also allows the interested reader to adapt the case study presented to different environments.

Evaluating a web-based clinical decision support system for language disorders screening in a nursery school

Background: Early and effective identification of developmental disorders during childhood remains a critical task for the international community. The second highest prevalence of common developmental disorders in children are language delays, which are frequently the first symptoms of a possible disorder. Objective: This paper evaluates a Web-based Clinical Decision Support System (CDSS) whose aim is to enhance the screening of language disorders at a nursery school. The common lack of early diagnosis of language disorders led us to deploy an easy-to-use CDSS in order to evaluate its accuracy in early detection of language pathologies. This CDSS can be used by pediatricians to support the screening of language disorders in primary care. Methods: This paper details the evaluation results of the ?Gades? CDSS at a nursery school with 146 children, 12 educators, and 1 language therapist. The methodology embraces two consecutive phases. The first stage involves the observation of each child?s language abilities, carried out by the educators, to facilitate the evaluation of language acquisition level performed by a language therapist. Next, the same language therapist evaluates the reliability of the observed results. Results: The Gades CDSS was integrated to provide the language therapist with the required clinical information. The validation process showed a global 83.6% (122/146) success rate in language evaluation and a 7% (7/94) rate of non-accepted system decisions within the range of children from 0 to 3 years old. The system helped language therapists to identify new children with potential disorders who required further evaluation. This process will revalidate the CDSS output and allow the enhancement of early detection of language disorders in children. The system does need minor refinement, since the therapists disagreed with some questions from the CDSS knowledge base (KB) and suggested adding a few questions about speech production and pragmatic abilities. The refinement of the KB will address these issues and include the requested improvements, with the support of the experts who took part in the original KB development. Conclusions: This research demonstrated the benefit of a Web-based CDSS to monitor children?s neurodevelopment via the early detection of language delays at a nursery school. Current next steps focus on the design of a model that includes pseudo auto-learning capacity, supervised by experts.

Evaluating the quality of riparian forest vegetation: The Riparian Forest Evaluation (RFV) index

Aim of study: This paper presents a novel index, the Riparian Forest Evaluation (RFV) index, for assessing the ecological condition of riparian forests. The status of riparian ecosystems has global importance due to the ecological and social benefits and services they provide. The initiation of the European Water Framework Directive (2000/60/CE) requires the assessment of the hydromorphological quality of natural channels. The Directive describes riparian forests as one of the fundamental components that determine the structure of riverine areas. The RFV index was developed to meet the aim of the Directive and to complement the existing methodologies for the evaluation of riparian forests. Area of study: The RFV index was applied to a wide range of streams and rivers (170 water bodies) inSpain. Materials and methods: The calculation of the RFV index is based on the assessment of both the spatial continuity of the forest (in its three core dimensions: longitudinal, transversal and vertical) and the regeneration capacity of the forest, in a sampling area related to the river hydromorphological pattern. This index enables an evaluation of the quality and degree of alteration of riparian forests. In addition, it helps to determine the scenarios that are necessary to improve the status of riparian forests and to develop processes for restoring their structure and composition. Main results: The results were compared with some previous tools for the assessment of riparian vegetation. The RFV index got the highest average scores in the basins of northernSpain, which suffer lower human influence. The forests in central and southern rivers got worse scores. The bigger differences with other tools were found in complex and partially altered streams and rivers. Research highlights: The study showed the index’s applicability under diverse hydromorphological and ecological conditions and the main advantages of its application. The utilization of the index allows a better understanding of the status of riparian forests, and enhances improvements in the conservation and management of riparian areas.

Sources of selection bias in evaluating social programs: An interpretation of conventional measures and evidence on the effectiveness of matching as a program evaluation method

This paper decomposes the conventional measure of selection bias in observational studies into three components. The first two components are due to differences in the distributions of characteristics between participant and nonparticipant (comparison) group members: the first arises from differences in the supports, and the second from differences in densities over the region of common support. The third component arises from selection bias precisely defined. Using data from a recent social experiment, we find that the component due to selection bias, precisely defined, is smaller than the first two components. However, selection bias still represents a substantial fraction of the experimental impact estimate. The empirical performance of matching methods of program evaluation is also examined. We find that matching based on the propensity score eliminates some but not all of the measured selection bias, with the remaining bias still a substantial fraction of the estimated impact. We find that the support of the distribution of propensity scores for the comparison group is typically only a small portion of the support for the participant group. For values outside the common support, it is impossible to reliably estimate the effect of program participation using matching methods. If the impact of participation depends on the propensity score, as we find in our data, the failure of the common support condition severely limits matching compared with random assignment as an evaluation estimator.

A quantitative method for evaluating the stabilities of nucleic acids

We report a general method for screening, in solution, the impact of deviations from canonical Watson-Crick composition on the thermodynamic stability of nucleic acid duplexes. We demonstrate how fluorescence resonance energy transfer (FRET) can be used to detect directly free energy differences between an initially formed “reference” duplex (usually a Watson-Crick duplex) and a related “test” duplex containing a lesion/alteration of interest (e.g., a mismatch, a modified, a deleted, or a bulged base, etc.). In one application, one titrates into a solution containing a fluorescently labeled, FRET-active, reference duplex, an unlabeled, single-stranded nucleic acid (test strand), which may or may not compete successfully to form a new duplex. When a new duplex forms by strand displacement, it will not exhibit FRET. The resultant titration curve (normalized fluorescence intensity vs. logarithm of test strand concentration) yields a value for the difference in stability (free energy) between the newly formed, test strand-containing duplex and the initial reference duplex. The use of competitive equilibria in this assay allows the measurement of equilibrium association constants that far exceed the magnitudes accessible by conventional titrimetric techniques. Additionally, because of the sensitivity of fluorescence, the method requires several orders of magnitude less material than most other solution methods. We discuss the advantages of this method for detecting and characterizing any modification that alters duplex stability, including, but not limited to, mutagenic lesions. We underscore the wide range of accessible free energy values that can be defined by this method, the applicability of the method in probing for a myriad of nucleic acid variations, such as single nucleotide polymorphisms, and the potential of the method for high throughput screening.

VAR models and methods for monetary and health economics

Negli ultimi anni i modelli VAR sono diventati il principale strumento econometrico per verificare se può esistere una relazione tra le variabili e per valutare gli effetti delle politiche economiche. Questa tesi studia tre diversi approcci di identificazione a partire dai modelli VAR in forma ridotta (tra cui periodo di campionamento, set di variabili endogene, termini deterministici). Usiamo nel caso di modelli VAR il test di Causalità di Granger per verificare la capacità di una variabile di prevedere un altra, nel caso di cointegrazione usiamo modelli VECM per stimare congiuntamente i coefficienti di lungo periodo ed i coefficienti di breve periodo e nel caso di piccoli set di dati e problemi di overfitting usiamo modelli VAR bayesiani con funzioni di risposta di impulso e decomposizione della varianza, per analizzare l'effetto degli shock sulle variabili macroeconomiche. A tale scopo, gli studi empirici sono effettuati utilizzando serie storiche di dati specifici e formulando diverse ipotesi. Sono stati utilizzati tre modelli VAR: in primis per studiare le decisioni di politica monetaria e discriminare tra le varie teorie post-keynesiane sulla politica monetaria ed in particolare sulla cosiddetta "regola di solvibilità" (Brancaccio e Fontana 2013, 2015) e regola del GDP nominale in Area Euro (paper 1); secondo per estendere l'evidenza dell'ipotesi di endogeneità della moneta valutando gli effetti della cartolarizzazione delle banche sul meccanismo di trasmissione della politica monetaria negli Stati Uniti (paper 2); terzo per valutare gli effetti dell'invecchiamento sulla spesa sanitaria in Italia in termini di implicazioni di politiche economiche (paper 3). La tesi è introdotta dal capitolo 1 in cui si delinea il contesto, la motivazione e lo scopo di questa ricerca, mentre la struttura e la sintesi, così come i principali risultati, sono descritti nei rimanenti capitoli. Nel capitolo 2 sono esaminati, utilizzando un modello VAR in differenze prime con dati trimestrali della zona Euro, se le decisioni in materia di politica monetaria possono essere interpretate in termini di una "regola di politica monetaria", con specifico riferimento alla cosiddetta "nominal GDP targeting rule" (McCallum 1988 Hall e Mankiw 1994; Woodford 2012). I risultati evidenziano una relazione causale che va dallo scostamento tra i tassi di crescita del PIL nominale e PIL obiettivo alle variazioni dei tassi di interesse di mercato a tre mesi. La stessa analisi non sembra confermare l'esistenza di una relazione causale significativa inversa dalla variazione del tasso di interesse di mercato allo scostamento tra i tassi di crescita del PIL nominale e PIL obiettivo. Risultati simili sono stati ottenuti sostituendo il tasso di interesse di mercato con il tasso di interesse di rifinanziamento della BCE. Questa conferma di una sola delle due direzioni di causalità non supporta un'interpretazione della politica monetaria basata sulla nominal GDP targeting rule e dà adito a dubbi in termini più generali per l'applicabilità della regola di Taylor e tutte le regole convenzionali della politica monetaria per il caso in questione. I risultati appaiono invece essere più in linea con altri approcci possibili, come quelli basati su alcune analisi post-keynesiane e marxiste della teoria monetaria e più in particolare la cosiddetta "regola di solvibilità" (Brancaccio e Fontana 2013, 2015). Queste linee di ricerca contestano la tesi semplicistica che l'ambito della politica monetaria consiste nella stabilizzazione dell'inflazione, del PIL reale o del reddito nominale intorno ad un livello "naturale equilibrio". Piuttosto, essi suggeriscono che le banche centrali in realtà seguono uno scopo più complesso, che è il regolamento del sistema finanziario, con particolare riferimento ai rapporti tra creditori e debitori e la relativa solvibilità delle unità economiche. Il capitolo 3 analizza l’offerta di prestiti considerando l’endogeneità della moneta derivante dall'attività di cartolarizzazione delle banche nel corso del periodo 1999-2012. Anche se gran parte della letteratura indaga sulla endogenità dell'offerta di moneta, questo approccio è stato adottato raramente per indagare la endogeneità della moneta nel breve e lungo termine con uno studio degli Stati Uniti durante le due crisi principali: scoppio della bolla dot-com (1998-1999) e la crisi dei mutui sub-prime (2008-2009). In particolare, si considerano gli effetti dell'innovazione finanziaria sul canale dei prestiti utilizzando la serie dei prestiti aggiustata per la cartolarizzazione al fine di verificare se il sistema bancario americano è stimolato a ricercare fonti più economiche di finanziamento come la cartolarizzazione, in caso di politica monetaria restrittiva (Altunbas et al., 2009). L'analisi si basa sull'aggregato monetario M1 ed M2. Utilizzando modelli VECM, esaminiamo una relazione di lungo periodo tra le variabili in livello e valutiamo gli effetti dell’offerta di moneta analizzando quanto la politica monetaria influisce sulle deviazioni di breve periodo dalla relazione di lungo periodo. I risultati mostrano che la cartolarizzazione influenza l'impatto dei prestiti su M1 ed M2. Ciò implica che l'offerta di moneta è endogena confermando l'approccio strutturalista ed evidenziando che gli agenti economici sono motivati ad aumentare la cartolarizzazione per una preventiva copertura contro shock di politica monetaria. Il capitolo 4 indaga il rapporto tra spesa pro capite sanitaria, PIL pro capite, indice di vecchiaia ed aspettativa di vita in Italia nel periodo 1990-2013, utilizzando i modelli VAR bayesiani e dati annuali estratti dalla banca dati OCSE ed Eurostat. Le funzioni di risposta d'impulso e la scomposizione della varianza evidenziano una relazione positiva: dal PIL pro capite alla spesa pro capite sanitaria, dalla speranza di vita alla spesa sanitaria, e dall'indice di invecchiamento alla spesa pro capite sanitaria. L'impatto dell'invecchiamento sulla spesa sanitaria è più significativo rispetto alle altre variabili. Nel complesso, i nostri risultati suggeriscono che le disabilità strettamente connesse all'invecchiamento possono essere il driver principale della spesa sanitaria nel breve-medio periodo. Una buona gestione della sanità contribuisce a migliorare il benessere del paziente, senza aumentare la spesa sanitaria totale. Tuttavia, le politiche che migliorano lo stato di salute delle persone anziane potrebbe essere necessarie per una più bassa domanda pro capite dei servizi sanitari e sociali.

Mobility of the elderly and handicapped: a methodology for evaluating the effectiveness of transportation improvements for the elderly and handicapped. Final report.

Transportation Department, Office of University Research, Washington, D.C.

Field test plan for evaluating the cooperative breath analyzer. Technical report.

National Highway Traffic Safety Administration, Washington, D.C.

Evaluation of design analysis procedures and acceptance criteria for roadside hardware. Volume III - evaluating pre-report 230 crash tests. Final report.

Federal Highway Administration, Safety Design Division, McLean, Va.