Evaluation Framework EFI for Measuring the Impact of Learning, Education and Training

This article introduces the Evaluation Framework EFI for the Impact Measurement of learning, education and training: The Evaluation Framework for Impact Measurement was developed for specifying the evaluation phase and its objectives and tasks within the IDEAL Reference Model for the introduction and optimization of quality development within learning, education and training. First, a description of the Evaluation Framework for Impact Measurement will be provided, followed by a brief overview of the IDEAL Reference Model. Finally, an example for the implementation of the Evaluation Framework for Impact Measurement within the ARISTOTELE project is presented.

Community Policing in Switzerland's Major Urban Areas: An Observational Study of the Implementation and Impact Using Geospatial Data Mining

La présente étude est à la fois une évaluation du processus de la mise en oeuvre et des impacts de la police de proximité dans les cinq plus grandes zones urbaines de Suisse - Bâle, Berne, Genève, Lausanne et Zurich. La police de proximité (community policing) est à la fois une philosophie et une stratégie organisationnelle qui favorise un partenariat renouvelé entre la police et les communautés locales dans le but de résoudre les problèmes relatifs à la sécurité et à l'ordre public. L'évaluation de processus a analysé des données relatives aux réformes internes de la police qui ont été obtenues par l'intermédiaire d'entretiens semi-structurés avec des administrateurs clés des cinq départements de police, ainsi que dans des documents écrits de la police et d'autres sources publiques. L'évaluation des impacts, quant à elle, s'est basée sur des variables contextuelles telles que des statistiques policières et des données de recensement, ainsi que sur des indicateurs d'impacts construit à partir des données du Swiss Crime Survey (SCS) relatives au sentiment d'insécurité, à la perception du désordre public et à la satisfaction de la population à l'égard de la police. Le SCS est un sondage régulier qui a permis d'interroger des habitants des cinq grandes zones urbaines à plusieurs reprises depuis le milieu des années 1980. L'évaluation de processus a abouti à un « Calendrier des activités » visant à créer des données de panel permettant de mesurer les progrès réalisés dans la mise en oeuvre de la police de proximité à l'aide d'une grille d'évaluation à six dimensions à des intervalles de cinq ans entre 1990 et 2010. L'évaluation des impacts, effectuée ex post facto, a utilisé un concept de recherche non-expérimental (observational design) dans le but d'analyser les impacts de différents modèles de police de proximité dans des zones comparables à travers les cinq villes étudiées. Les quartiers urbains, délimités par zone de code postal, ont ainsi été regroupés par l'intermédiaire d'une typologie réalisée à l'aide d'algorithmes d'apprentissage automatique (machine learning). Des algorithmes supervisés et non supervisés ont été utilisés sur les données à haute dimensionnalité relatives à la criminalité, à la structure socio-économique et démographique et au cadre bâti dans le but de regrouper les quartiers urbains les plus similaires dans des clusters. D'abord, les cartes auto-organisatrices (self-organizing maps) ont été utilisées dans le but de réduire la variance intra-cluster des variables contextuelles et de maximiser simultanément la variance inter-cluster des réponses au sondage. Ensuite, l'algorithme des forêts d'arbres décisionnels (random forests) a permis à la fois d'évaluer la pertinence de la typologie de quartier élaborée et de sélectionner les variables contextuelles clés afin de construire un modèle parcimonieux faisant un minimum d'erreurs de classification. Enfin, pour l'analyse des impacts, la méthode des appariements des coefficients de propension (propensity score matching) a été utilisée pour équilibrer les échantillons prétest-posttest en termes d'âge, de sexe et de niveau d'éducation des répondants au sein de chaque type de quartier ainsi identifié dans chacune des villes, avant d'effectuer un test statistique de la différence observée dans les indicateurs d'impacts. De plus, tous les résultats statistiquement significatifs ont été soumis à une analyse de sensibilité (sensitivity analysis) afin d'évaluer leur robustesse face à un biais potentiel dû à des covariables non observées. L'étude relève qu'au cours des quinze dernières années, les cinq services de police ont entamé des réformes majeures de leur organisation ainsi que de leurs stratégies opérationnelles et qu'ils ont noué des partenariats stratégiques afin de mettre en oeuvre la police de proximité. La typologie de quartier développée a abouti à une réduction de la variance intra-cluster des variables contextuelles et permet d'expliquer une partie significative de la variance inter-cluster des indicateurs d'impacts avant la mise en oeuvre du traitement. Ceci semble suggérer que les méthodes de géocomputation aident à équilibrer les covariables observées et donc à réduire les menaces relatives à la validité interne d'un concept de recherche non-expérimental. Enfin, l'analyse des impacts a révélé que le sentiment d'insécurité a diminué de manière significative pendant la période 2000-2005 dans les quartiers se trouvant à l'intérieur et autour des centres-villes de Berne et de Zurich. Ces améliorations sont assez robustes face à des biais dus à des covariables inobservées et covarient dans le temps et l'espace avec la mise en oeuvre de la police de proximité. L'hypothèse alternative envisageant que les diminutions observées dans le sentiment d'insécurité soient, partiellement, un résultat des interventions policières de proximité semble donc être aussi plausible que l'hypothèse nulle considérant l'absence absolue d'effet. Ceci, même si le concept de recherche non-expérimental mis en oeuvre ne peut pas complètement exclure la sélection et la régression à la moyenne comme explications alternatives. The current research project is both a process and impact evaluation of community policing in Switzerland's five major urban areas - Basel, Bern, Geneva, Lausanne, and Zurich. Community policing is both a philosophy and an organizational strategy that promotes a renewed partnership between the police and the community to solve problems of crime and disorder. The process evaluation data on police internal reforms were obtained through semi-structured interviews with key administrators from the five police departments as well as from police internal documents and additional public sources. The impact evaluation uses official crime records and census statistics as contextual variables as well as Swiss Crime Survey (SCS) data on fear of crime, perceptions of disorder, and public attitudes towards the police as outcome measures. The SCS is a standing survey instrument that has polled residents of the five urban areas repeatedly since the mid-1980s. The process evaluation produced a "Calendar of Action" to create panel data to measure community policing implementation progress over six evaluative dimensions in intervals of five years between 1990 and 2010. The impact evaluation, carried out ex post facto, uses an observational design that analyzes the impact of the different community policing models between matched comparison areas across the five cities. Using ZIP code districts as proxies for urban neighborhoods, geospatial data mining algorithms serve to develop a neighborhood typology in order to match the comparison areas. To this end, both unsupervised and supervised algorithms are used to analyze high-dimensional data on crime, the socio-economic and demographic structure, and the built environment in order to classify urban neighborhoods into clusters of similar type. In a first step, self-organizing maps serve as tools to develop a clustering algorithm that reduces the within-cluster variance in the contextual variables and simultaneously maximizes the between-cluster variance in survey responses. The random forests algorithm then serves to assess the appropriateness of the resulting neighborhood typology and to select the key contextual variables in order to build a parsimonious model that makes a minimum of classification errors. Finally, for the impact analysis, propensity score matching methods are used to match the survey respondents of the pretest and posttest samples on age, gender, and their level of education for each neighborhood type identified within each city, before conducting a statistical test of the observed difference in the outcome measures. Moreover, all significant results were subjected to a sensitivity analysis to assess the robustness of these findings in the face of potential bias due to some unobserved covariates. The study finds that over the last fifteen years, all five police departments have undertaken major reforms of their internal organization and operating strategies and forged strategic partnerships in order to implement community policing. The resulting neighborhood typology reduced the within-cluster variance of the contextual variables and accounted for a significant share of the between-cluster variance in the outcome measures prior to treatment, suggesting that geocomputational methods help to balance the observed covariates and hence to reduce threats to the internal validity of an observational design. Finally, the impact analysis revealed that fear of crime dropped significantly over the 2000-2005 period in the neighborhoods in and around the urban centers of Bern and Zurich. These improvements are fairly robust in the face of bias due to some unobserved covariate and covary temporally and spatially with the implementation of community policing. The alternative hypothesis that the observed reductions in fear of crime were at least in part a result of community policing interventions thus appears at least as plausible as the null hypothesis of absolutely no effect, even if the observational design cannot completely rule out selection and regression to the mean as alternative explanations.

Evaluation of the agricultural vocational training programmes conducted by the Krishi Vigyan Kendras (Farm Science Centres) in Indian Punjab

Krishin Vigyan Kendras-KVKs (Farm Science Centres) have been established by the Indian Council of Agricultural Research in 569 districts. The trust areas of KVKs are refinement and demonstration of technologies, and training of farmers and extension functionaries. Imparting vocational trainings in agriculture and allied fields for the rural youth is one of its mandates. The study was undertaken to do a formative and summative (outcome and impact) evaluation of the beekeeping and mushroom growing vocational training programmes in the Indian state of Punjab. One-group pre and post evaluation design was employed for conducting a formative and outcome evaluation. The knowledge tests were administered to 35 beekeeping and 25 mushroom cultivation trainees, before and after the training programmes organized in 2004. The trainees significantly gained in knowledge. A separate sample of 640 trainees, trained prior to 2004, was selected for finding the adoption status. Out of 640, a sample of 200 was selected by proportionate sampling technique out of three categories, namely: non-adopters, discontinued-adopters and continued-adopters for evaluating the long-term impact of these training programmes. Ex-post-facto one-shot case study design was applied for this impact analysis. The vocational training programmes have resulted in continued-adoption of beekeeping and mushroom cultivation enterprises by 20% and 51% trained farmers, respectively. Age and trainee occupation had significant influence on the adoption decision of beekeeping vocation, whereas education and family income significantly affected the adoption decision of mushroom cultivation. The continued adopters of beekeeping and mushroom growing had increased their family income by 49% and 24%, respectively. These training programmes are augmenting the dwindling farm income of the farmers in Indian Punjab.

Performance of small and medium enterprises and the impact of environmental variables: evidence from Vietnam

thesis is developed from a real life application of performance evaluation of small and medium-sized enterprises (SMEs) in Vietnam. The thesis presents two main methodological developments on evaluation of dichotomous environment variable impacts on technical efficiency. Taking into account the selection bias the thesis proposes a revised frontier separation approach for the seminal Data Envelopment Analysis (DEA) model which was developed by Charnes, Cooper, and Rhodes (1981). The revised frontier separation approach is based on a nearest neighbour propensity score matching pairing treated SMEs with their counterfactuals on the propensity score. The thesis develops order-m frontier conditioning on propensity score from the conditional order-m approach proposed by Cazals, Florens, and Simar (2002), advocated by Daraio and Simar (2005). By this development, the thesis allows the application of the conditional order-m approach with a dichotomous environment variable taking into account the existence of the self-selection problem of impact evaluation. Monte Carlo style simulations have been built to examine the effectiveness of the aforementioned developments. Methodological developments of the thesis are applied in empirical studies to evaluate the impact of training programmes on the performance of food processing SMEs and the impact of exporting on technical efficiency of textile and garment SMEs of Vietnam. The analysis shows that training programmes have no significant impact on the technical efficiency of food processing SMEs. Moreover, the analysis confirms the conclusion of the export literature that exporters are self selected into the sector. The thesis finds no significant impact from exporting activities on technical efficiency of textile and garment SMEs. However, large bias has been eliminated by the proposed approach. Results of empirical studies contribute to the understanding of the impact of different environmental variables on the performance of SMEs. It helps policy makers to design proper policy supporting the development of Vietnamese SMEs.

Almetrics: measuring the influence of the media on the social impact of research

Social media is changing the way we interact, present ideas and information and judge the quality of content and contributions. In recent years there have been hundreds of platforms to freely share all kinds of information and connect across networks. These new tools generate activity statistics and interactions among users such as mentions, retweets, conversations, comments on blogs or Facebook; managers references showing popularity ratings of more references shared by other researchers or repositories that generate statistics of visits or downloads of articles. This paper analyzes that have meaning and implications altmetrics, what are its advantages and critical platforms (Almetric.com, ImpactStory, Plos altmetrics, PlumX), reports progress and benefits for authors, publishers and librarians. It concluded that the value of alternative metrics as a complementary tool citation analysis is evident, although it is suggested that you should dig deeper into this issue to unravel the meaning and the potential value of these indicators to assess their potential.

Los desaf??os de las pol??ticas y la gesti??n social en Am??rica Latina

La pobreza est?? aumentando en t??rminos absolutos de manera sistem??tica desde 1990 y, desde comienzos del nuevo siglo, tambi??n en valores relativos. Hay, sin embargo, una buena nueva: el gasto p??blico social ha aumentado tambi??n de manera sistem??tica desde entonces. ??Qu?? se ha hecho con la enorme magnitud de recursos destinada a financiar las pol??ticas sociales? Hay, por un lado, limitaciones que surgen de la forma en que ellas se dise??an e implementan. La gesti??n social tradicional asume que el impacto perseguido se producir?? autom??ticamente como resultado de la mera entrega de los bienes o servicios a la poblaci??n destinataria. Aparece, adem??s, la masiva introducci??n de mecanismos de mercado en las pol??ticas sociales, bajo el supuesto que permitir??an mejorar su eficiencia e impacto. Teniendo en cuenta las limitaciones precedentes, se sugieren en este art??culo tres senderos, estos son: la superaci??n del estilo tradicional de gestionar los programas y proyectos sociales; el aprendizaje por medio de la experiencia de la evaluaci??n de impacto y del monitoreo; y la generaci??n de un estilo m??s participativo en el proceso de dise??o, gesti??n y evaluaci??n de las pol??ticas sociales.

Programas de cuidados integrados para pessoas com esquizofrenia ou perturbação esquizoafetiva : estudo sobre a exequibilidade, implementação e resultados de um programa de cuidados integrados para pessoas com esquizofrenia ou perturbação esquizoafectiva em Portugal

RESUMO: De acordo com o estado da arte, existem intervenções psicofarmacológicas, psicológicas e psicossocias, com evidência científica dos seus resultados, no tratamento de pessoas com esquizofrenia e perturbação esquizoafectiva. No entanto, muitos destes doentes, não procuram ajuda dos serviços de saúde mental, não recebem os referidos cuidados ou não são detectados nem seguidos por estes. Esta realidade levou ao desenvolvimento de programas integrados, intervenções e estudos mais específicos, nomeadamente para tentar ultrapassar os obstáculos na acessibilidade aos cuidados de saúde e na continuidade de seguimento destes doentes. No conjunto das dificuldades apuradas, as questões da exequibilidade (feasibility) e da implementação, têm tido particular relevo na literatura científica recente, bem como a melhor forma de vencer as respectivas barreiras e adaptar essas intervenções às varias realidades, culturas e recursos. Objectivos: Objectivos gerais:1) Avaliar a exequibilidade e a implementação inicial de um programa de cuidados integrados, para pessoas com esquizofrenia ou perturbação esquizoafectiva, no contexto clínico das equipas de saúde mental comunitárias de um departamento de psiquiatria do Serviço Nacional de Saúde, em Portugal, com os recursos materiais e humanos existentes; 2) Avaliar o impacto deste programa, nestes doentes e na respectiva prestação de cuidados de saúde mental. Metodologia. Elaborámos um programa de cuidados integrados (Programa Integrar) com base no modelo clínico de case management, com seguimento mantido e integrado. Cada doente passou a ter um terapeuta de referência, um plano individual de cuidados e manteve o seguimento com o seu psiquiatra assistente. Foram seleccionadas intervenções, nomeadamente, psicoeducativas, familiares, estratégias para lidar com os sintomas e a doença, prevenção de recaídas e intervenções para melhorar o funcionamento social e ocupacional. A estas intervenções foi sempre associado o tratamento psicofarmacológico. O estudo delineado incluiu dois componentes: avaliação da exequibilidade e implementação inicial do programa de cuidados integrados (componente A) e avaliação do impacto deste programa (componente B), através de um estudo de intervenção, prospectivo, naturalista, não aleatorizado e não ontrolado. A amostra do estudo resultou das sucessivas referenciações, para o Programa Integrar, de pessoas com os diagnósticos de esquizofrenia ou perturbação esquizoafectiva, seguidas nas cinco equipas de saúde mental comunitárias do Departamento de Psiquiatria do Centro Hospitalar de Lisboa Ocidental, com uma área assistencial correspondente a uma população de, aproximadamente, 400 000 pessoas. Definimos etapas, estratégias, parâmetros e indicadores para o estudo da exequibilidade do programa. Efectuámos a monitorização e a avaliação de tarefas, procedimentos e intervenções recomendadas aos terapeutas de referência. Realizámos duas avaliações, uma no início do programa e outra após um ano de intervenção. Foram avaliadas as seguintes dimensões (com indicação do acrónimo do instrumento de avaliação utilizado entre parêntesis): psicopatologia (BPRS), depressão (MADRS), necessidades (CAN), incapacidade (DAS), actividade social e ocupacional (SOFAS), atitude em relação à medicação (DAI), insight (SAI), qualidade de vida (WHOQOL-S) e satisfação (POCS). Resultados: Dos 146 doentes que foram incluídos no estudo, 97 (66%) eram do sexo masculino e 49 (34%) do sexo feminino, com uma idade média de 36 anos. Destes oentes,116 (79,4%)tinham o diagnóstico (ICD10) de esquizofrenia e 30 (20,6%) de perturbação esquizoafectiva. Os restantes dados sociodemográficos eram típicos de populações afins em serviços de saúde mental nacionais. Do total de doentes (146) que iniciaram o estudo, 26 (18%) abandonaram o seguimento neste programa. Para o componente A da investigação (estudo de exequibilidade) salientamos: exerceram funções a totalidade (15) dos terapeutas de referência que receberam formação, 76 % efectuaram o número mínimo recomendado de sessões / ano por doente (≥18), 44,9 fizeram o número mínimo de sessões familiares pretendido (≥ 3). Nas intervenções mais específicas foram atingidos os objectivos em mais de 75% dos doentes, à excepção das intervenções domiciliárias (19,4%), prevenção do abuso de substâncias (45,4%) e do risco de suicídio (34,3%). O plano individual de cuidados foi realizado em 98 % dos doentes e em 38,9 % dos casos ocorreu a participação da família. Neste plano, a média de objectivos definidos foi de 5 e a média de objectivos atingidos correspondeu a 3 (p= 0,001). Na primeira avaliação, estavam a frequentar estruturas de reabilitação psicossocial 42 doentes (28,8%) e,12 meses após, esse número passou para 80 (74,1%).Também aumentou o número de doentes com actividade profissional a tempo completo, de 8 (7,4%) para 18 (16,7%). No componente B do estudo (avaliação do impacto do programa), em termos de psicopatologia, e para as pontuações médias globais do BPRS, ocorreu uma diminuição entre a primeira e a segunda avaliação (p=0,001), tal como nas subescalas: sintomas positivos (p=0,003), sintomas negativos (p=0,002), sintomas de mania (p=0,002) e sintomas de depressão/ansiedade (p=0,001). Na avaliação da depressão (p= 0,001) e da incapacidade (p=0,003), as diferenças foram significativas e favoráveis. O mesmo não sucedeu na atitude em relação à medicação (p=0,690) nem na escala de avaliação do insight (p=0,079). Em relação ao funcionamento social e ocupacional, qualidade de vida e satisfação dos doentes, ocorreu uma melhoria significativa da primeira para a segunda avaliação As necessidades sem resposta mais frequentes, na primeira avaliação, corresponderam aos itens: actividades diárias, contactos sociais, relações íntimas, relacionamento sexual, benefícios sociais, sintomas psicóticos, sofrimento psicológico, informação sobre a doença / tratamento e gestão/problemas de dinheiro. Para todos estes últimos nove itens, verificou-se uma diferença estatisticamente significativa, entre a primeira e a segunda avaliação, com diminuição destas necessidades, excepto nas relações íntimas, relacionamento sexual e nos problemas de dinheiro. Na distribuição dos três estados de necessidades, para todos os itens, diminuíram as necessidades sem resposta e as necessidades com resposta parcial e aumentaram as situações em que deixaram de se verificar necessidades relevantes. Dos resultados obtidos para outros indicadores clínicos e de utilização dos cuidados, será importante referir que na comparação do ano anterior com o ano em que decorreu o programa, o número de doentes da amostra internados diminuiu 64,1%, bem como a média do número de internamentos (p=0,001). Em relação à duração dos internamentos, no ano anterior ao programa, os 39 doentes internados, tiveram um total de dias de internamento de 1522, sendo que, no ano do programa, para os 14 doentes internados, o total foi de 523 dias. Em termos absolutos, ocorreu uma redução de 999 dias (menos 65,6% dias). Também se verificou uma diminuição de 45,6 % de recaídas (p=0,001).Discussão e conclusões A exequibilidade do programa de cuidados integrados permitiu a aplicação do modelo clínico de case management, com seguimento mantido e integrado, através do qual cada doente passou a ter um terapeuta de referência assim como, em 98% casos, um plano individual de cuidados. As famílias continuaram a ser o principal suporte para os doentes, mas surgiram dificuldades quando se pretendeu uma participação mais activa destas no tratamento.A diminuição do número e da duração dos internamentos constituíram importantes resultados com implicações não só em termos clínicos mas também económicos. Os valores obtidos, para as diferentes variáveis, também sugerem o impacto favorável do Programa Integrar a nível da psicopatologia, das necessidades, da incapacidade, do funcionamento social e ocupacional, da qualidade de vida e da satisfação dos doentes. O mesmo não sucedeu para o insight e para a mudança de atitudes dos doentes em relação à medicação, resultados que devem ser igualmente considerados em futuros reajustamentos deste programa ou no desenvolvimento de novos programas. Como principais conclusões podemos referir que: 1) Foi possível a exequibilidade de um programa de cuidados integrados inovador e a implementação inicial desse programa, para doentes com esquizofrenia ou perturbação esquizoafectiva, com os recursos humanos e materiais existentes, no contexto clínico das equipas de saúde mental comunitárias, de um departamento de psiquiatria e saúde mental, em Portugal; 2) Na avaliação do impacto do programa, os resultados obtidos indiciam potencialidades de aplicação, deste programa de cuidados integrados, com vista à melhoria clínica e psicossocial destes doentes. Devem ser realizados estudos de replicação, ou complementares à presente investigação, no entanto, os dados obtidos são encorajadores para o desenvolvimento de programas similares, a nível nacional e internacional, que possam beneficiar um grupo mais alargado de doentes.------------ABSTRACT: Although there are psychological and psychosocial interventions well supported by scientific evidence, which show benefit when combined with psychopharmacological treatments, we know that a significant number of people with schizophrenia or schizoaffective disorders, do not seek help from mental health services, do not receive the care mentioned and are not detected or followed-up by them. This reality led to the development of integrated programs, interventions and more specific studies, to try to overcome the obstacles in the accessibility to the health services and on the follow-up of these patients. Amongst the barriers identified, feasibility and implementation of those programs have been of special relevance in recent scientific literature, as well as the best way to overcome such difficulties and adapt the interventions to the various realities, cultures and resources. Objectives: General objectives were defined: 1) Assessment of the feasibility and initial implementation of an integrated care program, for people with schizophrenia or schizoaffective disorder, in the clinical setting of community mental health teams, in a psychiatric department from the national health service in Portugal; 2) Impact evaluation of the integrated care program, for these patients and their mental health care delivery. Methods: We drew up an integrated care program (Program Integrar) based on the clinical case management model, with continuous and integrated follow-up. Each patient got one case manager, an individual care plan and kept the same psychiatrist. Were selected the appropriated interventions, namely: psycho-educative, family-based interventions, strategies for dealing with the symptoms and the disorder, relapse prevention and interventions to improve social and occupational functioning. These interventions were always associated with psychopharmacological treatment. The investigation was outline with two parts: assessment of the feasibility and initial implementation of the Program Integrar (part A of the study) and impact evaluation of the program (part B of the study). We designed a naturalistic, prospective, intervention study, non-randomized and without control group. Our chosen sample was made with successive referrals of patients with the diagnosis of schizophrenia or schizoaffective disorder, followedup in one of the five community mental health teams of the Psychiatric Department of Centro Hospitalar Lisboa Ocidental, with a catchment area for a population of about 400 000 people. Different stages, strategies, criteria and indicators for studying the feasibility of the program and its implementation were set and the tasks, procedures and recommended interventions of the case managers were monitored and evaluated. We did two assessments with an interval of one year and we evaluated the following dimensions (the acronym of the assessment instrument used in brackets): psychopathology (BPRS), depression (MADRS), needs (CAN), disability (DAS), social and occupational functioning (SOFAS), attitude toward medication (DAI), insight (SAI), quality of life (WHOQOL-S) and satisfaction (POCS). Results: Of the 146 patients who started the study, 97 (66%) were male and 49 (34%) females with a mean age of 36 years. Of these, 116 (79,4%) were diagnosed (ICD10) with schizophrenia and 30 (20,6%) with schizoaffective disorder. The other socio-demographic data were typical of populations within Portuguese mental health services. Of all patients (146), who started the program, 26 (18%) of patients left the program (program dropout rate). Of the regarding part A of the study, which focused on feasibility, the following is of note: all professionals who had been trained for this purpose (15) acted as case manager, 76% did the recommended minimum number of sessions / year per patient (≥18) and 44,9% did the minimum number of family sessions desired (≥ 3). For the more specific interventions the parameters set out were met for more than 75% of patients, with the exception of domiciliar interventions (19.4%), prevention of substance abuse (45.4%) and suicide risk prevention(34.3%). The individual care plan was done for 98% of patients and in 38,9% of cases this involved family participation. For this plan the mean objectives defined were 5 and in average was achieved 3 (p=0,001). On the first assessment, 42 patients (28.8%) were attending psychosocial rehabilitation structures and 12 months later that number rose up to 80 (74,1%). Regarding their employment status, in the first assessment 8 (7,4%) were in full time employment and in the second evaluation the number rise to 18 (16,7%). For part B of the study (impact program evaluation), in terms of psychopathology, global mean scores for the BPRS, decreased (p=0,001), as did the four sub scales: positive symptoms (p=0,003); negative symptoms (p=0,002); manic symptoms (p=0,002) and symptoms of depression/anxiety (p=0,001). Both in the evaluation of depression (p=0,001), as in the assessment of disability (p=0,003), the differences were significant. However, this was not the case with attitudes towards medication (p=0,690) and with insight evaluation (p=0,079). In relation to social and occupational functioning, quality of life and patient satisfaction there was a statistically significant improvement from the first to the second assessment. The most commonly unmet needs in the first assessment were daily activities, social contacts, intimate relationships, sexual relations, social benefits, psychotic symptoms,psychological distress, information about the disorder / treatment and money problems money management. Of these, in the second assessment, all of those nine unmet needs showed significant improvement, excepted intimate relationships, sexual relations and Money problems / money management. In the distribution of the three states of needs for all items, it happened a decreased in unmet needs and partially met needs and increased in the situations where relevant needs were no longer found. For other clinical indicators it is important to note, when we compared the year prior to this program and the year after, there were fewer hospitalizations (reduction of 64,1% of admissions) and in the mean number of admissions (p=0,001). Regarding the length of hospitalization in the year prior to the program, the 39 patients admitted had a total of 1522 hospital days, and in the year of the program for the 14 hospitalized patients, the total was 523 days. In absolute terms, there was a reduction of 999 days (65,6%). There was also a 45,6% reduction of relapses (p = 0,001). Discussion and Conclusions: The feasibility of the integrated care program allowed the application of the clinical case management model, with continuous follow-up. Each patient got a case manager and in 98% of the cases they also got an individual plan of care. Families continued to be the main support for patients but, difficulties occurred when it was claimed a more active participation. The decrease in the number and duration of admissions were important findings with implications not only in clinical terms but also in economic field. The achieved results for the different variables can also indicate the favorable impact of this program, at the level of psychopathology, needs, disability, social and occupationa functioning, quality of life and patient satisfaction. The same did not happen for the evaluation of insight and in the changes of attitudes towards medication. These data should also be considered for future readjustments of this program and for the developing of new programs.Finally, the two-overview conclusions are: 1) It was possible the feasibility of an integrated care program and initial implementation of this innovative program, for patients with schizophrenia or schizoaffective disorder, with the human and material resources available in the clinical context of the community mental health teams, in a psychiatry and mental health department of the national health service in Portugal; 2) In assessing the impact of the program, the results suggest potential application of this integrated care program, to improve clinical state and psychosocial variables for these patients. There should be done studies to replicate these results, however the results obtained are promising for the development of similar programs at nationally and internationally level, that could benefit a wider group of patients.

Avaluació de l'aprofitament de biomassa disponible per a la producció d'energia calorífica al Parc Natural de l'Alt Pirineu

En aquest article es resumeixen els resultats més destacats del projecte dut a terme pel grup Projecte Biomassa. Aquest estudi té per objectiu l’avaluació de la viabilitat energètica, econòmica i ambiental de l’aprofitament de la biomassa forestal com a recurs energètic a les forests d’Arestui, Baiasca, Montenartró i Virós situades dins el Parc Natural de l’Alt Pirineu (PNAP). En el projecte s’ha fet una revisió dels Plans d’Ordenació Forestal (POF’s) de les esmentades forests per tal d’adequar-les als requeriments del PNAP i així obtenir resultats sobre el potencial de biomassa sosteniblement extraïble, sobre el seu potencial energètic i plantejar un escenari d’aprofitament a mode de cas pràctic. A més a més, s’han realitzat també un balanç socioeconòmic, un balanç d’emissions de CO2 i una avaluació d’impacte ambiental per tal de determinar la viabilitat de l’ús de biomassa al PNAP com a recurs energètic. Com a resultat final s’ha obtingut que el procés d’aprofitament de la biomassa per a la producció d’energia calorífica esdevé una opció de futur viable i positiva ja que la implantació del nou procés comporta beneficis a nivell social, econòmic i ambiental.

Modelo de gestión del entrenamiento basado en simulación orientado a la evaluación de impacto

Este documento forma parte del proyecto Fomento de las TIC para mejorar el aprendizaje a través de simulación en centros de salud (SIMBASE). Management model for simulation based-training oriented towards impact evaluation Versión en inglés disponible en http://www.simbase.co/results/)

Some conceptual difficulties regarding "net" multipliers

Multipliers are routinely used for impact evaluation of private projects and public policies at the national and subnational levels. Oosterhaven and Stelder (2002) correctly pointed out the misuse of standard 'gross' multipliers and proposed the concept of 'net' multiplier as a solution to this bad practice. We prove their proposal is not well founded. We do so by showing that supporting theorems are faulty in enunciation and demonstration. The proofs are flawed due to an analytical error but the theorems themselves cannot be salvaged as generic, non-curiosum counterexamples demonstrate. We also provide a general analytical framework for multipliers and, using it, we show that standard 'gross' multipliers are all that is needed within the interindustry model since they follow the causal logic of the economic model, are well defined and independent of exogenous shocks, and are interpretable as predictors for change.

Una aproximación teórico-práctica para el costeo de cápitas de paquetes de salud

Metodología para la estimación de cápitas de planes de salud. Se propone un método de cuantificación de insumos a emplear, la estimación de costos indirectos asociados a las intervenciones y un método de anualización para los costos fijos que generalmente trascienden en su vida útil al período de duración de una cápita. Finalmente, a través de un ejercicio práctico de estimación de la cápita del Componente de Salud del Programa Oportunidades de México se ilustra la aplicación del desarrollo metodológico expuesto en la parte inicial.

El programa Proniño en Colombia: su efecto en la reducción del trabajo infantil

El objetivo de este documento es evaluar empíricamente el efecto que tiene una mayor o menor exposición del programa Proniño de la Fundación Telefónica en la cantidad de horas trabajadas de los niños, niñas y adolescentes. Tomando información desde el 2010 y hasta el 2012 se evalúa empíricamente el impacto en la duración a la exposición del tratamiento, eliminando el sesgo de selección por duración a través de la metodología de Propensity Score Matching. Los resultados muestran que la exposición al tratamiento sí logra reducir el número de horas trabajadas de los niños, niñas y adolescentes, alcanzando los niveles más altos de reducción cuando el tiempo de exposición al programa es más amplio; en este caso, tres años y particularmente para el grupo de edad comprendido entre los 12 a 14 años. Finalmente se evidencia que el programa es más efectivo en la reducción del número de horas trabajadas a la semana de los niños (hombres) que de las niñas (mujeres).

Características de los de niños trabajadores en Colombia en el año 2010 por tipo de empleador

Introducción: El trabajo infantil es una problemática mundial que lleva a los niños a adquirir responsabilidades de adultos desde temprana edad, con el fin de obtener recursos económicos que contribuyan a mejorar la economía familiar. El presente trabajo estudia las características de los niños trabajadores en Colombia en el año 2010, teniendo en cuenta el tipo de empleador. Métodos: Estudio observacional analítico de corte transversal que se realizó empleando datos de la Encuesta Nacional de Demografía y Salud Colombia-2010. Se incluyeron en el análisis todos los niños entre los 6 y 17 años con datos en la encuesta. Resultados: En total se incluyeron 51.496 niños. A comparación de los niños no trabajadores, los niños que trabajan, independientemente de su empleador, tienen mayor edad, vienen de hogares más pobres, son con mayor frecuencia hombres, no se encuentran escolarizados y pertenecen al régimen subsidiado o no están afiliados al sistema de seguridad social. Los niños que trabajan con sus familias refieren peor estado de salud, pertenecen con mayor frecuencia a la etnia indígena, reciben subsidios de alimentación, tienen con menor frecuencia presencia de su madre y viven en áreas rurales. Los niños que trabajan con un tercero están afiliados con menor frecuencia al régimen de seguridad social especial, tienen menor probabilidad de haber visitado un médico en el último año, el jefe de hogar no es hombre con mayor frecuencia, y viven principalmente en áreas rurales. Finalmente los niños que trabajan independiente pertenecen con mayor frecuencia a la etnia afro-descendiente y viven en hogares donde el jefe de hogar es mujer. Conclusiones: Los niños trabajadores presentan características comunes independientemente de su tipo de empleador, pero de acuerdo al tipo de empleador también se encuentran características particulares.

CGIAR research program on grain legumes

EXECUTIVE SUMMARY Background and context The Grain Legumes CRP was established to bring all research and development work on grain legumes within the CGIAR system under one umbrella. It was set up to provide public goods outcomes to serve the needs of the sustainable production and consumption of grain legumes in the developing world, capitalising upon their properties that enhance the natural resource base upon which production so unequivocally depends. The choice of species and research foci were finalised following extensive consultation with all stakeholders (though perhaps fewer end users), and cover all disciplines that contribute to long-lasting solutions to the issues of developing country production and consumption. ICRISAT leads Grain Legumes and is partnered by the CGIAR centers ICARDA, IITA and CIAT and a number of other important partners, both public and private, and of course farmers in the developed and developing world. Originally in mid-2012 Grain Legumes was structured around eight Product Lines (PL) (i.e. technological innovations) intersecting five Strategic Components (SC) (i.e. arranged as components along the value chain). However, in 2015, it was restructured along a more R4D output model leading to Intermediate Development Outcomes (IDOs). Thus five Flagship Projects (FP) more closely reflecting a systematic pipeline of progression from fundamental science, implementation of interventions and the development of capacity and partnerships to promote and adopt impactful outcomes: FP1) Managing Productivity through crop interactions with biotic and abiotic constraints; FP2) Determination of traits that address production constraints and opportunities; FP3) Trait Deployment of those traits through breeding; FP4) Seed Systems, post-harvest processing and nutrition; FP5) Capacity-Building and Partnerships. Another three cross-cutting FPs analyse the broader environment surrounding the adoption of outputs, the capitalising of investments in genomics research, and a focus on the Management and Governance of Grain Legumes: FP6) Knowledge, impacts, priorities and gender organisation; FP7) Tools and platforms for high throughput genotyping and bioinformatics; and FP8) Management and Governance. Five FPs focus on R4D; FPs 5 and 6 are considered cross-cutting; FP 7 has a technical focus and FP 8 has an overarching objective. Over the three year period since its inception in July 1012, Grain Legumes has had a total budget of $140 million, with $62M originally to come from W1/W2 and the remaining $78M to come from W3/bilateral. In actuality only $45M came from W1/W2 but $106M from W3/bilateral corresponding to 106% of expectation. Purpose, scope and objectives of the external evaluation Principally, the evaluation of Grain Legumes is to ensure that the program is progressing in an effective manner towards addressing the system-level outcomes of the CGIAR as they relate to grain legumes. In essence, the evaluation aims to provide essential evaluative information for decision-making by Program Management and its funders on issues such as extension, expansion and structuring of the program and adjustments in relevant parts of the program. Subsequent to the formal signing of the agreed terms of reference, the evaluation team was also invited to comment upon the mooted options for merging and/or disaggregating of Grain Legumes. The audiences are therefore manifold, from the CGIAR Fund Council and Consortium, the Boards of Trustees of the four component CGIAR centres, the Grain Legumes Steering, Management and Independent Advisory Committees, to the researchers and others involved in the delivery of R4D outcomes and their partner organisations. The evaluation was not only summative in measuring results from Grain Legumes at arm’s length; it was also formative in promoting learning and improvements, and developmental in nurturing adaption to transformational change with time. The evaluation report was written in a manner that allows for engagement of key partners and funders in a dialogue as to how to increase ownership and a common understanding of how the goals are to be achieved. We reviewed research undertaken before the CRPs but leading to impacts during Grain Legumes, and research commenced over the past 2.5 years. For related activities pre- and post-commencement of Grain Legumes, we reviewed the relevance of activities and their relation to CGIAR and the Grain Legumes goals, whether they were likely to lead to the outcomes and impacts as documented in the Grain Legumes proposal, and the quality of the science underpinning the likelihood to deliver outcomes. Throughout, we were cognisant of the extent of the reach of CGIAR centres’ activities, and those of stakeholders upon which the impact of CGIAR R4D depends. Within our remit we evaluated the original and modified management and governance structures, and all the processes/responsibilities managed within those structures. Besides the evaluation of the technical and managerial issues of Grain Legumes, we addressed cross-cutting issues of gender sensitivity, capacity building and the creation and nurturing of partnerships. The evaluation also has the objective to provide information relating to the development of full proposals for the new CRP funding cycle. The evaluation addressed six overarching questions developed from the TOR questions (listed in the Inception Report, 2015 [http://1drv.ms/1POQSZh] and others including cross-cutting issues, phrasing them within the context of traditional evaluation criteria: 1. Relevance: Global development, urbanisation and technological innovation are progressing rapidly, are the aims and focus of Grain Legumes coherent, robust, fit for purpose and relevant to the global community? 2. Efficiency: Is the structure and effectiveness of leadership across Grain Legumes developing efficient partnership management and project management across PLs? 3. Quality of science: Is Grain Legumes utilising a wide range of technologies in a way that will increase our fundamental understanding of the biology that underpins several PLs; and are collected data used in the most effective way? 4. Effectiveness: Are Product Lines strategic contributors to the overarching aims and vision for Grain Legumes? 5. Impact: Are the impact pathways that underlie each PL well defined, measureable and achievable; and are they sufficiently defined in terms of beneficiaries? Does progress towards achieving outputs and outcomes from the major research areas indicate a lasting benefit for CGIAR and the communities it serves? 6. Sustainability: Is Grain Legumes managing the increasing level of restricted funding in terms of program quality and effectiveness, including attracting and retaining quality staff? Questions for the evaluation of governance and management focused on accountability, transparency, the effectiveness and success of program execution, change management processes and communication methods, taking account of the effects of CGIAR reform. The three crosscutting issues were considered as follows: i) gender balance in program delivery, e.g. whether each PL is able to contribute to the increased income, food security, nutrition, environmental and resource conservation for resource-poor women and men existing in rural livelihoods; ii) are internal and external capacity gaps identified/met, is capacity effectively developed within each product line, and are staff at all levels engaged in contributing ideas towards capacity building; and iii) is there effective involvement of partners in research and activity programming, what are the criteria for developing partnerships, how they are formalised and how is communication between partners and within Grain Legumes managed? It was not in remit to search for output, outcomes or impact, however as highlighted later, much of our time was spent on searching for information to support claims of impact, since Grain Legumes had no effective dedicated M&E in place at the time of undertaking the review. Approach and methodology The evaluation was conducted when Grain Legumes had been operational for approximately 3 years. The approach and methodology followed that outlined in the Inception Report [http://1drv.ms/1POQSZh]. The CCEE Team based its findings, conclusions and recommendations on data collection from several sources:  review of program documents, communications with the CO, minutes and presentations from all management and governance committee meetings  review of previous assessments and evaluations  sampling of Grain Legume projects in 7 countries1  more than 66 face to face interviews, a further 133 persons in groups and 4 phone/Skype conversations: ICRISAT, ICARDA, CIAT and IITA staff, partners and stakeholders. Meetings with one Independent Science and Partnership Council (ISPC) member.  meetings with over 100 people in 16 external groups, such as farmers’ groups  online survey completed by 126 (33.4%) scientists who contribute to Grain Legumes and a number of non-CGIAR partners and Management representatives  bibliometric review of 10 publications within each PL to qualitatively assess the design, conduct, analysis and presentation of results  quantitative and qualitative self-assessment of the contributions of each of the PLs to the six criteria and 3 cross-cutting issues of evaluation mentioned above completed by PLCs (see below). We reviewed the Logical Framework that underpins the desired Goals, or Impacts of Grain Legumes, and the links between the outputs and inputs as they related to the organisational units of Grain Legumes. The logical framework approach to planning and management of Grain Legumes activities implies a linear process, leading from activities, outputs, outcomes, to impacts, but within such an approach there may be room for a more systems dynamics approach allowing for feedback at every step and within every step, in order to refine and improve upon the respective activities as new results, ideas, and directions come to light. We then developed a matrix that summarised quantitatively and qualitatively the contributions of each of the PLs to the six criteria and 3 cross-cutting issues of evaluation mentioned above. Main findings and conclusions Grain legume production and consumption remain of great importance to the food security of not inconsiderable populations in the developing world, and merit sustained research investment. We conclude that Grain Legumes continues to contribute significant returns to research investments by the CGIAR, and such investment should continue. The global research community looks to the CGIAR for leadership in Grain legumes, but needs to be assured of value adding when bringing CGIAR centres under the expected umbrella of synergy. However, there is considerable scope for improving the efficiency with which outcomes are achieved. We note that an absence of an effective M&E has hampered the assessment of the effectiveness of proposed impact pathways. Likewise progress has been hampered by the limited numbers of research partnerships with Advanced Institutes and by budgetary constraints (lamented for their stifling effects on continuation of ongoing exciting research). The unworkable management structure constrains the CRP Director’s leadership role; responsibility without authority will never lead to effective outcomes. Good fortune is responsible for many of the successes of Grain Legumes, underpinned by a devoted work force across the participating CGIAR centres and partners. The quality of the science is not uniformly high, and we believe that mentoring of scientists should be given priority where quality is poor. Simplified yet informative reporting is an imperative to this. World class science underpins the identification of, and molecular basis for, traits important for yield improvement and this expertise should be extended to all grain legume species, capitalising upon the germplasm collections. The linking of Grain Legumes with regional research and development consortia has been very successful, with outcomes aligning with those of Grain Legumes. We see that with declining funding consolidation of research effort based on likely successes will be necessary, and welcome the move afoot to incorporate grain legumes into an agri-food system focused on successful value chains that deliver sustainable outcomes. Relevance and Strategy Grain Legumes has geographic and disciplinary relevance, addressing the major supply chain issues of variety development seed system and agronomy, with some attention to quality and postharvest marketing systems. The CRP has provided the opportunity to cut ongoing and to initiate new research. Research funded by the Gates Foundation (Anon, 2014) suggests that the need for improvement is greatest in Africa and advocates reducing the number of crop by country combinations when resources are sparse. The lesser research investment in Latin America, however, is not in line with the regions’ dependency on legumes. In spite of the fact that there is no evidence of strong inter-partner CGIAR centre or internal synergy, the program is still moving ahead on most fronts in line with the overall project logframe. This is in spite of continual pushing and pulling by in particular donors and the CO. However, to quantify real impact, we believe Grain Legumes must have access to reliable baseline data on production and consumption, and this is missing. Similarly, there is little evidence of the proposed ‘Inclusive Market Oriented Development’ (IMOD) framework being used to assist with priority setting. The product lines, eight of which cover most of the historical programmes in place in the partner CGIAR centres at the commencement of the Grain Legumes, do not cover all the constraints for formal constraints analysis was not undertaken at the inception of the Grain Legumes, and some of this additionally identified research is undertaken under the umbrella of the FPs; this needs to be rationalised. We found the PLs to be isolated in activity, even with minimally-integrated activities within each PL, with little evidence of synergy between PLs. Even though the SCs should ensure a systems approach, as with the new FPs, we did not get a feel that this is so. The underplaying of agronomy, and production practices may be one reason for this. We believe that treating legume crops as if they were horticultural crops will increase farmer returns from investment. The choice of Flagship Projects makes sense, with the flow of activity firstly around crop management and agronomy followed by the logical sequence of trait discovery, incorporation into improved varieties, dissemination of those varieties through appropriate seed chains leading to market impacts, and the capacity building required at all steps. One obvious omission, however, is the lack of a central and strategic policy on the role of transgenics in Grain Legumes. We found four notable comparative advantages for Grain Legumes: the access to germplasm of component species, the use of the phenotyping facility at ICRISAT, the approach for village level industry for IPM, and the emphasis on hybrid pigeonpea. Efficiency Each centre has strong control of, and emphasis on, their ‘species’ domains, and ownership of the same detracts from possible synergy. Without synergy or value add, the Grain Legumes brings with it no comparative advantage over each centre continuing their own pre-CRP research agendas. We found little evidence of integration of programmes between centres and almost no cross-centre authorship of publications, such as could have occurred with the integrated cross-centre approaches to stress tolerance including crop modelling: the one publication (Gaur et al., 2015) on heat tolerance by ICRISAT, CIAT and ICARDA does not provide any keys to inter-centre collaboration. The integration of each centre with NARS and university research programmes is good, but the cross-centre links with NARS are poor. A better coordinated integration with Grain Legumes, , rather than through the individual centres, may reduce transactions costs for NARS, Monitoring and evaluation is, as noted throughout our report, one area of Grain Legumes research management that has not been given the attention it should have received. If it had have received proper attention, some of the issues of poor efficiency might have been nipped in the bud. A strong monitoring and evaluation system would have provided the baseline data and set the milestones that would have allowed both efficiency and effectiveness to be better appraised. We found no attempt to define comparative advantages of the CGIAR centres and their R4D activities, although practice showed the better grasp of CIAT in developing innovative seed distribution systems. During field visits and interviews, the CCEE Team observed shortcomings in the communication processes within Grain Legumes and with the broader scientific community and the public. For example, the public face of the program on the internet is out of date. Survey findings, however, suggest that information is shared freely and routinely within the PL within which scientists work. Some external issues, such as those with funding, low W1/W2 and poor sustainability of funding (especially if funding is top heavy with a few agencies), undermine research investment and confidence of partners in the system (e.g. as voiced by researchers working on crops and countries not included in TL III and the cessation of ongoing competitively-funded projects especially in India), but other issues attributable to the governance and management of the Grain Legumes, such as opaque integration of W3/bilaterals with W1/W2 funding require attention. Offsetting this, the existence of the Grain Legumes did mobilise additional funding [that it would not have if Grain Legumes did not exist]. We were concerned that Grain Legumes is simply not recognised outside of the CRP, with a limited www presence and centres promote themselves, rather than Grain Legumes (with exception in IITA). This is not a good move if one wishes to increase investment in the Grain Legumes. Although funding agencies require cost:benefit ratios, for example for each PL we faced difficulty in determining comparative value for money between investment in different types of research, and in being able to clearly attribute research and development outcomes to financial investment. There was also a time CCEE frame issue too. There is poor interaction with the private sector, notably in areas where they have a comparative financial advantage. We questioned in particular the apparent lack of interaction with the major agro-chemical companies, with respect to the development of herbicide tolerant (HT) grain legumes and the lack of evidence that the regulatory and trade aspects related to herbicide tolerant crops had been considered. Quality of science The quality of the science is highly variable across Grain Legumes, with pockets of real excellence that are linked to good levels of productivity, whereas other PLs are struggling to deliver quality publications, and outputs and outcomes that are based on these. There is much evidence of gradualism in terms of research output and outcomes, i.e. essentially the same activities that were ongoing at the time of the launch of Grain Legumes are still in place. However, there are examples of game changers including those from valuable investments in genomics, phenotyping, and bio-control. We were pleased to see large proportions of collaboration on publications with non-CGIAR centres, reflecting cooperation with partners in developed and developing countries. The value of collaboration when ensuring quality of science cannot be stressed highly enough both within the CRP, and with other global and national partners. PLs should be given incentives to collaborate with other CRPs and external institutions. There is little cohesion between PLs and with other CRPs as evidenced by publications, although there are some exceptions. We suspect the reasons for this are driven by funding. Productivity from the different PLs is also highly variable and it is not clear what other activities staff are engaged in since, in some PLs, they do not appear to lead to quality publications. Effectiveness Grain Legumes has been very effective in addressing component issues of research, but not the continuum from variety development to legumes on someone’s dinner plate. Our overall assessment of the effectiveness of Grain Legumes in stimulating synergy, innovation and impact indicate that gradualism is more prevalent than innovation. It also shows, as do publications, that there is little integration of disciplines or a focus on ‘systems’. The absence of socio-economists from research teams is evident in the general lack of an end user focus. However, research on genomics, plant breeding and seed systems have made great strides forward, on the brink of delivering impact. Agronomy has been a poor sister, but some of the competitive grants within Grain Legumes have unearthed some potential game changers, such as objective use of transplanting as an agronomic practice. As mentioned earlier, the lack of effective M&E (however, this was part of some major projects such as TL II/TL III), and therefore the ability to monitor impact pathways and achievement of impact, implies no systematic management of data. This creates difficulty when attempting to evaluate the achievement of the Grain Legumes objectives. One might have expected at least one attempt to try to develop publications between centres arguing for similar biologies/research approaches, bringing species together under one umbrella, but we did not find any evidence for this. It is most unfortunate that, due to budgetary cuts, the new ‘schemes’, e.g. competitive grants and scholarships, were cut off before gaining a foothold. With 8 species addressed by Grain Legumes, it is not unexpected that there will be little evidence of shared protocols across centres/species. One rare example was that hosted by the United States Department of Agriculture (USDA) on shared methods for phenotyping of legume germplasm. Researchers from CIAT, IITA, ICRISAT and three USDA stations attended, focusing in simple canopy temperature and root morphology measurements. It is our belief that as a set of research centres, the CGIAR centres should be focusing on the research for which they have a comparative advantage. While imposing the restructure to FPs, which is fine for development objectives and outcomes (funded through W3/bilateral), it is less so for a research institute, and the structure should not detract from the more basic work expected of an international CGIAR centre (or set of centres as in a CRP). Impact It is well known that research does not always lead to scientific breakthroughs. Also, activities such as plant breeding are long term; making impacts difficult to assess. We believe that sufficient progress with genomics and associated research has been made to warrant impact, but we are unable to quantify the levels of impact, or the timeframe for the same. Work in Grain Legumes has enormous potential for real impact in scientific research, commercial, farming, smallholder and household communities, much of which is being realised. However, the PLs need to become more adept at providing convincing cases that are strongly evidenced for these impacts, as this is likely to be a key factor in leveraging future funding. Claimed gains must be referenced against baseline data, and these are not always readily available. The CCEE Team realises that such impact evaluation represents a significant drain on resources, and Grain Legumes should determine whether the balance of costs to benefits favours such investment. Interviews conducted by the CCEE during site visits showed that PLs are quantifying the area of adoption of varieties, but in most cases they are not measuring the impact on environment, health/nutrition. Since the health and nutritional benefits and the environmental gains from growing legumes are major arguments for supporting grain legume research, the community is currently missing substantial opportunities to strengthen its own case for continued support. Whilst there are some impressive examples of considering the whole value chain, e.g. white beans from production through to export; in the main, the pipeline to end user is somewhat piece-meal, with no clear definition of the end user nor differential responsibility of Grain Legumes and of partners. The lack of robust time-defined impact pathways is highlighted in Section 7.4, and even though developed for PL5, timeframes are essential for measuring progress against prediction. Sustainability In summary, there is general acknowledgement that future funding is likely to become more limited, specifically in W1&2 and there is understandable concern over the support for the staff and basic infrastructure that underpin the Grain Legumes programme. For example, it is reported that staffing in parts of CIAT has been dependent on W1&2 and that this is too unstable to re-establish a critical mass. The present system whereby W3 and bilateral projects do not pay a realistic level of overheads means that such projects are being effectively subsidised by W1&2. This position is not sustainable in the long term as there will be a progressive but definite loss of basic skills and resources in the core centres. The only obvious options to prevent this outcome include a severe reduction in the fixed costs of the centres and/or a refusal to accept W3 and bilateral funding with an inadequate overhead component. In the latter case, there is an obvious danger that funders will move their resources away from the CGIAR system towards other, perhaps less expensive, suppliers of research, and possibly more relevant development expertise. This issue must be addressed. As the Grain Legumes moves into the future, and if sustainable funding cannot be assured, decisions must be made concerning a reduction in activities, keeping some caretaker breeding maintenance, and focus (as has TL III) on fewer species and a reduced geographic focus. Cross cutting issues: Gender, capacity building and partnerships Gender is not mainstreamed, but there is some evidence that this is improving, especially with dedicated gender specialists and the slow integration of gender across CRPs. There is a need to approach gender through the vision of agriculture as a social practice, with recognition of what changes will be acceptable culturally and what not, and capitalising upon the perceived and actual features of production and processing that grain legumes are primarily women-based crops. Gender awareness may be high among Scientists, but it appears to be a predominantly passive attribute with few proactively seeking opportunities for gender equity. It is, however, a sound sensitivity base on which to build. Nevertheless, examples of notable gender initiatives were identified during field visits. For example, in Benin, the development of biocontrol technologies has enthusiastically integrated diversity, engaging with women farmers’ and youths while maintaining cultural norms. Women are gathering and processing, youths are taking the product to market. The implication is that several groups benefit, rather than domination by the majority group. In Malawi, innovative approaches have been developed to improving nutrition for children, such as incorporating nutrient enriched bean flour products into snacks. In India, scientists collaborating with gender scientists and socio-economists are identifying the impact of mechanical harvesting on agricultural labour and the potential displacement of female labourers. In Kenya, a novel initiative is improving the accessibility of certified seed for new varieties. Seed suppliers have introduced small packs of grain legume seed at low unit cost, which are being purchased by young people and women. Capacity building efforts for external partners are not clearly aligned with the research mandate and delivery of Grain Legumes. However, there are a number of training activities that are being undertaken by Grain Legumes, largely through the W3/bilateral project. Gender balance never reaches parity, but it appears that efforts are made to include female participants. Within the evaluation timeframe it was not possible to conduct external surveys to further validate or review external capacity building efforts in Grain Legumes. Training of scientists is significant, with >40 benefiting. Postgraduate training is varied across PLs, and there is some opportunity to increase the numbers being supervised. We consider that support for postgraduates at ICRISAT could be better coordinated, satisfying more of the students’ needs. It is important, however, to follow up investments in capacity building by monitoring effectiveness, career progressions and so on. Training activities appear to be rather centre-specific, not following a coordinated programme managed by, nor at the level of, the Grain Legumes. Numbers of persons trained and their gender are important, but a measure of the effectiveness of the training is more important. Although optimism is expressed by the great majority of Research Managers that partnerships were working well to leverage knowledge and research capacities, scientists have a less favourable view, particularly in terms of their incentives to participate. It seems likely that the activities taking place within Grain Legumes were, in the most part, continuations of previous collaborations. This is not surprising in light of the reduction in the emphasis on partnerships as Grain Legumes evolved to a funded project, and the consequent lack of opportunity and ambition for establishing novel partnerships. Where they exist, partnerships are good on the whole, especially with US. They could be expanded where comparative advantages exist (for example with Canada and Australia for machine harvestable legumes), but some earlier identified partnerships, e.g. with Turkey, have not been capitalised upon. Others experience problems of variety access (the embargo on exports of some sources of materials from India), yet others do have relevance e.g. imported Brazilian varieties in pre-release in Ethiopia (even though two of the three are from CIAT materials). Governance and Management The standard format of committee structure and responsibilities is common to other CRPs, as are the attendant problems. One of the major problems is that the Grain Legumes Director has responsibility but no authority; hence, even with the support of the RMC, the Director is unable to ‘direct’ in the literal sense of the work the activities of Grain Legumes. We also see the same sense of helplessness with the role of the PLCs. They have responsibility but no authority in managing the affairs of their PL, and they have no access to funds with which to promote intellectual collaboration and cooperation. Minutes from governance and management meetings do not reflect the compromised weak position of the Director and the associated difficulties in the management of Grain Legumes. Nor do the minutes reflect concerns about the amount of time spent by scientists in meetings for planning, integration, evaluation and reporting. Many scientists reported significant opportunity costs in participating in the ongoing imposed [by the CO] evolution of Grain Legumes and CRPs in general. The changes brought in by the CO have not helped promote any greater authority and capacity of the Grain Legumes Director to direct. Likewise, they do not address any of the issues with the conflict of interest in having the Lead Centre chair the Steering Committee. Indeed, we believe that the combining of the Steering Committee with the Independent Advisory Committee, besides becoming unwieldy in number, annuls any sense of independence in advice offered to the Grain Legumes management. We have concerns with the declining proportion of W1/W2 funds (as expressed in the section on Sustainability), and believe that when basic financial planning takes place, integration of W1/W2 and W3/bilateral sources must occur, and be linked to anticipated outcomes and impacts. This will ensure a close alignment of collaborators’ and partners’ objectives and contributions to that of the Grain Legumes. We also queried the process for, and the formality, or lack of, surrounding, the approval of annual budgets, and the level of priority setting when budgets are cut. Recommendations for Grain Legumes The CCEE Team makes the following recommendations, critical issues are highlighted in bold, and those that require action by an entity other than the Grain Legumes Research Management Committee or Project Management united are identified in a footnote. Relevance and Strategy Recommendation 1: A period of consistency is necessary to raise confidence, morale and trust across scientists, managers and partners to foster the assembly of enduring Grain Legumes outcomes2.  There needs to be a concerted effort to undertake baseline studies and to implement a robust M&E activity during this period. Without these data the foundation for integrated research in grain legumes is jeopardised.  There is a strong need to link more closely with the private sector, especially where there are financial and other comparative advantages to do so. Recommendation 2: The agronomic and physiological trait targets of Grain Legumes (tolerance to changing climate patterns, to the pests and diseases of today and of the future, incorporation of quality traits and adaptations to intensive production systems [machine-harvestability and herbicide tolerance], and short season high yielding characters) are all worthy of continued investment when selecting for improved varieties.  There needs to be a common strategy, implemented across centres and species, as to how to address these trait targets through conventional and modern breeding approaches, but only if adequate funding is assured and secured and if a consistency and unity of purpose can be achieved across a large-scale. This should take the form of cross-species coordinated research programmes to address these breeding targets that cooperate across centres and make efficient use of facilities and other resources.  The CRP should undertake a detailed strategic review of the role of transgenics across the range of targets in the mandate crops. Efficiency Recommendation 3: The lack of an effective M&E process is a significant omission, not least in terms of more efficient use of resources and the lack of baseline data with which to measure impact, and must be rectified.  Reinforcing Recommendation 1, an effective M&E system initially directed towards baseline studies must be implemented.  Transaction costs may be reduced through bilateral projects, which are seen as more cost effective than W1/W2 where transaction costs are disproportionately higher. Recommendation 4: To improve communication and coordination within the CRP, and with a broader audience:  There is a priority need for a central database containing, names of staff associated with Grain Legumes and their time commitments, their responsibilities, and involvement in CRP activities, their progress and achievements, their publications, plans of training, travel, and other opportunities for interaction.  Regular global meetings of staff involved in managing PLs, the entire CRP management staff and the IAC are essential for effective coordination of all activity within Grain Legumes.  The website must be given a complete overhaul and improvement and then regular maintenance must be provided to keep it current. Quality of Science Recommendation 5: It is essential to continue investment in good science and to institute a change from gradualism in research output and outcomes to an expectation of innovative and concrete achievements that can be attributed clearly to people, centres and core facilities.  A cost:benefit analysis and subsequent strategic planning must be undertaken to justify further investment in the genomics and phenotyping facilities at ICRISAT especially as such technologies advance rapidly. Strategic planning and coordination must also be implemented for capitalising on the investment in crop simulation modelling. (The phenotyping facility of ICRISAT needs to focus on delivering some outcomes, not only outputs.)  PLs should be given incentives to collaborate with other CRPs and external institutions. The CCEE recommends special recognition of high quality collaborative papers, thereby encouraging increased quality of the research programmes and widening the penetration of research impacts.  More importance should be placed on the quality of publication, rather than quantity of outputs and there should be recognition of other types of outputs from Grain Legumes. The CRP Director must be party to this.  If staff are engaged in activities that relate more to impact than publication then this needs to be monitored and recorded and a clearer understanding developed of what constitutes a pathway to impact and how success of such activities can be evaluated. A system must be devised and incorporated into the M&E to enable recognition of other types of outputs (non- publication based) from Grain Legumes, e.g. varieties for breeders. Effectiveness Recommendation 6: To develop greater synergy, Grain Legumes should review management processes and the direction of research activities. In particular, far more extensive integration of research and knowledge exchange should take place across both African and Asian continents so that the best aspects of both can be shared. A multidisciplinary approach is recommended that considers processing solutions, as well as breeding solutions, to capitalise upon the nutritional benefits of the grain legume crops. We recommend:  A better collaboration with social scientists at the design stage of experiments in order to improve the utility of the work carried out and to understand its reach.  Supporting3 the adoption of best practice electronic data collection, central storage and open access, particularly of genomic data, for public use.  Given the focus on the link between phenotyping and genotyping, we note that there is a lack of congruence between the populations that are being phenotyped and those being genotyped, and therefore these could be better aligned within each species.  Concentrating investment external to Grain Legumes on scaling up production of varieties with the most promising trait profiles to meet the basic seed requirement.  Developing a more holistic approach that coordinates an understanding of the disease pathology and epidemiology, and of new chemicals before they become commercially available, together with agronomic practice such that recommendations can be made for growers. Continuing work to establish whether agronomic factors hold true in different environments and to assess GxE effects within breeding programmes. Such rigorous trial practices should be used to inform the evaluation of breeding lines and to provide phenotype data to associate with markers for traits such as heat, drought and herbicide tolerance.  Considering grain legumes as if they were vegetable crops in terms of the strategy for intensification of production, both from the management perspective and for seed systems, will be a useful development objective into the future. This will bring about more rapid intensification and is likely to increase farmer returns from investment. Recommendation 7: The CGIAR centres should focus in on the research for which they have a comparative advantage. While imposing the restructure to FPs, which is fine for development objectives and outcomes (funded through W3/bilateral) it is less so for a research institute, and should not detract from the more basic work expected of an international CGIAR centre (or set of centres in a CRP).  Collaborative approaches should be explored within Grain Legumes, e.g. similar biologies/research approaches, bringing species together under one umbrella. Similarly better alignment is needed to address the lack of congruence between the populations that are being phenotyped and those being genotyped.  Despite positive impacts from research in genomics, plant breeding and seed systems, the lack of an effective M&E, already mentioned elsewhere, has reduced the ability to monitor impact pathways. This must be addressed.  The absence of socio-economists from research teams is evident in the general lack of an end user focus. Responsibilities of the different actors in the whole value chain must be considered and identified when developing impact targets, and the pathway leading to them, for individual projects. People with socio-economist skills must be part of the team from project inception so that appropriate frameworks are incorporated for measuring and influencing sociological and economic changes brought about by Grain Legumes research. Impact Recommendation 8: PLs need to become more adept at providing convincing cases in which impact is strongly evidenced, as this is likely to be a key factor in leveraging future funding.  Claimed gains must be referenced against baseline data, and these are not always readily available. The CCEE Team realises that such impact evaluation represents a significant drain on resources, and Grain Legumes should determine whether the balance of costs to benefits favours such investment.  It is essential that Grain Legumes provides training to staff on what constitutes impact and how it can be recorded.  Specific, rather than generalised, potential impacts arising from activity within Grain Legumes should be defined at the time of justifying the programme of work and a pathway to impact should form part of the documentation prepared ahead of a piece of research commencing. . In other words, centres should submit work plans to Grain Legumes before they are undertaken using W1/W2 funds Recommendation 9: The reporting activity must be streamlined to a single (brief) format that can be used to report to Grain Legumes, Centres and to donors for special project activities4. Sustainability Recommendation 10: As Grain Legumes moves into the future, and if sustainable funding cannot be assured, decisions must be made concerning a reduction in activities, keeping some caretaker breeding maintenance, and focus (as has TL III) on fewer species and a reduced geographic focus. Zeigler (Director General of IRRI) states “…time and effort would be better spent … making tough decisions about which programs deserve the precious support.”  The present system whereby W3 and bilateral projects do not pay a realistic level of overheads means that such projects are being effectively subsidised by W1&2 and there will be a progressive but definite loss of basic skills and resources in the core centres. To prevent this outcome it is necessary to significantly reduce the fixed costs of the centres and/or refuse to accept W3 and bilateral funding without an adequate overhead component.  In the absence of long term certainty, the scale of the budget allocated to each of the new CRPs should be very conservative, a feature that can only be achieved by restricting/reducing the scope, probably quite significantly. Cross cutting issues: Gender, capacity building and partnerships Recommendation 11: The challenge for Grain Legumes is to achieve pro-active gender mainstreaming, which facilitates opportunities for gender diversity within all activities, from employment processes through research to end users.  Strategic measurable gender indicators need to be embedded in research design, for instance, through specific IDOs for each of the flagships projects. Accurate baseline data are also required to facilitate M&E reviews of progress.  Implementation of the Gender Strategy is the responsibility of everyone, not solely the Gender Team. Thus, ownership could be encouraged by setting personal development for key personnel objectives with specific outcomes, e.g. employment practices or research outcomes.  Recognising the positive gender initiatives in progress or planned, feedback must be communicated and integrated into broader research planning to share opportunities, methods and outcomes.  In addition to promoting gender equity in research, Grain Legumes also needs to ensure that working environments are gender sensitive and that recruitment processes, including promotion opportunities are equitable. Gender imbalance in management should be actively examined to identify further opportunities for developing female leadership. Recommendation 12: It is recommended that a training plan be devised to ensure that capacity building efforts are more clearly aligned with the research mandate, delivery and timeframe of Grain Legumes. Moreover, we recommend that ICRISAT develop a strategy to treat their new cohort of researchers more equitably in the future. Recommendation 13: To develop a more coherent strategic programme designed to eliminate overlap and promote synergy between programmes with common aims, Grain Legumes should hold a meeting with a range of partners. Governance Recommendation 14: Governance processes should be re-assessed and the structure altered to ensure that the Grain Legumes Director has the authority and budget control to drive the execution of strategy.  The ISC should be truly independent and given the power to influence strategic decisions before they become final. We also recommend that PLCs are provided with the authority to manage the direction and finances of their PL; and that ring-fenced funds are provided for the promotion of collaboration, coordination and staff training5. The way ahead In our view, having seen the ineffectiveness of much of the attempts [or lack of attempts] to harness synergies between multiple centres, and of the strength in few or sole centre partnerships, we believe that there is little to justify a full retention of the 8 legume species and 4 CGIAR centres in a CRP. TL I and II and PABRA have shown to be reasonably good cross-centre and single centre integrated programmes, but even they suffer from incomplete value chain approaches to increasing rural incomes while increasing food and nutritional security; they both need multi-faceted solutions which are not immediately forthcoming from Grain Legumes. It is important to embed Grain Legumes research within the agri-food systems these crops serve. Figure ES1 broadly shows the perceived current and potential degrees of synergy between centres, PLs and species, and is discussed more in the text. It is clear that the value chains for individual species from trait determination to nutritional impact have more cohesion than do the individual activities (e.g. trait deployment) across species. For this reason we believe that the future for research in Grain Legumes is best addressed by focusing on each of the species separately, and within an ecosystem framework; any synergy for research across species can be effected through communication and not necessarily through obligatory cooperative research. The ecosystem framework will allow for strengthening of agronomy type systems research, the arguments for benefits of inclusion of grain legumes in cropping systems, which is notable by its absence in much of what Grain Legumes currently undertakes. Figure ES1. Current and potential degrees of synergy between centres, PLs and crop species We therefore agree with the innovation in agri-food systems approach of the CG, and believe that Grain Legumes rightly belongs in the Dryland Cereals and Legumes Agri-food Systems. We believe that the option of combining the crops of dryland cereals and legumes in the cereal-legume-livestock systems of subsistence farming communities for whole-farm productivity is closest to the best way forward. Indeed the inclusion of grain legumes may not warrant even a CRP alone, rather the legume components should fit in with the major crops that determine the production systems. Legumes will always be subservient to the major cereals, as necessary adjuncts to the whole production system, providing both nutritional diversity and environmental services, neither achievable from cereals alone. Figure ES2. Most suitable option for integration of Grain Legumes and Dryland Cereals into an Agri-Food Systems CRP Most suitable option for integration of Grain Legumes and Dryland Cereals into an Agri-Food Systems CRP, which  Incorporates ex-Dryland Systems, Dryland Cereals, Grain Legumes, some HumidTropics, some ex-Livestock &Fisheries into a new CRP  Will cover full agri-food system VC for all 8 legumes in all ecologies, but must interact (dock) with the relevant AFS-CRPs for the dominant cereal in the relevant ecology  Hence, will need to negotiate with other Agrifood Systems-CRPs on who does what for legumes  In addition, responsible for sorghum and millet in the mixed dryland crop-livestock agro-ecologies For major game changers to be effected, we believe that the game has to change, and there is little evidence of this. The direction of CRPs is the correct route, but the journey has not yet come to its destination. A major change of game [such as the adoption of a Flagship Project approach as exemplified by the Australian CSIRO – where flagships contract services from centres of research excellence] would be painful to implant. The CGIAR system is going down the right pathway but it has not gone far enough.

Uma metodologia de avaliação de impacto de políticas públicas no Brasil

Esta dissertação examina as políticas públicas brasileiras do período posterior à promulgação da Constituição Federal de 1988 pela ótica da relação Estado-Sociedade. Ela desenvolve uma metodologia compreensiva para avaliar impactos das políticas públicas no Brasil com base na influência dos princípios fundamentais da Constituição brasileira, das crenças da Sociedade, das premissas do Estado, da epistemologia sobre avaliação de impacto de políticas públicas e das relações internacionais multilaterais sobre as instituições nacionais.