Evaluation policy and strategy: Programme Planning and Evaluation Unit (PPEU) Programme Planning and Operations Division (PPOD) April 2014

Includes bibliography

Evaluation report of the regional workshop on the measurement of poverty in the Caribbean

1. The member and associate member countries of the Economic Commission for Latin America and the Caribbean/Caribbean Development and Cooperation Committee (ECLAC/CDCC) have committed to pursuing and achieving the Millennium Development Goals, a common set of goals and targets to bring all people up to minimum acceptable standards of human development by 2015. 2. However, in spite of various capacity-building initiatives, Caribbean countries continued to experience difficulties in addressing additional demands of monitoring and measuring progress created by the Millennium Development Goals and other Internationally Agreed Development Goals. Therefore, it was necessary to implement activities to ensure the further building/strengthening of institutional capabilities for generating reliable social, economic and environmental statistics among Caribbean States. 3. The ECLAC project entitled “Strengthening the Capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals and other Internationally Agreed Development Goals” sought to build and strengthen institutional capabilities for generating and compiling reliable social, economic and environmental statistics in the Caribbean subregion, through the provision of technical support, as well as the conduct of training workshops for statisticians and policymakers. 4. Within the objectives of that project, ECLAC Subregional Headquarters for the Caribbean convened a regional training workshop on the measurement of poverty in the Caribbean in Port of Spain, to build the capacity of government officials and other relevant stakeholders. 5. The overall objective of the workshop was to develop and strengthen the national technical capacity of public officials in data processing, systematization and dissemination of poverty indicators and measurement in the Caribbean subregion. The workshop further sought to review and discuss the current approaches to poverty measurement and monitoring in an effort to identify methods to ensure that monitoring and reporting of the Millennium Development Goals were conducted according to internationally agreed upon methodologies. Furthermore, the workshop also intended to review different methods of poverty measurements, including the multidimensional methodology for the measurement of poverty. 6. Participants were introduced to different methods of poverty measurements and other aggregation proposals which would enable countries to better measure progress towards Goal 1 on poverty, report on it and apply evidence-based approaches to national policymaking and planning.

Qualifying instrument for evaluation of food and nutritional care in hospital

Establishing criteria for hospital nutrition care ensures that quality care is delivered to patients. The responsibility of the Hospital Food and Nutrition Service (HFNS) is not always well defined, despite efforts to establish guidelines for patient clinical nutrition practice. This study describes the elaboration of an Instrument for Evaluation of Food and Nutritional Care (IEFNC) aimed at directing the actions of the Hospital Food and Nutrition Service. This instrument was qualified by means of a comparative analysis of the categories related to hospital food and nutritional care, published in the literature. Elaboration of the IEFNC comprised the following stages: (a) a survey of databases and documents for selection of the categories to be used in nutrition care evaluation, (b) a study of the institutional procedures for nutrition practice at two Brazilian hospitals, in order to provide a description of the sequence of actions that should be taken by the HFNS as well as other services participating in nutrition care, (c) design of the IEFNC based on the categories published in the literature, adapted to the sequence of actions observed in the routines of the hospitals under study, (d) application of the questionnaire at two different hospitals that was mentioned in the item (b), in order to assess the time spent on its application, the difficulties in phrasing the questions, and the coverage of the instrument, and (e) finalization of the instrument. The IEFNC consists of 50 open and closed questions on two areas of food and nutritional care in hospital: inpatient nutritional care and food service quality. It deals with the characterization and structure of hospitals and their HFNS, the actions concerning the patients' nutritional evaluation and monitoring, the meal production system, and the hospital diets. "This questionnaire is a tool that can be seen as a portrait of the structure and characteristics of the HFNS and its performance in clinical and meal management dietitian activities." (Nutr Hosp. 2012;27:1170-1177) DOI:10.3305/nh.2012.27.4.5868

Essays in policy evaluation

This dissertation consists of three empirical studies that aim at providing new evidence in the field of public policy evaluation. In particular, the first two chapters focus on the effects of the European cohesion policy, while the third chapter assesses the effectiveness of Italian labour market incentives in reducing long-term unemployment. The first study analyses the effect of EU funds on life satisfaction across European regions , under the assumption that projects financed by structural funds in the fields of employment, education, health and environment may affect the overall quality of life in recipient regions. Using regional data from the European Social Survey in 2002-2006, it resorts to a regression discontinuity design, where the discontinuity is provided by the institutional framework of the policy. The second study aims at estimating the impact of large transfers from a centralized authority to a local administration on the incidence of white collar crimes. It merges a unique dataset on crimes committed in Italian municipalities between 2007 and 2011 with information on the disbursement of EU structural funds in 2007-2013 programming period, employing an instrumental variable estimation strategy that exploits the variation in the electoral cycle at local level. The third study analyses the impact of an Italian labour market policy that allowed firms to cut their labour costs on open-ended job contracts when hiring long-term unemployed workers. It takes advantage of a unique dataset that draws information from the unemployment lists in Veneto region and it resorts to a regression discontinuity approach to estimate the effect of the policy on the job finding rate of long-term unemployed workers.

Implementation of an Anthropomorphic Phantom for the Evaluation of Proton Therapy Treatment Procedures

With an increasing number of institutions offering proton therapy, the number of multi-institutional clinical trials involving proton therapy will also increase in the coming years. The Radiological Physics Center monitors sites involved in clinical trials through the use of site visits and remote auditing with thermoluminescent dosimeters (TLD) and mailable anthropomorphic phantoms. Currently, there are no heterogeneous phantoms that have been commissioned to evaluate proton therapy. It was hypothesized that an anthropomorphic pelvis phantom can be designed to audit treatment procedures (patient simulation, treatment planning and treatment delivery) at proton facilities to confirm agreement between the measured dose and calculated dose within 5%/3mm with a reproducibility of 3%. A pelvis phantom originally designed for use with photon treatments was retrofitted for use in proton therapy. The relative stopping power (SP) of each phantom material was measured. Hounsfield Units (HU) for each phantom material were measured with a CT scanner and compared to the relative stopping power calibration curve. The tissue equivalency for each material was calculated. Two proton treatment plans were created; one which did not correct for material SP differences (Plan 1) and one plan which did correct for SP differences (Plan 2). Film and TLD were loaded into the phantom and the phantom was irradiated 3 times per plan. The measured values were compared to the HU-SP calibration curve and it was found that the stopping powers for the materials could be underestimated by 5-10%. Plan 1 passed the criteria for the TLD and film margins with reproducibility under 3% between the 3 trials. Plan 2 failed because the right-left film dose profile average displacement was -9.0 mm on the left side and 6.0 mm on the right side. Plan 2 was intended to improve the agreements and instead introduced large displacements along the path of the beam. Plan 2 more closely represented the actual phantom composition with corrected stopping powers and should have shown an agreement between the measured and calculated dose within 5%/3mm. The hypothesis was rejected and the pelvis phantom was found to be not suitable to evaluate proton therapy treatment procedures.

Emergency Provider Use of Plain Radiographs in the Evaluation of Pediatric Constipation

Thesis (Master's)--University of Washington, 2016-06

Peer mentoring works! How peer mentoring enhances student success in Higher Education:evaluation toolkit

This toolkit, published by the HEA, provides colleagues across the Sector with the practical and methodological tools to empirically evaluate peer mentoring and tutoring. This evaluation kit provides two data collection tools that may be adopted and adapted to meet institutional requirements. The first of these is a survey, developed out of the original survey used in the Peer Mentoring Works Project. Some questions have been added as a result of reflexive application of colleagues and students input as the project has progressed. The second part of the toolkit comprises a qualitative interview guide. This guide is similar to the one used in the research, but again it has been further developed as a result of the project. It may be adapted for use in focus groups or one-to-one interviews. The final document within the kit is a sample consent form.

Developing a robust self evaluation framework for active learning:the first stage of an ERASMUS+ project (QAEMarketPlace4HEI)

In ensuring the quality of learning and teaching in Higher Education, self-evaluation is an important component of the process. An example would be the approach taken within the CDIO community whereby self-evaluation against the CDIO standards is part of the quality assurance process. Eight European universities (Reykjavik University, Iceland; Turku University of Applied Sciences, Finland; Aarhus University, Denmark; Helsinki Metropolia University of Applied Sciences, Finland; Ume? University, Sweden; Telecom Bretagne, France; Aston University, United Kingdom; Queens University Belfast, United Kingdom) are engaged in an EU funded Erasmus + project that is exploring the quality assurance process associated with active learning. The development of a new self-evaluation framework that feeds into a ?Marketplace? where participating institutions can be paired up and then engage in peer evaluations and sharing around each institutions approach to and implementation of active learning. All of the partner institutions are engaged in the application of CDIO within their engineering programmes and this has provided a common starting point for the partnership to form and the project to be developed. Although the initial focus will be CDIO, the longer term aim is that the approach could be of value beyond CDIO and within other disciplines. The focus of this paper is the process by which the self-evaluation framework is being developed and the form of the draft framework. In today?s Higher Education environment, the need to comply with Quality Assurance standards is an ever present feature of programme development and review. When engaging in a project that spans several countries, the wealth of applicable standards and guidelines is significant. In working towards the development of a robust Self Evaluation Framework for this project, the project team decided to take a wide view of the available resources to ensure a full consideration of different requirements and practices. The approach to developing the framework considered: a) institutional standards and processes b) national standards and processes e.g. QAA in the UK c) documents relating to regional / global accreditation schemes e.g. ABET d) requirements / guidelines relating to particular learning and teaching frameworks e.g. CDIO. The resulting draft self-evaluation framework is to be implemented within the project team to start with to support the initial ?Marketplace? pairing process. Following this initial work, changes will be considered before a final version is made available as part of the project outputs. Particular consideration has been paid to the extent of the framework, as a key objective of the project is to ensure that the approach to quality assurance has impact but is not overly demanding in terms of time or paperwork. In other words that it is focused on action and value added to staff, students and the programmes being considered.

The political economy of co-ordination challenges in the National Health Service: a postpositivist evaluation of diabetes policy and governance

The present PhD thesis develops and applies an evaluative methodology suited to the evaluation of policy and governance in complex policy areas. While extensive literatures exist on the topic of policy evaluation, governance evaluation has received less attention. At the level of governance, policymakers confront choices between different policy tools and governance arrangements in their attempts to solve policy problems, including variants of hierarchy, networks and markets. There is a need for theoretically-informed empirical research to inform decision-making at this level. To that end, the PhD develops an approach to evaluation by combining postpositivist policy analysis with heterodox political economy. Postpositivist policy analysis recognises that policy problems are often contested, that choices between policy options can involve significant trade-offs and that knowledge of policy options is itself dispersed and fragmented. Similarly, heterodox economics combines a concept of incommensurable values with an appreciation of the strengths and weaknesses of different institutional arrangements to realise them. A central concept of the field is coordination, which orientates policy analysis to the interactions of stakeholders in policy processes. The challenge of governance is to select the appropriate policy tools and arrangements which facilitate coordination. Via a postpositivist exploration of stakeholder ‘frames’, it is possible to ascertain whether coordination is occurring and to identify problems if it is not. Evaluative claims of governance can be made where arrangements can be shown to frustrate the realisation of shared values and objectives. The research makes a contribution to knowledge in a number of ways a) a distinctive evaluative approach that could be applied to other areas of health and public policy b) greater appreciation of the strengths and weaknesses of different forms of evidence in public policy and in particular health policy and c) concrete policy proposals for the governance and organisation of diabetes services, with implications for the NHS more broadly.

Evaluation of a Brief Intervention to Assist Health Visitors and Community Practitioners to Engage with Fathers as Part of the Healthy Child Initiative

Aims: To improve engagement of Health Visitors and Community Practitioners delivering the Healthy Child Programme with fathers. To evaluate a one-day, father-focused workshop with a supporting handbook for Practitioners. To identify institutional and organisational barriers to engagement with fathers. Background: The UK government policy encourages health professionals to engage with fathers. This derives from robust evidence that fathers’ early involvement with their children impacts positively on emotional, behavioural and educational development. Yet, there is little evidence that the importance of engaging fathers is reflected in Health Visitor training or that primary-care services are wholly embracing father-inclusive practice. The Fatherhood Institute (FI), a UK charity, has developed a workshop for Practitioners delivering the Healthy Child Programme. Method: A ‘before and after’ evaluation study, comprising a survey followed by telephone interviews, evaluated the impact of the FI workshop on Health Visitors’ and Community Practitioners’ knowledge, attitudes and behaviour in practice. A total of 134 Health Visitors and Community Practitioners from eight NHS Trusts in England attended the workshop from November 2011 to January 2014 at 12 sites. A specially constructed survey, incorporating a validated questionnaire, was administered before the workshop, immediately afterwards and three months later. Telephone interviews further explored participants’ responses. Findings: Analysis of the questionnaire data showed that the workshop and handbook improved participants’ knowledge, attitudes and behaviour in practice. This was sustained over a three-month period. In telephone interviews, most participants said that the workshop had raised their awareness of engaging fathers and offered them helpful strategies. However, they also spoke of barriers to engagement with fathers. NHS Trusts need to review the training and education of Health Visitors and Community Practitioners and take a more strategic approach towards father-inclusive practice and extend services to meet the needs of fathers.

Assessing the resilience of a real-world social-ecological system: lessons from a multidisciplinary evaluation of a South African pastoral system

In the past decades, social-ecological systems (SESs) worldwide have undergone dramatic transformations with often detrimental consequences for livelihoods. Although resilience thinking offers promising conceptual frameworks to understand SES transformations, empirical resilience assessments of real-world SESs are still rare because SES complexity requires integrating knowledge, theories, and approaches from different disciplines. Taking up this challenge, we empirically assess the resilience of a South African pastoral SES to drought using various methods from natural and social sciences. In the ecological subsystem, we analyze rangelands’ ability to buffer drought effects on forage provision, using soil and vegetation indicators. In the social subsystem, we assess households’ and communities’ capacities to mitigate drought effects, applying agronomic and institutional indicators and benchmarking against practices and institutions in traditional pastoral SESs. Our results indicate that a decoupling of livelihoods from livestock-generated income was initiated by government interventions in the 1930s. In the post-apartheid phase, minimum-input strategies of herd management were adopted, leading to a recovery of rangeland vegetation due to unintentionally reduced stocking densities. Because current livelihood security is mainly based on external monetary resources (pensions, child grants, and disability grants), household resilience to drought is higher than in historical phases. Our study is one of the first to use a truly multidisciplinary resilience assessment. Conflicting results from partial assessments underline that measuring narrow indicator sets may impede a deeper understanding of SES transformations. The results also imply that the resilience of contemporary, open SESs cannot be explained by an inward-looking approach because essential connections and drivers at other scales have become relevant in the globalized world. Our study thus has helped to identify pitfalls in empirical resilience assessment and to improve the conceptualization of SES dynamics.