An evaluation approach for a new paradigm - health care integration

This paper explores an approach to the implementation and evaluation of integrated health service delivery. It identifies the key issues involved in integration evaluation, provides a framework for assessment and identifies areas for the development of new tools and measures. A proactive role for evaluators in responding to health service reform is advocated.

Energy policies for rural electrification: a social multi-criteria evaluation approach

In this article, a real-world case- study is presented with two general objectives: to give a clear and simple illustrative example of application of social multi-criteria evaluation (SMCE) in the field of rural renewable energy policies, and to help in understanding to what extent and under which circumstances solar energy is suitable for electrifying isolated farmhouses. In this sense, this study might offer public decision- makers some insight on the conditions that favour the diffusion of renewable energy, in order to help them to design more effective energy policies for rural communities.

Validation of a novel approach to subjective quality evaluation of conventional and 3D broadcasted video services

Transmission errors are the main cause of degradation of the quality of real broadcasted video services. Therefore, knowing their impact on the quality of experience of the end users is a crucial issue. For instance, it would help to improve the performance of the distribution systems, and to develop monitoring tools to automatically estimate the quality perceived by the end users. In this paper we validate a subjective evaluation approach specifically designed to obtain meaningful results of the effects of degradations caused by transmission errors. This methodology has been already used in our previous works with monoscopic and stereoscopic videos. The validation is done by comparing the subjective ratings obtained for typical transmission errors with the proposed methodology and with the standard method Absolute Category Rating. The results show that the proposed approach could provide more representative evaluations of the quality of experience perceived by end users of conventional and 3D broadcasted video services.

Evaluation of boundary-layer type in a weather forecast model utilising long-term Doppler lidar observations

Many studies evaluating model boundary-layer schemes focus either on near-surface parameters or on short-term observational campaigns. This reflects the observational datasets that are widely available for use in model evaluation. In this paper we show how surface and long-term Doppler lidar observations, combined in a way to match model representation of the boundary layer as closely as possible, can be used to evaluate the skill of boundary-layer forecasts. We use a 2-year observational dataset from a rural site in the UK to evaluate a climatology of boundary layer type forecast by the UK Met Office Unified Model. In addition, we demonstrate the use of a binary skill score (Symmetric Extremal Dependence Index) to investigate the dependence of forecast skill on season, horizontal resolution and forecast leadtime. A clear diurnal and seasonal cycle can be seen in the climatology of both the model and observations, with the main discrepancies being the model overpredicting cumulus capped and decoupled stratocumulus capped boundary-layers and underpredicting well mixed boundary-layers. Using the SEDI skill score the model is most skillful at predicting the surface stability. The skill of the model in predicting cumulus capped and stratocumulus capped stable boundary layer forecasts is low but greater than a 24 hr persistence forecast. In contrast, the prediction of decoupled boundary-layers and boundary-layers with multiple cloud layers is lower than persistence. This process based evaluation approach has the potential to be applied to other boundary-layer parameterisation schemes with similar decision structures.

Addressing Multilevel Program Complexity by Evaluation Design

In this paper, we present the evaluation design for a complex multilevel program recently introduced in Switzerland. The evaluation embraces the federal level, the cantonal program level, and the project level where target groups are directly addressed. We employ Pawson and Tilley’s realist evaluation approach, in order to do justice to the varying context factors that impact the cantonal programs leading to varying effectiveness of the implemented activities. The application of the model to the canton of Uri shows that the numerous vertical and horizontal relations play a crucial role for the program’s effectiveness. As a general learning for the evaluation of complex programs, we state that there is a need to consider all affected levels of a program and that no monocausal effects can be singled out in programs where multiple interventions address the same problem. Moreover, considering all affected levels of a program can mean going beyond the borders of the actual program organization and including factors that do not directly interfere with the policy delivery as such. In particular, we found that the relationship between the cantonal and the federal level was a crucial organizational factor influencing the effectiveness of the cantonal program.

Empowerment Evaluation in Spain: The Critical Friend Role in Working with Rural Communities

Rural communities in Cuenca (Spain) are characterized by a great social dislocation, mostly due to the low population density in these areas. In this way, the existence of groups of citizens able to be active agents of their development process is a critical aspect for any community-based development process in this Spanish region. The Institute of Community Development of Cuenca (IDC) has been working with this type of groups for the last 30 years focusing on the organizational empowerment of the rural communities. Main tools in this process have been the empowerment evaluation approach and the critical friend role when helping the groups to achieve their objectives and reinforcing them. This chapter analyses the empowerment process and how the critical friend role is nourished by the facilitator figure.

Symmetric and asymmetric evaluation of the Tokamap in comparison with direct symplectic integration

Comparing the Poincaré plots of the Tokamap and the underlying Hamiltonian system reveals large differences. This stems from the particular choice of evaluation of the singular perturbations present in the system (a series of δ functions). A symmetric evaluation approach is proposed and shown to yield results that almost perfectly match the Hamiltonian system. © 2005 The American Physical Society.

The evaluation of information systems in the organisational context of the National Health Service. Available in 2 volumes.

This thesis describes a project which has investigated the evaluation of information systems. The work took place in, and is related to, a specific organisational context, that of the National Health Service (NHS). It aims to increase understanding of the evaluation which takes place in the service and the way in which this is affected by the NHS environment. It also investigates the issues which surround some important types of evaluation and their use in this context. The first stage of the project was a postal survey in which respondents were asked to describe the evaluation which took place in their authorities and to give their opinions about it. This was used to give an overview of the practice of IS evaluation in the NHS and to identify its uses and the problems experienced. Three important types of evaluation were then examined in more detail by means of action research studies. One of these dealt with the selection and purchase of a large hospital information system. The study took the form of an evaluation of the procurement process, and examined the methods used and the influence of organisational factors. The other studies are concerned with post-implementation evaluation, and examine the choice of an evaluation approach as well as its application. One was an evaluation of a community health system which had been operational for some time but was of doubtful value, and suffered from a number of problems. The situation was explored by means of a study of the costs and benefits of the system. The remaining study was the initial review of a system which was used in the administration of a Breast Screening Service. The service itself was also newly operational and the relationship between the service and the system was of interest.

A methodology for hierarchical image segmentation evaluation

This paper proposes a method to evaluate hierarchical image segmentation procedures, in order to enable comparisons between different hierarchical algorithms and of these with other (non-hierarchical) segmentation techniques (as well as with edge detectors) to be made. The proposed method builds up on the edge-based segmentation evaluation approach by considering a set of reference human segmentations as a sample drawn from the population of different levels of detail that may be used in segmenting an image. Our main point is that, since a hierarchical sequence of segmentations approximates such population, those segmentations in the sequence that best capture each human segmentation level of detail should provide the basis for the evaluation of the hierarchical sequence as a whole. A small computational experiment is carried out to show the feasibility of our approach.

A case study on a design-informed developmental evaluation

Developmental evaluation (DE) is an evaluation approach that aims to support the development of an innovation (Patton, 1994, 2011). This aim is achieved through supporting clients’ information needs through evaluative inquiry as they work to develop and refine the innovation. While core concepts and principles are beginning to be articulated and refined, challenges remain as to how to focus a developmental evaluation beyond those knowledge frameworks most immediate to clients to support innovation development. Anchoring a DE in knowledge frameworks other than those of the clients might direct attention to issues not yet obvious to clients, but which might further the goal of supporting innovation development if attended to. Drawing concepts and practices from the field of design may be one avenue with which to inform developmental evaluation in achieving its aim. Through a case study methodology, this research seeks to understand the nuances of operationalizing the guiding principles of DE as well as to investigate the utility, feasibility, and consequences of integrating design concepts and practices into developmental evaluation (design-informed developmental evaluation, “DI-DE”). It does so by documenting the efforts of a design-informed developmental evaluator and a task force of educators and researchers in a Faculty of Education as they work to develop a graduate-level education program. A systematic review into those purposeful efforts made to introduce DI-DE thinking into task force deliberations, and an analysis into the responses and consequences of those efforts shed light on what it had meant to practice DI-DE. As a whole, this research on evaluation is intended to further contemporary thinking about the closely coupled relationship between program development and evaluation in complex and dynamic environments.

European Stroke Organisation (ESO) guidelines for the management of temperature in patients with acute ischemic stroke.

BACKGROUND: Hyperthermia is a frequent complication in patients with acute ischemic stroke. On the other hand, therapeutically induced hypothermia has shown promising potential in animal models of focal cerebral ischemia. This Guideline Document presents the European Stroke Organisation guidelines for the management of temperature in patients with acute ischemic stroke. METHODS: A multidisciplinary group identified related questions and developed its recommendations based on evidence from randomized controlled trials elaborating the Grading of Recommendations Assessment, Development, and Evaluation approach. This Guideline Document was reviewed within the European Stroke Organisation and externally and was approved by the European Stroke Organisation Guidelines Committee and the European Stroke Organisation Executive Committee. RESULTS: We found low-quality evidence, and therefore, we cannot make any recommendation for treating hyperthermia as a means to improve functional outcome and/or survival in patients with acute ischemic stroke and hyperthermia; moderate evidence to suggest against routine prevention of hyperthermia with antipyretics as a means to improve functional outcome and/or survival in patients with acute ischemic stroke and normothermia; very low-quality evidence to suggest against routine induction of hypothermia as a means to improve functional outcome and/or survival in patients with acute ischemic stroke. CONCLUSIONS: The currently available data about the management of temperature in patients with acute ischemic stroke are limited, and the strengths of the recommendations are therefore weak. We call for new randomized controlled trials as well as recruitment of eligible patients to ongoing randomized controlled trials to allow for better-informed recommendations in the future.

Un outil d’évaluation neurocognitive des interactions humain-machine

De plus en plus de recherches sur les Interactions Humain-Machine (IHM) tentent d’effectuer des analyses fines de l’interaction afin de faire ressortir ce qui influence les comportements des utilisateurs. Tant au niveau de l’évaluation de la performance que de l’expérience des utilisateurs, on note qu’une attention particulière est maintenant portée aux réactions émotionnelles et cognitives lors de l’interaction. Les approches qualitatives standards sont limitées, car elles se fondent sur l’observation et des entrevues après l’interaction, limitant ainsi la précision du diagnostic. L’expérience utilisateur et les réactions émotionnelles étant de nature hautement dynamique et contextualisée, les approches d’évaluation doivent l’être de même afin de permettre un diagnostic précis de l’interaction. Cette thèse présente une approche d’évaluation quantitative et dynamique qui permet de contextualiser les réactions des utilisateurs afin d’en identifier les antécédents dans l’interaction avec un système. Pour ce faire, ce travail s’articule autour de trois axes. 1) La reconnaissance automatique des buts et de la structure de tâches de l’utilisateur, à l’aide de mesures oculométriques et d’activité dans l’environnement par apprentissage machine. 2) L’inférence de construits psychologiques (activation, valence émotionnelle et charge cognitive) via l’analyse des signaux physiologiques. 3) Le diagnostic de l‘interaction reposant sur le couplage dynamique des deux précédentes opérations. Les idées et le développement de notre approche sont illustrés par leur application dans deux contextes expérimentaux : le commerce électronique et l’apprentissage par simulation. Nous présentons aussi l’outil informatique complet qui a été implémenté afin de permettre à des professionnels en évaluation (ex. : ergonomes, concepteurs de jeux, formateurs) d’utiliser l’approche proposée pour l’évaluation d’IHM. Celui-ci est conçu de manière à faciliter la triangulation des appareils de mesure impliqués dans ce travail et à s’intégrer aux méthodes classiques d’évaluation de l’interaction (ex. : questionnaires et codage des observations).

Évaluation d’un programme de développement professionnel en santé publique : le laboratoire de promotion de la santé

Il y a quelques décennies, l’émergence du discours de la promotion de la santé infléchissait un nouveau tournant à la santé publique, orientant sa pratique vers l’action communautaire, participative et intersectorielle. Parallèlement, au Québec, la réforme du système de santé de 2004 réorganisait le niveau de gouverne locale à travers la création des centres de santé et de services sociaux (CSSS). Ceux-ci doivent articuler les secteurs des soins et de la santé publique à travers un continuum de services qui va de la promotion de la santé aux soins palliatifs. Ces changements ont des implications majeures pour les acteurs de la santé et de la santé publique, qui doivent composer avec de nouveaux rôles professionnels et de nouvelles stratégies d’action. Le développement professionnel est considéré comme un levier potentiel pour soutenir ces changements. En 2009, une équipe de la Direction de la santé publique de l’Agence de santé et des services sociaux de Montréal concevait un programme de développement professionnel appelé le Laboratoire de promotion de la santé. Ce programme mise sur une approche d’apprentissage de groupe pour permettre aux professionnels des CSSS de développer de nouvelles compétences, une pratique réflexive ainsi que de nouvelles pratiques de promotion de la santé. Basée sur une méthodologie générale qualitative et une approche d’évaluation collaborative, cette thèse utilise plusieurs stratégies d’investigation afin d’évaluer le Laboratoire de promotion de la santé sous trois angles, qui renvoient à sa conceptualisation, à son implantation et à ses effets. Plus spécifiquement, elle vise à : (1) examiner la plausibilité de la théorie d’intervention du programme; (2) décrire et comprendre les processus d’apprentissage d’équipe ainsi que les facteurs qui les influencent; et (3) explorer, du point de vue des participants, les effets réflexifs du Laboratoire. Afin de répondre à ces objectifs, la thèse mobilise diverses perspectives théoriques liées à l’apprentissage individuel, d’équipe et organisationnel. Les résultats des analyses démontrent que : (1) malgré quelques améliorations possibles, le modèle du programme est généralement bien conçu pour parvenir aux résultats visés; (2) l’implantation de ce modèle dans deux sites a donné lieu à des processus d’apprentissage d’équipe différents, bien que conditionnés par des facteurs communs liés aux participants, à l’équipe, au contexte organisationnel et à l’implantation du programme; (3) tel que visé, les participants des deux sites ont développé de la réflexivité vis-à-vis leur pratique et leur rôle professionnel – cette réflexivité adoptant une fonction formative ou critique en regard de l’expérience professionnelle. Ces résultats soulignent le potentiel que représente l’évaluation de la théorie d’intervention pour améliorer la conceptualisation d’un programme de développement professionnel, ainsi que l’intérêt et la pertinence d’évaluer les processus d’apprentissage au niveau de l’équipe dans le cadre d’une approche collective de développement professionnel. De plus, ils appuient l’importance de l’apprentissage réflexif pour l’amélioration des pratiques et l’engagement social des professionnels. En ce sens, ils proposent différentes avenues qui ont le potentiel de consolider les capacités de la main-d’œuvre de santé publique et d’influer conséquemment sur son efficacité à améliorer la santé des collectivités dans le prochain siècle.

MitWirkung - Zukunft gestalten

Die Diskussion in den Planungswissenschaften beschreibt den Paradigmenwechsel vom so genannten DEAD-Model (Decide-Announce-Defend) zum Tripple-D-Model (Dialogue-Decide-Deliver) und beschäftigt sich intensiv mit dem Thema Governance. Komplexe Planungsaufgaben brauchen eine Vielfalt an Lösungsideen unterschiedlicher gesellschaftlicher Gruppen. Planung verfolgt u. a. die Umsetzung der Ziele einer nachhaltigen Entwicklung, die die Einbeziehung der Öffentlichkeit (Zivilgesellschaft, Unternehmen und Bürger) verlangt. Darüber hinaus wird eine Erweiterung der Perspektive über Verfahren und Steuerungsformen hinaus auf Akteure und Prozesse gefordert. Räumliche Entwicklungen sollen stärker im Zusammenhang mit Entscheidungsfindungsprozesse untersucht werden. Die Dissertation ergänzt eine wirkungsorientierte Perspektive, die Wirkungen, sowohl räumliche als auch soziale, in den Mittelpunkt der Betrachtung stellt. Sie stützt sich auf Beobachtungen, dass klassisches Projektmanagement für erfolgreiche Planungsprozesse nicht ausreicht, sondern zusätzlich Prozessmanagement braucht. Mit der Weiterentwicklung der partizipativen Planung, die zusätzlich in den Kontext gesellschaftlicher Lernprozesse und zukunftsfähiger Veränderungen gesellschaftlicher Bedingungen gestellt wird, ergänzt die Dissertation planungswissenschaftliche Theorien. Aus einem fachübergreifenden Blickwinkel wird die räumliche Planung in die Reihe von Management- und Organisationswissenschaften eingeordnet. Ausgehend von der Frage, welche räumlichen und sozialen Wirkungen durch Beteiligungsprozesse unter welchen Bedingungen erzielt werden, wurden Fallstudien aus der Wasserwirtschaft und ihre Prozessbiografien umfassend evaluiert. Als Evaluierungsmethode wurde ein von der EU-Kommission empfohlener Evaluierungsrahmen gewählt, der sowohl den Prozess selbst, seine Rahmenbedingungen und Durchführung, als auch Wirkungen analysiert und bewertet. Auf der Grundlage der Ergebnisse und theoretischer Erkenntnisse, vorrangig aus der Evaluationsforschung, wird ein umfassender Beteiligungsansatz konzipiert. Dabei handelt es sich um ein offenes Gerüst, in das sich bewährte und innovative Elemente strategisch gezielt integrieren lassen. Die Struktur verbindet verschiedene Beteiligungswerkzeuge unterschiedlicher Intensitäten und für unterschiedliche Zielgruppen zu einem Gesamtkonzept, mit dem Ziel, möglichst die gewünschten Wirkungen zu erreichen. Wesentlich an dem Ansatz ist, dass bereits das Prozessdesign unter Mitwirkung von Projektträgern, Beratern und Schlüsselakteuren erfolgt. Die partizipative Beteiligungsplanung bedeutet somit Klärung der Vorgehensweise und gleichzeitig Bewusstseins- und Kompetenzerweiterung der verantwortlichen Akteure. Im Ausblick werden künftige Forschungsaufgaben im Bereich der Mitwirkung in der räumlichen Planung formuliert und Handlungsmöglichkeiten aufgezeigt, um Partizipation als Teil planerischer „Alltagskultur“ weiterzuentwickeln. Dies erfolgt vor dem Hintergrund der Bedeutung von Partizipation und Bildung als Umsetzungsstrategie von Ideen der Landschaftsentwicklung und Nachhaltigkeit.