Method for the evaluation of social projects

This instrument was part of the research project "Research on Evaluation of Health and Education Plans and Programs in the Province of Buenos Aires", developed by the of Chair Preventive Psychology of the Psychology course of studies at the School of Humanities and Educational Sciences, National Univer- sity of La Plata (Argentina). The basis for proposing an assessment instrument is the need for a method enabling analysis, systematization of knowledge and the assignment of values distributed into scales and organized in general charts, on social programs. Its main concern is the analysis of health and education programs and projects, restricted to certain specific areas or regions, in search for theoretic trustworthi-ness, methodological accuracy as well as pragmatic operability. This is the result of four years of researching said programs at system, service and community levels.

Method for the evaluation of social projects

This instrument was part of the research project "Research on Evaluation of Health and Education Plans and Programs in the Province of Buenos Aires", developed by the of Chair Preventive Psychology of the Psychology course of studies at the School of Humanities and Educational Sciences, National Univer- sity of La Plata (Argentina). The basis for proposing an assessment instrument is the need for a method enabling analysis, systematization of knowledge and the assignment of values distributed into scales and organized in general charts, on social programs. Its main concern is the analysis of health and education programs and projects, restricted to certain specific areas or regions, in search for theoretic trustworthi-ness, methodological accuracy as well as pragmatic operability. This is the result of four years of researching said programs at system, service and community levels.

Method for the evaluation of social projects

This instrument was part of the research project "Research on Evaluation of Health and Education Plans and Programs in the Province of Buenos Aires", developed by the of Chair Preventive Psychology of the Psychology course of studies at the School of Humanities and Educational Sciences, National Univer- sity of La Plata (Argentina). The basis for proposing an assessment instrument is the need for a method enabling analysis, systematization of knowledge and the assignment of values distributed into scales and organized in general charts, on social programs. Its main concern is the analysis of health and education programs and projects, restricted to certain specific areas or regions, in search for theoretic trustworthi-ness, methodological accuracy as well as pragmatic operability. This is the result of four years of researching said programs at system, service and community levels.

The state of semantic technology today - overview of the first SEALS evaluation campaigns

This paper describes the first five SEALS Evaluation Campaigns over the semantic technologies covered by the SEALS project (ontology engineering tools, ontology reasoning tools, ontology matching tools, semantic search tools, and semantic web service tools). It presents the evaluations and test data used in these campaigns and the tools that participated in them along with a comparative analysis of their results. It also presents some lessons learnt after the execution of the evaluation campaigns and draws some final conclusions.

Infrastructure and Workflow for the Formal Evaluation of Semantic Search Technologies

This paper describes an infrastructure for the automated evaluation of semantic technologies and, in particular, semantic search technologies. For this purpose, we present an evaluation framework which follows a service-oriented approach for evaluating semantic technologies and uses the Business Process Execution Language (BPEL) to define evaluation workflows that can be executed by process engines. This framework supports a variety of evaluations, from different semantic areas, including search, and is extendible to new evaluations. We show how BPEL addresses this diversity as well as how it is used to solve specific challenges such as heterogeneity, error handling and reuse

Design and evaluation of acceleration strategies for speeding up the development of dialog applications

In this paper, we describe a complete development platform that features different innovative acceleration strategies, not included in any other current platform, that simplify and speed up the definition of the different elements required to design a spoken dialog service. The proposed accelerations are mainly based on using the information from the backend database schema and contents, as well as cumulative information produced throughout the different steps in the design. Thanks to these accelerations, the interaction between the designer and the platform is improved, and in most cases the design is reduced to simple confirmations of the “proposals” that the platform dynamically provides at each step. In addition, the platform provides several other accelerations such as configurable templates that can be used to define the different tasks in the service or the dialogs to obtain or show information to the user, automatic proposals for the best way to request slot contents from the user (i.e. using mixed-initiative forms or directed forms), an assistant that offers the set of more probable actions required to complete the definition of the different tasks in the application, or another assistant for solving specific modality details such as confirmations of user answers or how to present them the lists of retrieved results after querying the backend database. Additionally, the platform also allows the creation of speech grammars and prompts, database access functions, and the possibility of using mixed initiative and over-answering dialogs. In the paper we also describe in detail each assistant in the platform, emphasizing the different kind of methodologies followed to facilitate the design process at each one. Finally, we describe the results obtained in both a subjective and an objective evaluation with different designers that confirm the viability, usefulness, and functionality of the proposed accelerations. Thanks to the accelerations, the design time is reduced in more than 56% and the number of keystrokes by 84%.

A constraint-based approach to quality assurance in service choreographies.

Knowledge about the quality characteristics (QoS) of service com- positions is crucial for determining their usability and economic value. Ser- vice quality is usually regulated using Service Level Agreements (SLA). While end-to-end SLAs are well suited for request-reply interactions, more complex, decentralized, multiparticipant compositions (service choreographies) typ- ically involve multiple message exchanges between stateful parties and the corresponding SLAs thus encompass several cooperating parties with interde- pendent QoS. The usual approaches to determining QoS ranges structurally (which are by construction easily composable) are not applicable in this sce- nario. Additionally, the intervening SLAs may depend on the exchanged data. We present an approach to data-aware QoS assurance in choreographies through the automatic derivation of composable QoS models from partici- pant descriptions. Such models are based on a message typing system with size constraints and are derived using abstract interpretation. The models ob- tained have multiple uses including run-time prediction, adaptive participant selection, or design-time compliance checking. We also present an experimen- tal evaluation and discuss the benefits of the proposed approach.

Semantic evaluation at large scale (SEALS)

This paper describes the main goals and outcomes of the EU-funded Framework 7 project entitled Semantic Evaluation at Large Scale (SEALS). The growth and success of the Semantic Web is built upon a wide range of Semantic technologies from ontology engineering tools through to semantic web service discovery and semantic search. The evaluation of such technologies ? and, indeed, assessments of their mutual compatibility ? is critical for their sustained improvement and adoption. The SEALS project is creating an open and sustainable platform on which all aspects of an evaluation can be hosted and executed and has been designed to accommodate most technology types. It is envisaged that the platform will become the de facto repository of test datasets and will allow anyone to organise, execute and store the results of technology evaluations free of charge and without corporate bias. The demonstration will show how individual tools can be prepared for evaluation, uploaded to the platform, evaluated according to some criteria and the subsequent results viewed. In addition, the demonstration will show the flexibility and power of the SEALS Platform for evaluation organisers by highlighting some of the key technologies used.

Application of user-centered design in the development of an atomated/assisted testing system for self-service devices

Automated Teller Machines (ATMs) are sensitive self-service systems that require important investments in security and testing. ATM certifications are testing processes for machines that integrate software components from different vendors and are performed before their deployment for public use. This project was originated from the need of optimization of the certification process in an ATM manufacturing company. The process identifies compatibility problems between software components through testing. It is composed by a huge number of manual user tasks that makes the process very expensive and error-prone. Moreover, it is not possible to fully automate the process as it requires human intervention for manipulating ATM peripherals. This project presented important challenges for the development team. First, this is a critical process, as all the ATM operations rely on the software under test. Second, the context of use of ATMs applications is vastly different from ordinary software. Third, ATMs’ useful lifetime is beyond 15 years and both new and old models need to be supported. Fourth, the know-how for efficient testing depends on each specialist and it is not explicitly documented. Fifth, the huge number of tests and their importance implies the need for user efficiency and accuracy. All these factors led us conclude that besides the technical challenges, the usability of the intended software solution was critical for the project success. This business context is the motivation of this Master Thesis project. Our proposal focused in the development process applied. By combining user-centered design (UCD) with agile development we ensured both the high priority of usability and the early mitigation of software development risks caused by all the technology constraints. We performed 23 development iterations and finally we were able to provide a working solution on time according to users’ expectations. The evaluation of the project was carried out through usability tests, where 4 real users participated in different tests in the real context of use. The results were positive, according to different metrics: error rate, efficiency, effectiveness, and user satisfaction. We discuss the problems found, the benefits and the lessons learned in the process. Finally, we measured the expected project benefits by comparing the effort required by the current and the new process (once the new software tool is adopted). The savings corresponded to 40% less effort (man-hours) per certification. Future work includes additional evaluation of product usability in a real scenario (with customers) and the measuring of benefits in terms of quality improvement.

A web-based application for the management and evaluation of tutoring requests in PBL-based massive laboratories

One important steps in a successful project-based-learning methodology (PBL) is the process of providing the students with a convenient feedback that allows them to keep on developing their projects or to improve them. However, this task is more difficult in massive courses, especially when the project deadline is close. Besides, the continuous evaluation methodology makes necessary to find ways to objectively and continuously measure students' performance without increasing excessively instructors' work load. In order to alleviate these problems, we have developed a web service that allows students to request personal tutoring assistance during the laboratory sessions by specifying the kind of problem they have and the person who could help them to solve it. This service provides tools for the staff to manage the laboratory, for performing continuous evaluation for all students and for the student collaborators, and to prioritize tutoring according to the progress of the student's project. Additionally, the application provides objective metrics which can be used at the end of the subject during the evaluation process in order to support some students' final scores. Different usability statistics and the results of a subjective evaluation with more than 330 students confirm the success of the proposed application.

Quality of experience of adaptive video streaming: investigation in service parameters and subjective quality assessment methodology

The usage of HTTP adaptive streaming (HAS) has become widely spread in multimedia services. Because it allows the service providers to improve the network resource utilization and user׳s Quality of Experience (QoE). Using this technology, the video playback interruption is reduced since the network and server status in addition to capability of user device, all are taken into account by HAS client to adapt the quality to the current condition. Adaptation can be done using different strategies. In order to provide optimal QoE, the perceptual impact of adaptation strategies from point of view of the user should be studied. However, the time-varying video quality due to the adaptation which usually takes place in a long interval introduces a new type of impairment making the subjective evaluation of adaptive streaming system challenging. The contribution of this paper is two-fold: first, it investigates the testing methodology to evaluate HAS QoE by comparing the subjective experimental outcomes obtained from ACR standardized method and a semi-continuous method developed to evaluate the long sequences. In addition, influence of using audiovisual stimuli to evaluate the video-related impairment is inquired. Second, impact of some of the adaptation technical factors including the quality switching amplitude and chunk size in combination with high range of commercial content type is investigated. The results of this study provide a good insight toward achieving appropriate testing method to evaluate HAS QoE, in addition to designing switching strategies with optimal visual quality.

How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol

Background: Despite the existence of ample literature dealing, on the one hand, with the integration of innovations within health systems and team learning, and, on the other hand, with different aspects of the detection and management of intimate partner violence (IPV) within healthcare facilities, research that explores how health innovations that go beyond biomedical issues—such as IPV management—get integrated into health systems, and that focuses on healthcare teams’ learning processes is, to the best of our knowledge, very scarce if not absent. This realist evaluation protocol aims to ascertain: why, how, and under what circumstances primary healthcare teams engage (if at all) in a learning process to integrate IPV management in their practices; and why, how, and under what circumstances team learning processes lead to the development of organizational culture and values regarding IPV management, and the delivery of IPV management services. Methods: This study will be conducted in Spain using a multiple-case study design. Data will be collected from selected cases (primary healthcare teams) through different methods: individual and group interviews, routinely collected statistical data, documentary review, and observation. Cases will be purposively selected in order to enable testing the initial middle-range theory (MRT). After in-depth exploration of a limited number of cases, additional cases will be chosen for their ability to contribute to refining the emerging MRT to explain how primary healthcare learn to integrate intimate partner violence management. Discussion: Evaluations of health sector responses to IPV are scarce, and even fewer focus on why, how, and when the healthcare services integrate IPV management. There is a consensus that healthcare professionals and healthcare teams play a key role in this integration, and that training is important in order to realize changes. However, little is known about team learning of IPV management, both in terms of how to trigger such learning and how team learning is connected with changes in organizational culture and values, and in service delivery. This realist evaluation protocol aims to contribute to this knowledge by conducting this project in a country, Spain, where great endeavours have been made towards the integration of IPV management within the health system.

Evaluation of an integrated system for classification, assessment and comparison of services for long-term care in Europe: the eDESDE-LTC study

Background: The harmonization of European health systems brings with it a need for tools to allow the standardized collection of information about medical care. A common coding system and standards for the description of services are needed to allow local data to be incorporated into evidence-informed policy, and to permit equity and mobility to be assessed. The aim of this project has been to design such a classification and a related tool for the coding of services for Long Term Care (DESDE-LTC), based on the European Service Mapping Schedule (ESMS). Methods: The development of DESDE-LTC followed an iterative process using nominal groups in 6 European countries. 54 researchers and stakeholders in health and social services contributed to this process. In order to classify services, we use the minimal organization unit or “Basic Stable Input of Care” (BSIC), coded by its principal function or “Main Type of Care” (MTC). The evaluation of the tool included an analysis of feasibility, consistency, ontology, inter-rater reliability, Boolean Factor Analysis, and a preliminary impact analysis (screening, scoping and appraisal). Results: DESDE-LTC includes an alpha-numerical coding system, a glossary and an assessment instrument for mapping and counting LTC. It shows high feasibility, consistency, inter-rater reliability and face, content and construct validity. DESDE-LTC is ontologically consistent. It is regarded by experts as useful and relevant for evidence-informed decision making. Conclusion: DESDE-LTC contributes to establishing a common terminology, taxonomy and coding of LTC services in a European context, and a standard procedure for data collection and international comparison.