Addressing Multilevel Program Complexity by Evaluation Design

In this paper, we present the evaluation design for a complex multilevel program recently introduced in Switzerland. The evaluation embraces the federal level, the cantonal program level, and the project level where target groups are directly addressed. We employ Pawson and Tilley’s realist evaluation approach, in order to do justice to the varying context factors that impact the cantonal programs leading to varying effectiveness of the implemented activities. The application of the model to the canton of Uri shows that the numerous vertical and horizontal relations play a crucial role for the program’s effectiveness. As a general learning for the evaluation of complex programs, we state that there is a need to consider all affected levels of a program and that no monocausal effects can be singled out in programs where multiple interventions address the same problem. Moreover, considering all affected levels of a program can mean going beyond the borders of the actual program organization and including factors that do not directly interfere with the policy delivery as such. In particular, we found that the relationship between the cantonal and the federal level was a crucial organizational factor influencing the effectiveness of the cantonal program.

A case study on a design-informed developmental evaluation

Developmental evaluation (DE) is an evaluation approach that aims to support the development of an innovation (Patton, 1994, 2011). This aim is achieved through supporting clients’ information needs through evaluative inquiry as they work to develop and refine the innovation. While core concepts and principles are beginning to be articulated and refined, challenges remain as to how to focus a developmental evaluation beyond those knowledge frameworks most immediate to clients to support innovation development. Anchoring a DE in knowledge frameworks other than those of the clients might direct attention to issues not yet obvious to clients, but which might further the goal of supporting innovation development if attended to. Drawing concepts and practices from the field of design may be one avenue with which to inform developmental evaluation in achieving its aim. Through a case study methodology, this research seeks to understand the nuances of operationalizing the guiding principles of DE as well as to investigate the utility, feasibility, and consequences of integrating design concepts and practices into developmental evaluation (design-informed developmental evaluation, “DI-DE”). It does so by documenting the efforts of a design-informed developmental evaluator and a task force of educators and researchers in a Faculty of Education as they work to develop a graduate-level education program. A systematic review into those purposeful efforts made to introduce DI-DE thinking into task force deliberations, and an analysis into the responses and consequences of those efforts shed light on what it had meant to practice DI-DE. As a whole, this research on evaluation is intended to further contemporary thinking about the closely coupled relationship between program development and evaluation in complex and dynamic environments.

Problem solving in a middle school robotics design classroom

Little research has been conducted on how students work when they are required to plan, build and evaluate artefacts in technology rich learning environments such as those supported by tools including flow charts, Labview programming and Lego construction. In this study, activity theory was used as an analytic tool to examine the social construction of meaning. There was a focus on the effect of teachers’ goals and the rules they enacted upon student use of the flow chart planning tool, and the tools of the programming language Labview and Lego construction. It was found that the articulation of a teacher’s goals via rules and divisions of labour helped to form distinct communities of learning and influenced the development of different problem solving strategies. The use of the planning tool flow charting was associated with continuity of approach, integration of problem solutions including appreciation of the nexus between construction and programming, and greater educational transformation. Students who flow charted defined problems in a more holistic way and demonstrated more methodical, insightful and integrated approaches to their use of tools. The findings have implications for teaching in design dominated learning environments.

Design of open source framework for traffic and travel simulation

For the evaluation, design, and planning of traffic facilities and measures, traffic simulation packages are the de facto tools for consultants, policy makers, and researchers. However, the available commercial simulation packages do not always offer the desired work flow and flexibility for academic research. In many cases, researchers resort to designing and building their own dedicated models, without an intrinsic incentive (or the practical means) to make the results available in the public domain. To make matters worse, a substantial part of these efforts pertains to rebuilding basic functionality and, in many respects, reinventing the wheel. This problem not only affects the research community but adversely affects the entire traffic simulation community and frustrates the development of traffic simulation in general. For this problem to be addressed, this paper describes an open source approach, OpenTraffic, which is being developed as a collaborative effort between the Queensland University of Technology, Australia; the National Institute of Informatics, Tokyo; and the Technical University of Delft, the Netherlands. The OpenTraffic simulation framework enables academies from geographic areas and disciplines within the traffic domain to work together and contribute to a specific topic of interest, ranging from travel choice behavior to car following, and from response to intelligent transportation systems to activity planning. The modular approach enables users of the software to focus on their area of interest, whereas other functional modules can be regarded as black boxes. Specific attention is paid to a standardization of data inputs and outputs for traffic simulations. Such standardization will allow the sharing of data with many existing commercial simulation packages.

Improving the design of online applications for social benefit through a behaviour change model

Community support agencies routinely employ a web presence to provide information on their services. While this online information provision helps to increase an agency’s reach, this paper argues that it can be further extended by mapping relationships between services and by facilitating two-way communication and collaboration with local communities. We argue that emergent technologies, such as locative media and networking tools, can assist in harnessing this social capital. However, new applications must be designed in ways that both persuade and support community members to contribute information and support others in need. An analysis of the online presence of community service agencies and social benefit applications is presented against Fogg’s Behaviour Model. From this evaluation, design principles are proposed for developing new locative, collaborative online applications for social benefit.

Course evaluation : reconfigurations for learning with learning management systems

The introduction of online delivery platforms such as learning management systems (LMS) in tertiary education has changed the methods and modes of curriculum delivery and communication. While course evaluation methods have also changed from paper-based in-class-administered methods to largely online-administered methods, the data collection instruments have remained unchanged. This paper reports on a small exploratory study of two tertiary-level courses. The study investigated why design of the instruments and methods to administer surveys in the courses are ineffective measures against the intrinsic characteristics of online learning. It reviewed the students' response rates of the conventional evaluations for the courses over an eight-year period. It then compared a newly developed online evaluation and the conventional methods over a two-year period. The results showed the response rates with the new evaluation method increased by more than 80% from the average of the conventional evaluations (below 30%), and the students' written feedback was more detailed and comprehensive than in the conventional evaluations. The study demonstrated the possibility that the LMS-based learning evaluation can be effective and efficient in terms of the quality of students' participation and engagement in their learning, and for an integrated pedagogical approach in an online learning environment.

ITSM ProcessGuide – a longitudinal and multi-method field study for real-world DSR artifact evaluation

Process guidance supports users to increase their process model understanding, process execution effectiveness as well as efficiency, and process compliance performance. This paper presents a research in progress encompassing our ongoing DSR project on Process Guidance Systems and a field evaluation of the resulting artifact in cooperation with a company. Building on three theory-grounded design principles, a Process Guidance System artifact for the company’s IT service ticketing process is developed, deployed and used. Fol-lowing a multi-method approach, we plan to evaluate the artifact in a longitudinal field study. Thereby, we will not only gather self-reported but also real usage data. This article describes the development of the artifact and discusses an innovative evaluation approach.

Environmental indicator frameworks to design and assess environmental monitoring programs

Impact Assessment and Project Appraisal, vol. 22, n.1, March 2004, p. 47–62

Evaluation of the agricultural vocational training programmes conducted by the Krishi Vigyan Kendras (Farm Science Centres) in Indian Punjab

Krishin Vigyan Kendras-KVKs (Farm Science Centres) have been established by the Indian Council of Agricultural Research in 569 districts. The trust areas of KVKs are refinement and demonstration of technologies, and training of farmers and extension functionaries. Imparting vocational trainings in agriculture and allied fields for the rural youth is one of its mandates. The study was undertaken to do a formative and summative (outcome and impact) evaluation of the beekeeping and mushroom growing vocational training programmes in the Indian state of Punjab. One-group pre and post evaluation design was employed for conducting a formative and outcome evaluation. The knowledge tests were administered to 35 beekeeping and 25 mushroom cultivation trainees, before and after the training programmes organized in 2004. The trainees significantly gained in knowledge. A separate sample of 640 trainees, trained prior to 2004, was selected for finding the adoption status. Out of 640, a sample of 200 was selected by proportionate sampling technique out of three categories, namely: non-adopters, discontinued-adopters and continued-adopters for evaluating the long-term impact of these training programmes. Ex-post-facto one-shot case study design was applied for this impact analysis. The vocational training programmes have resulted in continued-adoption of beekeeping and mushroom cultivation enterprises by 20% and 51% trained farmers, respectively. Age and trainee occupation had significant influence on the adoption decision of beekeeping vocation, whereas education and family income significantly affected the adoption decision of mushroom cultivation. The continued adopters of beekeeping and mushroom growing had increased their family income by 49% and 24%, respectively. These training programmes are augmenting the dwindling farm income of the farmers in Indian Punjab.

Frontonasal dysplasia: clinical evaluation on audiological and brainstem electrophysiological profiles

Frontonasal dysplasia (FND) is a rare malformative complex affecting the frontal portion of the face, the eyes and the nose; it may occur singly or associated with other clinical signs. No systematic studies describing hearing in this condition were found. AIM: To evaluate hearing sensitivity and sound stimulus conduction from cochlea to brainstem in patients with clinical signs of FND. METHODS: 21 patients with isolated or syndromic FND were submitted to a clinical (otological/vestibular antecedents and otoscopy) and instrumental (pure tone and speech audiometry, tympanometry and brainstem auditory evoked response) hearing evaluation. DESIGN: A clinical, cross-sectional observational prospective study. RESULTS: Hearing thresholds were normal in 15 (70%) patients, abnormal in 5 (25%), mostly with conductive hearing loss; one patient did not cooperate with testing. The tympanometric curve was type A in 30 (72%) ears, type C in 5 (12%), type As in 4 (9%) and type B in 3 (7%). The auditory brainstem response (ABR) showed no abnormalities. CONCLUSION: Patients with FND showed no abnormalities in the auditory system from cochlea to brainstem in this study. Mild conductive hearing loss found in some is probably related to cleft palate. Further evaluation of hearing pathways at higher levels is recommended.

Effects of a health advocacy program to promote physical activity and nutrition behaviors among adolescents: An evaluation of the Young Leaders for Healthy Change program

Background. This study was designed to evaluate the effects of the Young Leaders for Healthy Change program, an internet-delivered program in the school setting that emphasized health advocacy skills-development, on nutrition and physical activity behaviors among older adolescents (13–18 years). The program consisted of online curricular modules, training modules, social media, peer and parental support, and a community service project. Module content was developed based on Social Cognitive Theory and known determinants of behavior for older adolescents. ^ Methods. Of the 283 students who participated in the fall 2011 YL program, 38 students participated in at least ten of the 12 weeks and were eligible for this study. This study used a single group-only pretest/posttest evaluation design. Participants were 68% female, 58% white/Caucasian, 74% 10th or 11th graders, and 89% mostly A and/or B students. The primary behavioral outcomes for this analysis were participation in 60-minutes of physical activity per day, 20-minutes of vigorous- or moderate- intensity physical activity (MVPA) participation per day, television and computer time, fruit and vegetable (FV) intake, sugar-sweetened beverage intake, and consumption of breakfast, home-cooked meals, and fast food. Other outcomes included knowledge, beliefs, and attitudes related to healthy eating, physical activity, and advocacy skills. ^ Findings. Among the 38 participants, no significant changes in any variables were observed. However, among those who did not previously meet behavioral goals there was an 89% increase in students who participated in more than 20 minutes of MVPA per day and a 58% increase in students who ate home-cooked meals 5–7 days per week. The majority of participants met program goals related to knowledge, beliefs, and attitudes prior to the start of the program. Participants reported either maintaining or improving to the goal at posttest for all items except FV intake knowledge, taste and affordability of healthy foods, interest in teaching others about being healthy, and ease of finding ways to advocate in the community. ^ Conclusions. The results of this evaluation indicated that promoting healthy behaviors requires different strategies than maintaining healthy behaviors among high school students. In the school setting, programs need to target the promotion and maintenance of health behaviors to engage all students who participate in the program as part of a class or club activity. Tailoring the program using screening and modifying strategies to meet the needs of all students may increase the potential reach of the program. The Transtheoretical Model may provide information on how to develop a tailored program. Additional research on how to utilize the constructs of TTM effectively among high school students needs to be conducted. Further evaluation studies should employ a more expansive evaluation to assess the long-term effectiveness of health advocacy programming.^

Evaluation forum /

"A review journal for employment and training professionals focusing on policy analysis and evaluation research in education, occupational training, work and welfare, and economic development."

Evaluation of a Post Discharge Intervention on 30-Day Re-admission and Emergency Room Use in Patients with Diabetes A Pilot Program

Thesis (Master's)--University of Washington, 2016-06