Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

Teacher Evidence Matrix : an evaluation using multiple lines of evidence

Welcome to the Teacher evidence matrix. This matrix is designed for highly qualified discipline experts to evaluate their teaching in a systematic manner. The primary purpose of the Teacher evidence matrix is to provide a tool that an academic staff member at university can annually review their teaching. The annual review will result in you being ready for performance, planning and review; promotion; awards; or employment application. This tool is designed for individual use and will lead to an action plan for implementation.

An evaluation of medication review reports across different settings

Background There is a growing body of evidence which supports that a pharmacist conducted medication review increases the health outcomes for patients. A pharmacist integrated into a primary care medical centre may offer many potential advantages in conducting medication reviews in this setting however research describing this is presently limited. Objective To compare medication review reports conducted by pharmacists practicing externally to a medical centre to those medication review reports conducted by an integrated practice pharmacist. The secondary objective was to compare medication review reports conducted by pharmacists in the patient’s home to those conducted in the medical centre. Setting A primary care medical centre, Brisbane, Australia Method A retrospective analysis of pharmacist conducted medication reviews prior to and after the integration of a pharmacist into a medical centre. Main outcome measures Types of drug related problems identified by the Pharma cists, recommended intervention for drug related problems made by the pharmacist, and the extent of implementation of pharmacist recommendations by the general practitioner. Results The primary drug related problem reported in the practice pharmacist phase was Additional therapy required as compared to Precautions in the external pharmacist phase. The practice pharmacist most frequently recommended to add drug with Additional monitoring recommended most often in the external pharmacists. During the practice pharmacist phase 71 % of recommendations were implemented and was significantly higher than the external pharmacist phase with 53 % of recommendations implemented (p\0.0001). Two of the 23 drug related problem domains differed significantly when comparing medication reviews conducted in the patient’s home to those conducted in the medical centre.

Professional conversations : a collaborative approach to support policy implementation, professional learning and practice change in ECEC

In 2012, Australia introduced a new National Quality Framework, comprising enhanced quality expectations for early childhood education and care services, two national learning frameworks and a new Assessment and Rating System spanning child care centres, kindergartens and preschools, family day care and outside school hours care. This is the linchpin in a series of education reforms designed to support increased access to higher quality early childhood education and care (ECEC) and successful transition to school. As with any policy change, success in real terms relies upon building shared understanding and the capacity of educators to apply new knowledge and to support change and improved practice within their service. With this in mind, a collaborative research project investigated the efficacy of a new approach to professional learning in ECEC: the professional conversation. This paper reports on the trial and evaluation of a series of professional conversations to support implementation of one element of the NQF, the Early Years Learning Framework (DEEWR,2009), and their capacity to promote collaborative reflective practice, shared understanding, and improved practice in ECEC. Set against the backdrop of the NQF, this paper details the professional conversation approach, key challenges and critical success factors, and the learning outcomes for conversation participants. Findings support the efficacy of this approach to professional learning in ECEC, and its capacity to support policy reform and practice change in ECEC.

Security analysis of Australian and E.U. e-passport implementation

This paper makes a formal security analysis of the current Australian e-passport implementation using model checking tools CASPER/CSP/FDR. We highlight security issues in the current implementation and identify new threats when an e-passport system is integrated with an automated processing system like SmartGate. The paper also provides a security analysis of the European Union (EU) proposal for Extended Access Control (EAC) that is intended to provide improved security in protecting biometric information of the e-passport bearer. The current e-passport specification fails to provide a list of adequate security goals that could be used for security evaluation. We fill this gap; we present a collection of security goals for evaluation of e-passport protocols. Our analysis confirms existing security weaknesses that were previously identified and shows that both the Australian e-passport implementation and the EU proposal fail to address many security and privacy aspects that are paramount in implementing a secure border control mechanism. ACM Classification C.2.2 (Communication/Networking and Information Technology – Network Protocols – Model Checking), D.2.4 (Software Engineering – Software/Program Verification – Formal Methods), D.4.6 (Operating Systems – Security and Privacy Protection – Authentication)

Formal security analysis of Australian e-passport implementation

This paper provides a detailed description of the current Australian e-passport implementation and makes a formal verification using model checking tools CASPER/CSP/FDR. We highlight security issues present in the current e-passport implementation and identify new threats when an e-passport system is integrated with an automated processing systems like SmartGate. Because the current e-passport specification does not provide adequate security goals, to perform a rational security analysis we identify and describe a set of security goals for evaluation of e-passport protocols. Our analysis confirms existing security issues that were previously informally identified and presents weaknesses that exists in the current e-passport implementation.

Causal behavioural profiles : efficient computation, applications, and evaluation

Analysis of behavioural consistency is an important aspect of software engineering. In process and service management, consistency verification of behavioural models has manifold applications. For instance, a business process model used as system specification and a corresponding workflow model used as implementation have to be consistent. Another example would be the analysis to what degree a process log of executed business operations is consistent with the corresponding normative process model. Typically, existing notions of behaviour equivalence, such as bisimulation and trace equivalence, are applied as consistency notions. Still, these notions are exponential in computation and yield a Boolean result. In many cases, however, a quantification of behavioural deviation is needed along with concepts to isolate the source of deviation. In this article, we propose causal behavioural profiles as the basis for a consistency notion. These profiles capture essential behavioural information, such as order, exclusiveness, and causality between pairs of activities of a process model. Consistency based on these profiles is weaker than trace equivalence, but can be computed efficiently for a broad class of models. In this article, we introduce techniques for the computation of causal behavioural profiles using structural decomposition techniques for sound free-choice workflow systems if unstructured net fragments are acyclic or can be traced back to S- or T-nets. We also elaborate on the findings of applying our technique to three industry model collections.

Evaluation of a community-based intervention to promote physical activity in youth : lessons from active winners

Purpose To test the effects of a community-based physical activity intervention designed to increase physical activity and to conduct an extensive process evaluation of the intervention. Design Quasi-experimental. Setting Two rural communities in South Carolina. One community received the intervention, and the other served as the comparison. Subjects Public school students who were in fifth grade at the start of the study (558 at baseline) were eligible to participate. A total of 436 students participated over the course of the study. Intervention The intervention included after-school and summer physical activity programs and home, school, and community components designed to increase physical activity in youth. The intervention took place over an 18-month period. Measures. Students reported after-school physical activity at three data collection points (prior to, during, and following the intervention) using the Previous Day Physical Activity Recall (PDPAR). They also completed a questionnaire designed to measure hypothesized psychosocial and environmental determinants of physical activity behavior The process evaluation used meeting records, documentation of program activities, interviews, focus groups, and heart rate monitoring to evaluate the planning and implementation of the intervention. Results There were no significant differences in the physical activity variables and few significant differences in the psychosocial variables between the intervention and comparison groups. The process evaluation indicated that the after-school and summer physical activity component of the intervention was implemented as planned, but because of resource and time limitations, the home, school, and community components were not implemented as planned. Conclusions The intervention did not have a significant effect on physical activity in the target population of children in the intervention community. This outcome is similar to that reported in other studies of community-based physical activity intervention.

Part 2. MOO methods for multidisciplinary design using parallel evolutionary algorithms, game theory and hierarchical topology. (Part 2.) Numerical aspects and implementation of model test cases

These lecture notes describe the use and implementation of a framework in which mathematical as well as engineering optimisation problems can be analysed. The foundations of the framework and algorithms described -Hierarchical Asynchronous Parallel Evolutionary Algorithms (HAPEAs) - lie upon traditional evolution strategies and incorporate the concepts of a multi-objective optimisation, hierarchical topology, asynchronous evaluation of candidate solutions , parallel computing and game strategies. In a step by step approach, the numerical implementation of EAs and HAPEAs for solving multi criteria optimisation problems is conducted providing the reader with the knowledge to reproduce these hand on training in his – her- academic or industrial environment.

Reframe : QUT's Evaluation Framework 2014 Communications Plan

2014 is the year for embedding the Reframe: QUT’s Evaluation Framework as core business within the University, with the following providing an outline of the Learning and Teaching Unit’s (LTU) planned activities to support this implementation. LTU is continuing its commitment to intensive support and engagement with the academic and teaching focused community within QUT. The Academic Quality and Standards team have extended the existing communication and dissemination activities into a comprehensive communication plan for 2014, with major initiatives defined within this document. It should be noted that these activities are aligned to the discussions from the 2013 Integrated Management of Feedback (IMF) Steering Group and as endorsed by the University Learning and Teaching Committee at their meeting in February, 2014.

Implementation of sustainable evidence-based practice for the assessment and management of pain in residential aged care facilities

Pain is common in residential aged care facilities (RACFs). In 2005, the Australian Pain Society developed 27 recommendations for good practice in the identification, assessment, and management of pain in these settings. This study aimed to address implementation of the standards and evaluate outcomes. Five facilities in Australia participated in a comprehensive evaluation of RACF pain practice and outcomes. Pre-existing pain management practices were compared with the 27 recommendations, before an evidence-based pain management program was introduced that included training and education for staff and revised in-house pain-management procedures. Post-implementation audits evaluated the program's success. Aged care staff teams also were assessed on their reports of self-efficacy in pain management. The results show that before the implementation program, the RACFs demonstrated full compliance on 6 to 12 standards. By the project's completion, RACFs demonstrated full compliance with 10 to 23 standards and major improvements toward compliance in the remaining standards. After implementation, the staff also reported better understanding of the standards (p < .001) or of facility pain management guidelines (p < .001), increased confidence in therapies for pain management (p < .001), and increased confidence in their training to assess pain (p < .001) and recognize pain in residents with dementia who are nonverbal (p = .003). The results show that improved evidence-based practice in RACFs can be achieved with appropriate training and education. Investing resources in the aged care workforce via this implementation program has shown improvements in staff self-efficacy and practice.

An evaluation of the neocognitron

We describe a sequence of experiments investigating the strengths and limitations of Fukushima's neocognitron as a handwritten digit classifier. Using the results of these experiments as a foundation, we propose and evaluate improvements to Fukushima's original network in an effort to obtain higher recognition performance. The neocognitron's performance is shown to be strongly dependent on the choice of selectivity parameters and we present two methods to adjust these variables. Performance of the network under the more effective of the two new selectivity adjustment techniques suggests that the network fails to exploit the features that distinguish different classes of input data. To avoid this shortcoming, the network's final layer cells were replaced by a nonlinear classifier (a multilayer perceptron) to create a hybrid architecture. Tests of Fukushima's original system and the novel systems proposed in this paper suggest that it may be difficult for the neocognitron to achieve the performance of existing digit classifiers due to its reliance upon the supervisor's choice of selectivity parameters and training data. These findings pertain to Fukushima's implementation of the system and should not be seen as diminishing the practical significance of the concept of hierarchical feature extraction embodied in the neocognitron. © 1997 IEEE.

Learnings, case studies and guidelines for establishing shared and collaborative service delivery in the non-government sector: Evaluation of the Multi-Tenant Service Centre (MTSC) Pilots Project

This report presents learnings, case studies, guidelines and resources for non-government organisations that are planning to implement shared or collaborative arrangements with other agencies. It summarises results from an evaluation of the implementation phase of the Multi-Tenant Service Centre (MTSC) Pilots Project, which was completed in June 2008. This evaluation shows that developing and implementing shared and collaborative arrangements is a complex process that presents many risks, challenges and barriers to success, but can have many potential benefits for non government organisations. As this report makes clear, there is no ‘one size fits all’ approach to this process. The MTSC Pilots Project was conducted by the Department of Communities (DoC), Queensland Government, as part of its Strengthening Non-Government Organisations strategy. The objective of the MTSC Pilots initiative was to co-locate separate service providers in an appropriately located centre, operating with effective and transparent management, which enabled service providers to improve client services. Three MTSC consortiums in Mackay, Caboolture and Toowoomba were selected as the pilots over a four year period from 2006 – 2010.

Course evidence matrix : a collaborative evaluation using multiple lines of evidence

Welcome to the Evaluation of course matrix. This matrix is designed for highly qualified discipline experts to evaluate their course, major or unit in a systemic manner. The primary purpose of the Evaluation of course matrix is to provide a tool that a group of academic staff at universities can collaboratively review the assessment within a course, major or unit annually. The annual review will result in you being ready for an external curricula review at any point in time. This tool is designed for use in a workshop format with one, two or more academic staff, and will lead to an action plan for implementation. I hope you find this tool useful in your assessment review.

Improving wound management for residents in Residential Aged Care Facilities: National dissemination and implementation of the evidence based Champions for Skin Integrity Program

Overview The incidence of skin tears, pressure injuries and chronic wounds increases with age [1-4] and therefore is a serious issue for staff and residents in Residential Aged Care Facilities (RACFs). A pilot project funded in Round 2 of the Encouraging Best Practice in Residential Aged Care (EBPRAC) program by the then Australian Government Department of Health and Ageing found that a substantial proportion of residents in aged care facilities experienced pressure injuries, skin tears or chronic wounds. It also found the implementation of the evidence based Champions for Skin Integrity (CSI) model of wound care was successful in significantly decreasing the prevalence and severity of wounds in residents, improving staff skills and knowledge of evidence based wound management, increasing staff confidence with wound management, increasing implementation of evidence based wound management and prevention strategies, and increasing staff awareness of their roles in evidence based wound care at all levels [5]. Importantly, during the project, the project team developed a resource kit on evidence based wound management. Two critical recommendations resulting from the project were that: - The CSI model or a similar strategic approach should be implemented in RACFs to facilitate the uptake of evidence based wound management and prevention - The resource kit on evidence based wound management should be made available to all Residential Aged Care Facilities and interested parties A proposal to disseminate or rollout the CSI model of wound care to all RACFs across Australia was submitted to the department in 2012. The department approved funding from the Aged Care Services Improvement Healthy Ageing Grant (ACSIHAG) at the same time as the Round 3 of the Encouraging Better Practice in Aged Care (EBPAC) program. The dissemination involved two crucial elements: 1. The updating, refining and distribution of a Champions for Skin Integrity Resource Kit, more commonly known as a CSI Resource Kit and 2. The presentation of intensive one day Promoting Healthy Skin “Train the Trainer” workshops in all capital cities and major regional towns across Australia Due to demand, the department agreed to fund a second round of workshops focussing on regional centres and the completion date was extended to accommodate the workshops. Later, the department also decided to host a departmental website for a number of clinical domains, including wound management, so that staff from the residential aged care sector had easy access to a central repository of helpful clinical resource material that could be used for improving the health and wellbeing of their older adults, consumers and carers. CSI Resource Kit Upgrade and Distribution: At the start of the project, a full evidence review was carried out on the material produced during the EBPRAC-CSI Stage 1 project and the relevant evidence based changes were made to the documentation. At the same time participants in the EBPRAC-CSI Stage 1 project were interviewed for advice on how to improve the resource material. Following this the documentation, included in the kit, was sent to independent experts for peer review. When this process was finalised, a learning designer and QUT’s Visual Communications Services were engaged to completely refine and update the design of the resources, and combined resource kit with the goal of keeping the overall size of the kit suitable for bookshelf mounting and the cost at reasonable levels. Both goals were achieved in that the kit is about the same size as a 25 mm A4 binder and costs between $19.00 and $28.00 per kit depending on the size of the print run. The dissemination of the updated CSI resource kit was an outstanding success. Demand for the kits was so great that a second print run of 2,000 kits was arranged on top of the initial print run of 4,000 kits. All RACFs across Australia were issued with a kit, some 2,740 in total. Since the initial distribution another 1,100 requests for kits has been fulfilled as well as 1,619 kits being distributed to participants at the Promoting Healthy Skin workshops. As the project was winding up a final request email was sent to all workshop participants asking if they required additional kits or resources to distribute the remaining kits and resources. This has resulted in requests for 200 additional kits and resources. Feedback from the residential aged care sector and other clinical providers who have interest in wound care has been very positive regarding the utility of the kit, (see Appendix 4). Promoting Healthy Skin Workshops The workshops also exceeded the project team’s initial objective. Our goal of providing workshop training for staff from one in four facilities and 450 participants was exceeded, with overwhelming demand for workshop places resulting in the need to provide a second round of workshops across Australia. At the completion of the second round, 37 workshops had been given, with 1286 participants, representing 835 facilities. A number of strategies were used to promote the workshops ranging from invitations included in the kit, to postcard mail-outs, broadcast emailing to all facilities and aged care networks and to articles and paid advertising in aged care journals. The most effective method, by far, was directly phoning the facilities. This enabled the caller to contact the relevant staff member and enlist their support for the workshop. As this is a labour intensive exercise, it was only used where numbers needed bolstering, with one venue rising from 3 registrants before the calls to 53 registrants after. The workshops were aimed at staff who had the interest and the capability of implementing evidence-based wound management within their facility or organisation. This targeting was successful in that a large proportion (68%) of participants were Registered Nurses, Nurse Managers, Educators or Consultants. Twenty percent were Endorsed Enrolled Nurses with the remaining 12% being made up of Personal Care Workers or Allied Health Professionals. To facilitate long term sustainability, the workshop employed train-the-trainer strategies. Feedback from the EBPRAC-CSI Stage 1 interviews was used in the development of workshop content. In addition, feedback from the workshop conducted at the end of the EBPRAC-CSI Stage 1 project suggested that change management and leadership training should be included in the workshops. The program was trialled in the first workshop conducted in Brisbane and then rolled out across Australia. Participants were asked to complete pre and post workshop surveys at the beginning and end of the workshop to determine how knowledge and confidence improved over the day. Results from the pre and post surveys showed significant improvements in the level of confidence in attendees’ ability to implement evidence based wound management. The results also indicated a significant increase in the level of confidence in ability to implement change within their facility or organisation. This is an important indication that the inclusion of change management/leadership training with clinical instruction can increase staff capacity and confidence in translating evidence into practice. To encourage the transfer of the evidence based content of the workshop into practice, participants were asked to prepare an Action Plan to be followed by a simple one page progress report three months after the workshop. These reports ranged from simple (e.g. skin moisturising to prevent skin tears), to complex implementation plans for introducing the CSI model across the whole organisation. Outcomes described in the project reports included decreased prevalence of skin tears, pressure injuries and chronic wounds, along with increased staff and resident knowledge and resident comfort. As stated above, some organisations prepared large, complex plans to roll out the CSI model across their organisation. These plans included a review of the organisation’s wound care system, policies and procedures, the creation of new processes, the education of staff and clients, uploading education and resource material onto internal electronic platforms and setting up formal review and evaluation processes. The CSI Resources have been enthusiastically sought and incorporated into multiple health care settings, including aged care, acute care, Medicare Local intranets (e.g. Map of Medicine e-pathways), primary health care, community and home care organisations, education providers and New Zealand aged and community health providers. Recommendations: Recommendations for RACFs, aged care and health service providers and government  Skin integrity and the evidence-practice gap in this area should be recognised as a major health issue for health service providers for older adults, with wounds experienced by up to 50% of residents in aged care settings (Edwards et al. 2010). Implementation of evidence based wound care through the Champions for Skin Integrity model in this and the pilot project has demonstrated the prevalence of wounds, wound healing times and wound infections can be halved.  A national program and Centre for Evidence Based Wound Management should be established to: - expand the reach of the model to other aged care facilities and health service providers for older adults - sustain the uptake of models such as the Champions for Skin Integrity (CSI) model - ensure current resources, expertise and training are available for consumers and health care professionals to promote skin integrity for all older adults  Evidence based resources for the CSI program and similar projects should be reviewed and updated every 3 – 4 years as per NH&MRC recommendations  Leadership and change management training is fundamental to increasing staff capacity, at all levels, to promote within-organisation dissemination of skills and knowledge gained from projects providing evidence based training Recommendations for future national dissemination projects  A formal program of opportunities for small groups of like projects to share information and resources, coordinate activities and synergise education programs interactively would benefit future national dissemination projects - Future workshop programs could explore an incentive program to optimise attendance and reduce ‘no shows’ - Future projects should build in the capacity and funding for increased follow-up with workshop attendees, to explore the reasons behind those who are unable to translate workshop learnings into the workplace and identify factors to address these barriers.