Illuminative evaluation as a method applied to Australian Government policy borrowing and implementation in higher education

Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.

Submission to KPMG review of the Federal Government Initiative to “analyse dementia program and services funded by the Department of Social Services"

The comments I make are based on my nearly twenty years involvement in the dementia cause at both a national and international level. In preparation, I read two papers namely the Ministerial Dementia Forum – Option Paper produced by KPMG Management Consultants (2014) and Analysis of Dementia Programmes and Services Funded by the Department of Social Services: Conversation Starter prepared by KPMG as a preparation document for those attending a workshop in Brisbane on April 22nd 2015. Dementia is a complex “syndrome” and as is often said, “when you meet one person with dementia, you have met one” meaning that no two persons with dementia are the same. Even in dementia care, Australia is a “lucky country” and there is much to be said for the quality and diversity of dementia care available for people living with dementia. Despite this, I agree with the many views expressed in the material I read that there is scope for improvement, especially in the way that services are coordinated. In saying that, I do not purport to have all the solutions nor claim to have the knowledge required to comment on all the programs covered by this review. If I appear to be a “biased” advocate for Alzheimer’s Australia across the States and Territories, it is because I have seen constant evidence of ordinary people doing extraordinary things with inadequate resources. Dementia care is not cheap and if those funding dementia services are primarily only interested in economic outcomes and benefits, the real purpose of this consultation will be defeated. In addition, nowhere in the material I have read is there any recognition that in many instances program funding is a complex mix of government (at all levels) and private funding. This makes reviewing those programs more complex and less able to be coordinated at a Departmental level. It goes without saying therefore that the Federal Government is not” the only player in this game”. Of all those participating in this review, Alzheimer’s Australia is best placed to comment on programs as it is more connected to people living with dementia and has probably the best record of consulting with them. It would appear however that their role has been reduced to that of a “bit player”. Without wanting to be critical, the Forum Report which deals with the comments made at a gathering of 70 individuals and organisations, only three (3) or 4.28% were actual carers of people living with dementia. Even if it is argued that a number of organisations present represented consumers, the percentage goes up only marginally to 8.57% which is hardly an endorsement of the forum being “consumer driven”. The predominance of those present were service providers, each with their own agenda and each seeking advantage for their “business”. The final point I want to make before commenting on more specific, program related issues, is that many programs being reviewed have a much longer history than is reflected in the material I have read. Their growth and development was pioneered by Alzheimer’s Australia organisations across the country often with no government funding. Attempts to bring about better coordination of programs were often at the behest of Alzheimer’s Australia but in the main were ignored. The opportunity to now put this right is long overdue.

[await full-text]Evaluation of QAS Workplace Health and Safety Action Plan

This report identifies the outcomes of a program evaluation of the five year Workplace Health and Safety Strategy (2012-2017), specifically, the engagement component within the Queensland Ambulance Service. As part of the former Department of Community Safety, their objective was to work towards harmonising the occupational health and safety policies and process to improve the workplace culture. The report examines and assess the process paths and resource inputs into the strategy, provides feedback on progress to achieving identified goals as well as identify opportunities for improvements and barriers to progress. Consultations were held with key stakeholders within QAS and focus groups were facilitated with managers and health and safety representatives of each Local Area Service Network.

Context-sensitive evaluation: Determining the context surrounding the implementation of a government policy

This article explains the essence of the context-sensitive parameters and dimensions in play at the time of an intervention, through the application of Rog’s (2012) model of contextual parameters. Rog’s model offers evaluators a structured approach to examine an intervention. The initial study provided a systematic way to clarify the scope, variables, timing, and appropriate evaluation methodology to evaluate the implementation of a government policy. Given that the government implementation of an educational intervention under study did not follow the experimental research approach, nor the double cycle of action research approach, the application of Rog’s model provided an in-depth understanding of the context-sensitive environment; it is from this clear purpose that the broader evaluation was conducted. Overall, when governments or institutions implement policy to invoke educational change (and this intervention is not guided by an appropriate evaluation approach), then program evaluation is achievable post-implementation. In this situation, Rog’s (2012) model of contextual parameters is a useful way to achieve clarity of purpose to guide the program evaluation.

Report on Red Light Camera Program: Operation from July 1988 to December 1989

EXECUTIVE SUMMARY (excerpts) The red light camera (RLC) program commenced in July 1988, with five cameras operating at 15 sites in metropolitan Adelaide. This report deals with the first eighteen months of operation, to December 1989. A number of recommendations have been made… PROGRAM EVALUATION … In 1989 dollars, the program was estimated to have achieved an accident reduction benefit of $1.4m in the first 12 months of operation, which is almost twice the benefit expected using the assumptions made when selecting the sites. (There are 8 recommendations, mostly specific to the particular program characteristics)

Evaluation of a workplace alcohol and other drug policy

BACKGROUND The current impetus for developing alcohol and/or other drugs (AODs) workplace policies in Australia is to reduce workplace AOD impairment, improve safety, and prevent AOD-related injury in the workplace. For these policies to be effective, they need to be informed by scientific evidence. Evidence to inform the development and implementation of effective workplace AOD policies is currently lacking. There does not currently appear to be conclusive evidence for the effectiveness of workplace AOD policies in reducing impairment and preventing AOD-related injury. There is also no apparent evidence regarding which factors facilitate or impede the success of an AOD policy, or whether, for example, unsuccessful policy outcomes were due to poor policy or merely poor implementation of the policy. It was the aim of this research to undertake a process, impact, and outcome evaluation of a workplace AOD policy, and to contribute to the body of knowledge on the development and implementation of effective workplace AOD policies. METHODS The research setting was a state-based power-generating industry in Australia between May 2008 and May 2010. Participants for the process evaluation study were individuals who were integral to either the development or the implementation of the workplace AOD policy, or both of these processes (key informants), and comprised the majority of individuals who were involved in the process of developing and/or implementing the workplace AOD policy. The sample represented the two main groups of interest—management and union delegates/employee representatives—from all three of the participating organisations. For the impact and outcome evaluation studies, the population included all employees from the three participating organisations, and participants were all employees who consented to participate in the study and who completed both the pre-and post-policy implementation questionnaires. Qualitative methods in the form of interviews with key stakeholders were used to evaluate the process of developing and implementing the workplace AOD policy. In order to evaluate the impact of the policy with regard to the risk factors for workplace AOD impairment, and the outcome of the policy in terms of reducing workplace AOD impairment, quantitative methods in the form of a non-randomised single group pre- and post-test design were used. Changes from Time 1 (pre) to Time 2 (post) in the risk factors for workplace AOD impairment, and changes in the behaviour of interest—(self-reported) workplace AOD impairment—were measured. An integration of the findings from the process, impact, and outcome evaluation studies was undertaken using a combination of qualitative and quantitative methods. RESULTS For the process evaluation study Study respondents indicated that their policy was developed in the context of comparable industries across Australia developing workplace AOD policies, and that this was mainly out of concern for the deleterious health and safety impacts of workplace AOD impairment. Results from the process evaluation study also indicated that in developing and implementing the workplace AOD policy, there were mainly ‗winners', in terms of health and safety in the workplace. While there were some components of the development and implementation of the policy that were better done than others, and the process was expensive and took a long time, there were, overall, few unanticipated consequences to implementing the policy and it was reported to be thorough and of a high standard. Findings also indicated that overall the policy was developed and implemented according to best-practice in that: consultation during the policy development phase (with all the main stakeholders) was extensive; the policy was comprehensive; there was universal application of the policy to all employees; changes in the workplace (with regard to the policy) were gradual; and, the policy was publicised appropriately. Furthermore, study participants' responses indicated that the role of an independent external expert, who was trusted by all stakeholders, was integral to the success of the policy. For the impact and outcome evaluation studies Notwithstanding the limitations of pre- and post-test study designs with regard to attributing cause to the intervention, the findings from the impact evaluation study indicated that following policy implementation, statistically significant positive changes with regard to workplace AOD impairment were recorded for the following variables (risk factors for workplace AOD impairment): Knowledge; Attitudes; Perceived Behavioural Control; Perceptions of the Certainty of being punished for coming to work impaired by AODs; Perceptions of the Swiftness of punishment for coming to work impaired by AODs; and Direct and Indirect Experience with Punishment Avoidance for workplace AOD impairment. There were, however, no statistically significant positive changes following policy implementation for Behavioural Intentions, Subjective Norms, and Perceptions of the Severity of punishment for workplace AOD impairment. With regard to the outcome evaluation, there was a statistically significant reduction in self-reported workplace AOD impairment following the implementation of the policy. As with the impact evaluation, these findings need to be interpreted in light of the limitations of the study design in being able to attribute cause to the intervention alone. The findings from the outcome evaluation study also showed that while a positive change in self-reported workplace AOD impairment following implementation of the policy did not appear to be related to gender, age group, or employment type, it did appear to be related to levels of employee general alcohol use, cannabis use, site type, and employment role. Integration of the process, impact, and outcome evaluation studies There appeared to be qualitative support for the relationship between the process of developing and implementing the policy, and the impact of the policy in changing the risk factors for workplace AOD impairment. That is, overall the workplace AOD policy was developed and implemented well and, following its implementation, there were positive changes in the majority of measured risk factors for workplace AOD impairment. Quantitative findings lend further support for a relationship between the process and impact of the policy, in that there was a statistically significant association between employee perceived fidelity of the policy (related to the process of the policy) and positive changes in some risk factors for workplace AOD impairment (representing the impact of the policy). Findings also indicated support for the relationship between the impact of the policy in changing the risk factors for workplace AOD impairment and the outcome of the policy in reducing workplace AOD impairment: positive changes in the risk factors for workplace AOD impairment (impact) were related to positive changes in self reported workplace AOD impairment (representing the main goal and outcome of the policy). CONCLUSIONS The findings from the research indicate support for the conclusion that the policy was appropriately implemented and that it achieved its objectives and main goal. The Doctoral research findings also addressed a number of gaps in the literature on workplace AOD impairment, namely: the likely effectiveness of AOD policies for reducing AOD impairment in the workplace, which factors in the development and implementation of a workplace AOD policy are likely to facilitate or impede the effectiveness of the policy to reduce workplace AOD impairment, and which employee groups are less likely to respond well to policies of this type. The findings from this research not only represent an example of translational, applied research—through the evaluation of the study industry's policy—but also add to the body of knowledge on workplace AOD policies and provide policy-makers with evidence which may be useful in the development and implementation of effective workplace AOD policies. Importantly, the findings espouse the importance of scientific evidence in the development, implementation, and evaluation of workplace AOD policies.

Student perception of a new integrated anatomy practical program: Does students' prior learning make a difference?

While there is evidence that science and non-science background students display small differences in performance in basic and clinical sciences, early in a 4-year, graduate entry medical program, this lessens with time. With respect to anatomy knowledge, there are no comparable data as to the impact previous anatomy experience has on the student perception of the anatomy practical learning environment. A study survey was designed to evaluate student perception of the anatomy practical program and its impact on student learning, for the initial cohort of a new medical school. The survey comprised 19 statements requiring a response using a 5-point Likert scale, in addition to a free text opportunity to provide opinion of the perceived educational value of the anatomy practical program. The response rate for a total cohort of 82 students was 89%. The anatomy practical program was highly valued by the students in aiding their learning of anatomy, as indicated by the high mean scores for all statements (range: 4.04-4.7). There was a significant difference between the students who had and had not studied a science course prior to entering medicine, with respect to statements that addressed aspects of the course related to its structure, organization, variety of resources, linkage to problem-based learning cases, and fairness of assessment. Nonscience students were more positive compared to those who had studied science before (P levels ranging from 0.004 to 0.035). Students less experienced in anatomy were more challenged in prioritizing core curricular knowledge. © 2011 Wiley-Liss, Inc.

A cross-site, comparative effectiveness study of an integrated HIV and substance use treatment program.

Co-occurrence of HIV and substance abuse is associated with poor outcomes for HIV-related health and substance use. Integration of substance use and medical care holds promise for HIV patients, yet few integrated treatment models have been reported. Most of the reported models lack data on treatment outcomes in diverse settings. This study examined the substance use outcomes of an integrated treatment model for patients with both HIV and substance use at three different clinics. Sites differed by type and degree of integration, with one integrated academic medical center, one co-located academic medical center, and one co-located community health center. Participants (n=286) received integrated substance use and HIV treatment for 12 months and were interviewed at 6-month intervals. We used linear generalized estimating equation regression analysis to examine changes in Addiction Severity Index (ASI) alcohol and drug severity scores. To test whether our treatment was differentially effective across sites, we compared a full model including site by time point interaction terms to a reduced model including only site fixed effects. Alcohol severity scores decreased significantly at 6 and 12 months. Drug severity scores decreased significantly at 12 months. Once baseline severity variation was incorporated into the model, there was no evidence of variation in alcohol or drug score changes by site. Substance use outcomes did not differ by age, gender, income, or race. This integrated treatment model offers an option for treating diverse patients with HIV and substance use in a variety of clinic settings. Studies with control groups are needed to confirm these findings.

A Pharmacology-Based Enrichment Program for Undergraduates Promotes Interest in Science.

There is a strong need to increase the number of undergraduate students who pursue careers in science to provide the "fuel" that will power a science and technology-driven U.S. economy. Prior research suggests that both evidence-based teaching methods and early undergraduate research experiences may help to increase retention rates in the sciences. In this study, we examined the effect of a program that included 1) a Summer enrichment 2-wk minicourse and 2) an authentic Fall research course, both of which were designed specifically to support students' science motivation. Undergraduates who participated in the pharmacology-based enrichment program significantly improved their knowledge of basic biology and chemistry concepts; reported high levels of science motivation; and were likely to major in a biological, chemical, or biomedical field. Additionally, program participants who decided to major in biology or chemistry were significantly more likely to choose a pharmacology concentration than those majoring in biology or chemistry who did not participate in the enrichment program. Thus, by supporting students' science motivation, we can increase the number of students who are interested in science and science careers.

An outpatient respiratory rehabilitation program:Findings from a community hospital

Objectives: To determine patient satisfaction with a community hospital's respiratory rehabilitation program and to assess changes in patient physical and emotional function and quality of life. Design: Pre- and post-program measures were made on a variety of physiological and psychosocial factors. A modified version of the Chronic Respiratory Disease Questionnaire was administered before and after the 8-week multidisciplinary and comprehensive respiratory rehabilitation program. The post-program questionnaire also included a number of service delivery and patient satisfaction and quality-of-life questions. Setting: Respiratory Rehabilitation Program at St. Joseph's Hospital, a community hospital in Brantford, Ont., in active partnership with the Brant County Lung Association. Brant County is located in Central West Ontario, and has both urban and rural areas and a population of approximately 125 000 people. Participants: Twenty-nine patients, with a diagnosis of moderate to severe chronic obstructive pulmonary disease (COPD) who were referred to the Fall 1997 and Spring 1998 programs, were enrolled in the study. Outcome measures: Changes in physical and emotional function, health knowledge, skills mastery, quality of life and satisfaction with the program. Results: Twenty-one of 29 patients completed the program. Statistically significant and clinically important improvements were found between all pre- and post-program evaluation scores (distance walked, fatigue, dyspnea, emotional function, skills mastery and health knowledge). Participants were very satisfied with the program and felt it improved their quality of life. Conclusion: The positive outcomes reported rom randomized controlled trials of respiratory rehabilitation programs can be achieved in a community hospital setting.

Factors affecting the successful implementation and sustainability of the Liverpool Care Pathway for dying patients – a realist evaluation

Objectives: The Liverpool Care Pathway for the dying patient (LCP) was designed to improve end-of-life care in generalist health care settings. Controversy has led to its withdrawal in some jurisdictions. The main objective of this research was to identify the influences that facilitated or hindered successful LCP implementation.

Method: An organisational case study using realist evaluation in one health and social care trust in Northern Ireland. Two rounds of semi-structured interviews were conducted with two policy makers and twenty two participants with experience and/or involvement in management of the LCP during 2011 and 2012.

Results: Key resource inputs included facilitation with a view to maintaining LCP ‘visibility’, reducing anxiety among nurses and increasing their confidence regarding the delivery of end-of-life care; and nurse and medical education designed to increase professional self-efficacy and reduce misuse and misunderstanding of the LCP. Key enabling contexts were consistent senior management support; ongoing education and training tailored to the needs of each professional group; and an organisational cultural change in the hospital setting that encompassed end-of-life care.

Conclusion: There is a need to appreciate the organizationally complex nature of intervening to improve end-of-life care. Successful implementation of evidence-based interventions for end-of-life care requires commitment to planning, training and ongoing review that takes account of different perspectives, institutional hierarchies and relationships and the educational needs of professional disciplines. There is a need also to recognise that medical consultants require particular support in their role as gatekeepers and as a lead communication channel with patients and their relatives.

Evaluation of a partnership between primary and secondary care providing an accessible Level 1 sexual health service in the community

Comprehensive testing for asymptomatic sexually transmitted infections in Northern Ireland has traditionally been provided by genitourinary medicine clinics. As patient demand for services has increased while budgets have remained limited, there has been increasing difficulty in accommodating this demand. In May 2013, the newly commissioned specialist Sexual Health service in the South Eastern Trust sought to pilot a new model of care working alongside a GP partnership of 12 practices. A training programme to enable GPs and practice nurses to deliver Level 1 sexual health care to heterosexual patients aged >16 years, in accordance with the standards of BASHH, was developed. A comprehensive care pathway and dedicated community health advisor supported this new model with close liaison between primary and secondary care. Testing for Chlamydia, gonorrhoea, HIV and syphilis was offered. The aims of the pilot were achieved, namely to provide accessible, cost-effective sexual health care within a framework of robust clinical governance. Furthermore, it uncovered a high positivity rate for Chlamydia, especially in young men attending their general practice, and demonstrated a high level of patient satisfaction. Moreover the capacity of secondary care to deliver Levels 2 and 3 services was increased.

Development, implementation, and evaluation of the Apollo model of pediatric rehabilitation service delivery

This article presents the experience of a rehabilitation program that un- dertook the challenge to reorganize its services to address accessibility issues and im- prove service quality. The context in which the reorganization process occurred, along with the relevant literature justifying the need for a new service delivery model, and an historical perspective on the planning; implementation; and evaluation phases of the process are described. In the planning phase, the constitution of the working committee, the data collected, and the information found in the literature are presented. Apollo, the new service delivery model, is then described along with each of its components (e.g., community, group, and individual interventions). Actions and lessons learnt during the implementation of each component are presented. We hope by sharing our experiences that we can help others make informed decisions about service reorganization to im- prove the quality of services provided to children with disabilities, their families, and their communities.