Increasing the rigour and trustworthiness of participatory evaluations : learnings from the field

Participatory evaluation and participatory action research (PAR) are increasingly used in community-based programs and initiatives and there is a growing acknowledgement of their value. These methodologies focus more on knowledge generated and constructed through lived experience than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that acknowledges and takes into account the diverse perspectives, values and interpretations of participants and evaluation professionals. However, evaluation rigour need not be lost in this approach. Increasing the rigour and trustworthiness of participatory evaluations and PAR increases the likelihood that results are seen as credible and are used to continually improve programs and policies.----- Drawing on learnings and critical reflections about the use of feminist and participatory forms of evaluation and PAR over a 10-year period, significant sources of rigour identified include:----- • participation and communication methods that develop relations of mutual trust and open communication----- • using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection----- • ongoing meta-evaluation and critical reflection----- • critically assessing the intended and unintended impacts of evaluations, using relevant theoretical models----- • using rigorous data analysis and reporting processes----- • participant reviews of evaluation case studies, impact assessments and reports.

Troubling Empowerment : A feminist critique of a rural women and communication technology project

Participatory research methodologies and interactive communication technologies (ICTs) are increasingly seen as offering ways of enhancing women’s empowerment and rural community development. However, some researchers suggest the need for caution about such claims. This book details findings from an evaluation of a feminist action research project that explored the impacts of ICTs for rural women in Queensland, Australia, in terms of personal, business and community development. Using praxis and poststructuralist feminist theories and methodologies, this innovative study presents a rigorous analysis and critique of women's empowerment and participation. This study demonstrates the value of adopting a critical yet pragmatic approach that takes diversity and difference, power-knowledge relations, and the contradictory effects of participation into account. This is argued to enable the development of more effective strategies for women’s empowerment, participation and inclusion. This book should be of particular interest to researchers, postgraduate students, and others working in the fields of communication, gender, and rural development, and feminist evaluation and ethnography.

Building research capacity in the nursing workforce : the design and evaluation of the nurse researcher role

Objectives: The Nurse Researcher Project (NRP) was initiated to support development of a nursing research and evidence based practice culture in Cancer Care Services (CCS) in a large tertiary hospital in Australia. The position was established and evaluated to inform future directions in the organisation.---------- Background: The demand for quality cancer care has been expanding over the past decades. Nurses are well placed to make an impact on improving health outcomes of people affected by cancer. At the same time, there is a robust body of literature documenting the barriers to undertaking and utilising research by and for nurses and nursing. A number of strategies have been implemented to address these barriers including a range of staff researcher positions but there is scant attention to evaluating the outcomes of these strategies. The role of nurse researcher has been documented in the literature with the aim to provide support to nurses in the clinical setting. There is, to date, little information in relation to the design, implementation and evaluation of this role.---------- Design: The Donabedian’s model of program evaluation was used to implement and evaluate this initiative.---------- Methods: The ‘NRP’ outlined the steps needed to implement the nurse researcher role in a clinical setting. The steps involved the design of the role, planning for the support system for the role, and evaluation of outcomes of the role over two years.---------- Discussion: This paper proposes an innovative and feasible model to support clinical nursing research which would be relevant to a range of service areas.---------- Conclusion: Nurse researchers are able to play a crucial role in advancing nursing knowledge and facilitating evidence based practice, especially when placed to support a specialised team of nurses at a service level. This role can be implemented through appropriate planning of the position, building a support system and incorporating an evaluation plan.

Evolving directions in health promotion workforce development

PROJECT CONTEXT: Leaders in the fields of public health and health promotion increasingly advocate a socio-ecological approach to meet contemporary and emerging population health challenges. It is essential that health promotion workforce development initiatives mirror the evolving direction of the field to facilitate translation of theory into practice. To date, there has been limited effort to map the socio-ecological approach into tertiary education curricula. PROJECT DESCRIPTION: This project was undertaken as part of the development process for an undergraduate health promotion degree in Queensland, Australia. A review of the health promotion workforce development literature was undertaken. Group processes, key informant interviews and a Delphi technique were used to engage health promotion academics and practitioners, including an International Health Promotion Expert Advisory Panel, and an Industry Advisory Group in defining the components of the program. FINDINGS: The consultative processes facilitated the development of an undergraduate health promotion degree program underpinned by the socio-ecological approach with strong emphases upon the processes or 'how you do it' of health promotion together with evidence-based decision making and practice. CONCLUSIONS: As the basis and practice of health promotion progresses toward a socio-ecological approach, workforce training needs to keep pace with these developments to ensure an appropriately skilled health promotion workforce to meet emerging population health challenges. The reported project and the degree program that has been developed is an example of one step towards achieving this important and necessary shift in health promotion workforce development in Australia.

Evaluation of Trippple : a field analysis

This paper illustrates a field research performed with a team of experts involved in the evaluation of Trippple, a system aimed at supporting the different phases of a tourist trip, in order to provide feedback and insights, both on the functionalities already implemented (that at the time of evaluation were available only as early and very unstable prototypes), and on the functionalities still to be implemented. We show how the involvement of professionals helped to focus on challenging aspects, instead of less important, cosmetic, issues and resulted profitable in terms of early feedback, issues spotted, and improvements suggested

A baseline study of Australia's community recycling enterprises(CRE)

This report presents the findings on a baseline study of Australia's community recycling enterprises(CREs). The study sought to document the activities and impacts of these enterprises and to understand the conditions under which they succeed. The purposes of the research were to generate evidence that can contribute to the development of practice and policy support for CREs, and to provide information that is useful to community groups wishing to establish new CREs. The study included a review of the existing literature in relation to CREs, an online survey of Australian CREs, and in-depth case studies of three CREs from various regions within Australia

Making prevention matter : a survey of child sexual abuse prevention programs in Australia

Throughout Australia, there is considerable concern about the prevalence of child sexual abuse. Sexual abuse is experienced by over 3,500 Australian children each year, causing long-term psychological, health and social problems to children, their families and communities, and significant economic costs to society as a whole. In many countries, the provision of school-based programs has been a core strategy in efforts to prevent child sexual abuse. However, little is known about the range of programs in use in Australia, the numbers of children who have received programs, and the contents and methods used in program delivery. This presentation reports on a detailed National survey of child sexual abuse prevention programs currently used in Australian primary schools. An online survey was conducted over a six-month period from November 2011 to April 2012 yielding detailed data from 38 programs. The presentation will provide an overview of: the scope and reach of programs; program content, teaching strategies and resources; barriers and facilitators to program adoption by schools; and program evaluation strategies.

Managing health risks of extreme weather events : need for a systematic approach

The Climate Commission recently outlined the trend of major extreme weather events in different regions of Australia, including heatwaves, floods, droughts, bushfires, cyclones and storms. These events already impose an enormous health and financial burden onto society and are projected to occur more frequently and intensely. Unless we act now, further financial losses and increasing health burdens seem inevitable. We seek to highlight the major areas for interdisciplinary investigation, identify barriers and formulate response strategies.

Pressure state response-based method for evaluating social benefits from smart grid development

In this paper, the inherent mechanism of social benefits associated with smart grid development is examined based on the pressure state response (PSR) model from resource economics. The emerging types of technology brought up by smart grid development are regarded as pressures. The improvements of the performance and efficiency of power system operation, such as the enhanced capability of accommodating renewable energy generation, are regarded as states. The effects of smart grid development on society are regarded as responses. Then, a novel method for evaluating social benefits from smart grid development is presented. Finally, the social benefits from smart grid development in a province in northwest China are carried out by using the developed evaluation system, and reasonable evaluation results are attained.

Using the SWAP to connect water, human rights and mining

It is important that industries’ water interactions respect the human right to water. Historically, within the mining industry there has been a disconnect between the management of sites’ internal water interactions and the consequences of their external impacts, including human rights impacts. This poses a challenge for the mining industry as it attempts to put the Ruggie Guiding Principles for Business and Human Rights into practice, particularly as United Nations has recently recognised the human right to water. A technical framework such as the Minerals Council of Australia’s Water Accounting Framework (WAF) can help to bridge this disconnect and to integrate human rights considerations into business practice by connecting a site’s external and internal water interactions and by encouraging regular monitoring of performance. However, at present the connection is limited since the WAF lacks the capability to formalise a site’s social water context. This work presents the Social Water Assessment Protocol (SWAP), a scoping tool consisting of a set of questions organised into taxonomic themes that capture a site’s social water context and that can be combined with the WAF to better connect human rights with mine water interactions.

From policy borrowing to implementation : an illuminative evaluation of learning and teaching in higher education in Australia (2002 to 2008)

This study documents and theorises the consequences of the 2003 Australian Government Reform Package focussed on learning and teaching in Higher Education during the period 2002 to 2008. This is achieved through the perspective of program evaluation and the methodology of illuminative evaluation. The findings suggest that the three national initiatives of that time, Learning and Teaching Performance Fund (LTPF), Australian Learning and Teaching Council (ALTC), and Australian Universities Quality Agency (AUQA), were successful in repositioning learning and teaching as a core activity in universities. However, there were unintended consequences brought about by international policy borrowing, when the short-lived nature of LTPF suggests a legacy of quality compliance rather than one of quality enrichment.

Best practice in peer-led curriculum content : informing an interactive program to improve passenger safety among high school seniors

Despite ongoing improvements in behaviour change strategies, licensing models and road law enforcement measures young drivers remain significantly over-represented in fatal and non-fatal road related crashes. This paper focuses on the safety of those approaching driving age and identifies both high priority road safety messages and relevant peer-led strategies to guide the development school programs. It summarises the review in a program logic model built around the messages and identified curriculum elements, as they may be best operationalised within the licensing and school contexts in Victoria. This paper summarises a review of common deliberate risk-taking and non-deliberate unsafe driving behaviours among novice drivers, highlighting risks associated with speeding, driving while fatigued, driving while impaired and carrying passengers. Common beliefs of young people that predict risky driving were reviewed, particularly with consideration of those beliefs that can be operationalised in a behaviour change school program. Key components of adolescent risk behaviour change programs were also reviewed, which identified a number of strategies for incorporation in a school based behaviour change program, including: a well-structured theoretical design and delivery, thoughtfully considered peer-selected processes, adequate training and supervision of peer facilitators, a process for monitoring and sustainability, and interactive delivery and participant discussions. The research base is then summarised in a program logic model with further discussion about the quality of the current state of knowledge of evaluation of behaviour change programs and the need for considerable development in program evaluation.

Illuminative evaluation as a method applied to Australian Government policy borrowing and implementation in higher education

Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.Any government deciding to invoke widespread change in its higher education sector through implementation of new policies impacts on every institution and all staff and students, often in both the time taken up and the heightened emotions caused. The central phenomenon that this study addresses is the process and consequences of policy changes in higher education in Australia. The aim of this article is to record the research design through the perspective (evaluation research), theoretical framework (program evaluation) and methods (content analysis, descriptive statistical analysis and bibliometric analysis) applied to the investigation of the 2003 federal government higher education reform package. This approach allows both the intended and unintended consequences arising from the policy implementation of three national initiatives focused on learning and teaching in higher education in Australia to surface. As a result, this program evaluation, also known in some disciplines as policy implementation analysis, will demonstrate the applicability of illuminative evaluation as a methodology and reinforce how program evaluation will assist and advise future government reform and policy implementation, and will serve as a legacy for future evaluative research.

A call for better care: the impact of postnatal contact services on women’s parenting confidence and experiences of postpartum care in Queensland, Australia

Background Universal postnatal contact services are provided in several Australian states, but their impact on women’s postnatal care experience has not been evaluated. Furthermore, there is lack of evidence or consensus about the optimal type and amount of postpartum care after hospital discharge for maternal outcomes. This study aimed to assess the impact of providing Universal Postnatal Contact Service (UPNCS) funding to public birthing facilities in Queensland, Australia on women’s postnatal care experiences, and associations between amount and type (telephone or home visits) of contact on parenting confidence, and perceived sufficiency and quality of postnatal care. Methods Data collected via retrospective survey of postnatal women (N = 3,724) were used to compare women who birthed in UPNCS-funded and non-UPNCS-funded facilities on parenting confidence, sufficiency of postnatal care, and perceived quality of postnatal care. Associations between receiving telephone and home visits and the same outcomes, regardless of UPNCS funding, were also assessed. Results Women who birthed in an UPNCS-funded facility were more likely to receive postnatal contact, but UPNCS funding was not associated with parenting confidence, or perceived sufficiency or perceived quality of care. Telephone contact was not associated with parenting confidence but had a positive dose–response association with perceived sufficiency and quality. Home visits were negatively associated with parenting confidence when 3 or more were received, had a positive dose–response association with perceived sufficiency and were positively associated with perceived quality when at least 6 were received. Conclusions Funding for UPNCS is unlikely to improve population levels of maternal parenting confidence, perceived sufficiency or quality of postpartum care. Where only minimal contact can be provided, telephone may be more effective than home visits for improving women’s perceived sufficiency and quality of care. Additional service initiatives may be needed to improve women’s parenting confidence.