Increasing the rigour and trustworthiness of participatory evaluations : learnings from the field

Participatory evaluation and participatory action research (PAR) are increasingly used in community-based programs and initiatives and there is a growing acknowledgement of their value. These methodologies focus more on knowledge generated and constructed through lived experience than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that acknowledges and takes into account the diverse perspectives, values and interpretations of participants and evaluation professionals. However, evaluation rigour need not be lost in this approach. Increasing the rigour and trustworthiness of participatory evaluations and PAR increases the likelihood that results are seen as credible and are used to continually improve programs and policies.----- Drawing on learnings and critical reflections about the use of feminist and participatory forms of evaluation and PAR over a 10-year period, significant sources of rigour identified include:----- • participation and communication methods that develop relations of mutual trust and open communication----- • using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection----- • ongoing meta-evaluation and critical reflection----- • critically assessing the intended and unintended impacts of evaluations, using relevant theoretical models----- • using rigorous data analysis and reporting processes----- • participant reviews of evaluation case studies, impact assessments and reports.

Rigour versus Relevance Revisited : Evidence from IS Conference Reviewing Practice

A commonly held belief in the IS discipline is that rigour and relevance are contrary to each other and that addressing both is virtually impossible. It is also believed widely that the editorial practices of our premier conferences and journals over-emphasise rigour on the cost of relevance. However, while these two topics have been filled with numerous subjective discussions, more solid evidence into the true relationship between rigour and relevance and the impact of conference editors on this relationship is still outstanding. This paper contributes to this debate by deriving empirical evidence from a comprehensive and detailed analysis of the characteristics of the submissions and the reviewing practices of three recent IS conferences. It provides first insights into the actual relationship between rigour and relevance and into the role conference chairs play in balancing rigour and relevance. Besides the outcomes that the current set of evaluation criteria does not provide a straight forward proxy for relevance to practitioners, the paper offers two main contributions. First, empirical insights are provided that rigour and relevance do in fact not have to be mutually exclusive. Second, the editorial practices at conferences are skewed towards rigorous papers rather than relevant papers.

Academic rigour in science assessment tasks

The critical problem of student disengagement and underachievement in the middle years of schooling (Years 4 . 9) has focussed attention on the quality of educational programs in schools, in Australia and elsewhere. The loss of enthusiasm for science in the middle years is particularly problematic given the growing demand for science professionals. Reshaping middle years programs has included an emphasis on integrating Information and Communication Technologies (ICTs) and improving assessment practices to engage students in higher cognitive processes and enhance academic rigour. Understanding the nature of academic rigour and how to embed it in students. science assessment tasks that incorporate the use of ICTs could enable teachers to optimise the quality of the learning environment. However, academic rigour is not clearly described or defined in the literature and there is little empirical evidence upon which researchers and teachers could draw to enhance understandings. This study used a collective case study design to explore teachers' understandings of academic rigour within science assessment tasks. The research design is based on a conceptual framework that is underpinned by socio-cultural theory. Three methods were used to collect data from six middle years teachers and their students. These methods were a survey, focus group discussion with teachers and a group of students and individual semi-structured interviews with teachers. Findings of the case study revealed six criteria of academic rigour, namely, higher order thinking, alignment, building on prior knowledge, scaffolding, knowledge construction and creativity. Results showed that the middle years teachers held rich understandings of academic rigour that led to effective utilisation of ICTs in science assessment tasks. Findings also indicated that teachers could further enhance their understandings of academic rigour in some aspects of each of the criteria. In particular, this study found that academic rigour could have been further optimised by: promoting more thoughtful discourse and interaction to foster higher order thinking; increasing alignment between curriculum, pedagogy, and assessment, and students. prior knowledge; placing greater emphasis on identifying, activating and building on prior knowledge; better differentiating the level of scaffolding provided and applying it more judiciously; fostering creativity throughout tasks; enhancing teachers‟ content knowledge and pedagogical content knowledge, and providing more in-depth coverage of fewer topics to support knowledge construction. Key contributions of this study are a definition and a model which clarify the nature of academic rigour.

But what makes it doctoral? Taking on the traditionalists : interdisciplinary, practice-led doctoral research in the creative industries - a case study in academic politics, research, rigour and relevance

This paper examines a Doctoral journey of interdisciplinary exploration, explication, examination...and exasperation. In choosing to pursue a practice-led doctorate I had determined from the outset that ‘writing 100,000 words that only two people ever read’, was not something which interested me. Hence, the oft-asked question of ‘what kind of doctorate’ I was engaged in, consistently elicited the response, “a useful one”. In order to satisfy my own imperatives of authenticity and usefulness, my doctoral research had to clearly demonstrate relevance to; productively inform; engage with; and add value to: wider professional field(s) of practice; students in the university courses I teach; and the broader community - not just the academic community. Consequently, over the course of my research, the question, ‘But what makes it Doctoral?’ consistently resounded and resonated. Answering that question, to satisfy not only the traditionalists asking it but, perhaps surprisingly, some academic innovators - and more particularly, myself as researcher - revealed academic/political inconsistencies and issues which challenged both the fundamental assumptions and actuality of practice-led research. This paper examines some of those inconsistencies, issues and challenges and provides at least one possible answer to the question: ‘But what makes it Doctoral?’

Intake rigour : ensuring only “reportable deaths” become coroners’ cases

The failure of medical practitioners to consistently discharge their obligation to report sudden or unnatural deaths to coroners has rightly prompted concern. Following recent public scandals, coroners and health authorities have increasingly developed procedures to ensure that concerning deaths are reported to coroners. However, the negative consequences of deaths being unnecessarily reported have received less attention: unnecessary intrusion into bereavement; a waste of public resources; and added delay and hindrance to the investigation of matters needing a coroner’s attention. Traditionally, coroners have largely, unquestioningly assumed jurisdiction over any deaths for which a medical practitioner has not issued a cause of death certificate. The Office of the State Coroner in Queensland has recently trialled a system to more rigorously assess whether deaths apparently resulting from natural causes, which have been reported to a coroner, should be investigated by the coroner, rather than being finalised by a doctor issuing a cause of death certificate. This article describes that trial and its results.

Validity, trustworthiness and rigour:reasserting realism in qualitative research

Aim The aim of this paper is to challenge recent sceptical approaches to the possibility of validating qualitative research and to underline the benefits of adopting a realist approach to validity. Background In recent discussion about the methodological bases for qualitative research it has been argued that, because different methodologies take different approaches to validity, attempts to develop a common set of validation criteria are futile. On the basis of this sceptical view, a number of strategies for judging qualitative research have been proposed. These include suggestions that: it should be judged according to aesthetic or rhetorical criteria, rather than epistemological validity; responsibility for appraisal should move from researchers to readers; each methodology should be assessed individually according to its own merits. Discussion None of these suggestions provide a viable alternative to validity, defined as the extent to which research reflects accurately that to which it refers. Because the form of research does not determine its content, replacement of epistemology by aesthetics is unsustainable. Because research reports mediate between writer and reader, a one-sided approach to this relationship constitutes a false dichotomy. If we accept the criterion of practitioner confidence as a means of judging methodological approaches, this involves rejection of judgement according to a methodology�s own merits. Conclusion If qualitative research is actually about something, and if it is required to provide beneficial information, then a realist approach to validity holds out greatest promise.

Methodological rigour in assessing participatory development

There have been few rigorous assessments of the effectiveness of participatory processes for natural resource management. In Bangladesh an approach known as Participatory Action Plan Development (PAPD) has been developed and applied. By combining problem identification and solution analysis by separate stakeholder groups with plenary sessions it is claimed to result in consensus and more effective community based management. Methodological issues in assessing the effectiveness of such development are discussed and good practice illustrated. Under the same project there were sites where PAPD had been used and others without its use so a comparative assessment could be made. However, for an appropriate assessment it is important to identify clear testable hypotheses regarding the expected benefits, appropriate measures, and other factors which may affect or confound the outcome. The paper illustrates how participatory assessment involving both individual opinions and focus groups can be systematically recorded, quantified and used with other data in statistical analysis. By using statistical modelling methods at an appropriate level of aggregation and controlling for other factors, benefits from PAPD were found to be significant. The systematic approaches and practices recommended from this example can be applied in similar situations to test the effectiveness of participatory processes using participatory assessments.

Why Workplace Coaching Research Needs Greater Scientific Rigour: a Systematic Review and Recommendations for Future Research

To address a significant gap in the workplace coaching literature, we provide an up-to-date, comprehensive review of the literature in order to inform researchers, practitioners and organizations of the current state of play in workplace coaching research. In our review, we apply a systematic assessment of methodological rigour of the extant workplace coaching literature in order to gain insights into the link between rigour and research outcomes. Our review is fully inclusive and therefore includes both quantitative and qualitative studies of workplace coaching including coaching provided by supervisors. We explore the potential antecedents, moderators and mediators impacting on coaching outcomes, such as the coachee and coach profile and coaching intervention variables. Informed by our systematic review and methodological assessment, specific recommendations will be made to guide future research in the field of workplace coaching effectiveness and theoretical development.

Linking Theory and Practice: Performing a Reality Check on a Model of IS Success

The issue of ‘rigour vs. relevance’ in IS research has generated an intense, heated debate for over a decade. It is possible to identify, however, only a limited number of contributions on how to increase the relevance of IS research without compromising its rigour. Based on a lifecycle view of IS research, we propose the notion of ‘reality checks’ in order to review IS research outcomes in the light of actual industry demands. We assume that five barriers impact the efficient transfer of IS research outcomes; they are lack of awareness, lack of understandability, lack of relevance, lack of timeliness, and lack of applicability. In seeking to understand the effect of these barriers on the transfer of mature IS research into practice, we used focus groups. We chose DeLone and McLean’s IS success model as our stimulus because it is one of the more widely researched areas of IS.

Higher Order Thinking through ICT: What do middle years teachers think really matters?

The middle years of schooling has emerged as an important focus in Australian education. Student disengagement and alienation, the negative effects of non-completion of the senior years of schooling and underachievement have raised concerns about the quality of education during the middle years. For many schools, reshaping the middle years has involved incorporating Information and Communication Technologies (ICT) to motivate students. However, simultaneously there is a need to ensure that programs are academically rigorous. There is little doubt that there are potential benefits to integrating ICT into programs for middle years’ students. However, little is known about how middle years’ teachers perceive higher order thinking, which is a component of academic rigour. This paper investigates the question of What are teachers’ perceptions of higher order thinking in an ICT environment? The study is underpinned by socio-cultural theory which is based on the belief that learning occurs through social interaction and that individuals are shaped by the social and cultural tools and instruments they engage with. This investigation used a collective case study design. Two methods were used for data collection. These methods are semi-structured interviews with individual teachers and a class and a focus group discussion with teachers. Findings indicate that teachers hold various perceptions of higher order thinking that lead to productive approaches to integrating ICT in middle years’ classrooms. The paper highlights that there may be a continuum of perceptions of higher order thinking with ICT. This continuum may inform professional developers who are guiding and supporting teachers to integrate ICT into middle years’ classrooms.