Funding grant proposals for scientific research : retrospective analysis of scores by members of grant review panel

Objectives: To quantify randomness and cost when choosing health and medical research projects for funding. Design: Analysis of retrospective data from grant review panels. Setting: The National Health & Medical Research Council of Australia. Participants/Data: All panel members’ scores for grant proposals submitted in 2009. Main outcome measure: The proportion of grant proposals that were always, sometimes and never funded after accounting for random variability arising from variation in panel members’ scores; the cost-effectiveness of different size assessment panels. Results: 59% of 620 funded grants were sometimes not funded when random variability was accounted for. Only 9% of grant proposals were always funded, 61% were never funded and 29% were sometimes funded. The extra cost per grant effectively funded from the most effective system was $18,541. Conclusions: Allocating funding for scientific research in health and medicine is costly and somewhat random. There are many useful research questions to be addressed that could improve current processes.

Peer review of teaching practice and resources : exploring new spaces to embrace cultural change

The implementation of systematic peer review as a professional development activity, and as a support for educational design activities is under-utilised in many Australian higher education institutions. This case study reports on the first stages of planning and implementation of an institution-wide project to enhance teaching and learning quality at a remote and regional university, where one of the major strategies for improvement is peer review. Through a systematic process of staff engagement in peer review, within and from outside the organisation, a substantial change in flexible learning is envisaged. A mix of new and different learning spaces are to be used in the project, including blended learning spaces for academic development. This paper describes the research framework that will guide the peer review process and examines the early findings of the design-based research. Leadership, awareness raising and development of a supportive community of inquiry are seen as key components for successful implementation of peer review. In addition, unique contextual elements add to the complexity of designing for transformative change within such a relatively new organization.

Building and sustaining a culture to support peer review of teaching

1. Background/context This presentation will report on emerging results from a two phase project funded by the Australian Learning and Teaching Council (ALTC). The project was designed in partnership with five universities and aimed to embed peer review within the local teaching and learning culture by using a distributive leadership framework. 2. The initiative/practice The presentation will highlight research outcomes that bring together both the fundamentals of peer review of teaching with the broader contextual elements of Integration, Leadership and Development. It will be demonstrated that peer review of teaching can be implemented and have advantages for academic staff, teaching evaluation and an organisation if attention is given to strategies that influence the contexts and cultures of teaching. Peer review as a strategy to develop excellence in teaching is considered from a holistic perspective that by necessity encompasses all elements of an educational environment. Results demonstrate achievements that can be obtained through working to foster conditions needed for sustainable leadership and change. The work has implications for policy, research, teaching development and student outcomes and has potential application world-wide. 3. Method(s) of evaluative data collection and analysis The 2 phase project collected focus group and questionnaire data to inform research results that were analysed using a thematic qualitative approach and statistical exploration. 4. Evidence of effectiveness The presentation will demonstrate the effectiveness of distributive leadership and strategic approaches to working for cultural change through the presentation of project findings.

Developing a culture to support peer review of teaching in higher education

This presentation addresses issues related to leadership, academic development and scholarship of teaching and learning, and highlights research funded by the Australian Office of Learning and Teaching (OLT) designed to embed and sustain peer review of teaching within the culture of 5 Australian universities: Queensland University of Technology, University of Technology, Sydney, University of Adelaide, Curtin University, and Charles Darwin University. Peer review of teaching in higher education will be emphasised as a professional process for providing feedback on teaching and learning practice, which if sustained, can become an effective ongoing strategy for academic development (Barnard et al, 2011; Bell, 2005; Bolt and Atkinson, 2010; McGill & Beaty 2001, 1992; Kemmis & McTaggart, 2000). The research affirms that using developmental peer review models (Barnard et al, 2011; D'Andrea, 2002; Hammersley-Fletcher & Orsmond, 2004) can bring about successful implementation, especially when implemented within a distributive leadership framework (Spillane & Healey, 2010). The project’s aims and objectives were to develop leadership capacity and integrate peer review as a cultural practice in higher education. The research design was a two stage inquiry process over 2 years. The project began in July 2011 and encompassed a development and pilot phase followed by a cascade phase with questionnaire and focus group evaluation processes to support ongoing improvement and measures of outcome. Leadership development activities included locally delivered workshops complemented by the identification and support of champions. To optimise long term sustainability, the project was implemented through existing learning and teaching structures and processes within the respective partner universities. Research outcomes highlight the fundamentals of peer review of teaching and the broader contextual elements of integration, leadership and development, expressed as a conceptual model for embedding peer review of teaching within higher education. The research opens a communicative space about introduction of peer review that goes further than simply espousing its worth and introduction. The conceptual model highlights the importance of development of distributive leadership capacity, integration of policies and processes, and understanding the values, beliefs, assumptions and behaviors embedded in an organizational culture. The presentation overviews empirical findings that demonstrate progress to advance peer review requires an ‘across-the-board’ commitment to embed change, and inherently demands a process that co-creates connection across colleagues, discipline groups, and the university sector. Progress toward peer review of teaching as a cultural phenomenon can be achieved and has advantages for academic staff, scholarship, teaching evaluation and an organisation, if attention is given to strategies that influence the contexts and cultures of teaching practice. Peer review as a strategy to develop excellence in teaching is considered from a holistic perspective that by necessity encompasses all elements of an educational environment and has a focus on scholarship of teaching. The work is ongoing and has implication for policy, research, teaching development and student outcomes, and has potential application world-wide.

LeaD-In : a cultural change model for peer review of teaching in higher education

Peer review of teaching is recognized increasingly as one strategy for academic development even though historically peer review of teaching is often unsupported by policy, action and culture in many Australian universities. Higher education leaders report that academics generally do not engage with peer review of teaching in a systematic or constructive manner, and this paper advances and analyses a conceptual model to highlight conditions and strategies necessary for the implementation of sustainable peer review in higher education institutions. The model highlights leadership, development and implementation, which are critical to the success and formation of a culture of peer review of teaching. The work arises from collaborative research funded by the Office for Learning and Teaching to foster and advance a culture of peer review of teaching across several universities in Australia.

Improving road safety through deterrence-based initiatives : a review of research

The efficacy of road safety countermeasures to deter motorists from engaging in illegal behaviours is extremely important when considering the personal and economic impact of road accidents on the community. Within many countries, deterrence theory has remained a cornerstone to criminology and criminal justice policy, particularly within the field of road safety, as policy makers and enforcement agencies attempt to increase perceptions regarding the certainty, severity and swiftness of sanctions for those who engage in illegal motoring behaviours. Using the Australian experience (particularly the tremendous amount of research into drink driving), the current paper reviews the principles underpinning deterrence theory, the utilisation of the approach within some contemporary road safety initiatives (e.g., Random Breath Testing) as well as highlights some methods to enhance a deterrent effect. The paper also provides direction for future deterrence-based research, in particular, considering the powerful impact of non-legal sanctions, punishment avoidance as well as creating culturally embedded behavioural change.

Developing a culture of peer review of teaching through a distributive leadership approach

Teaching is a core function of higher education and must be effective if it is to provide students with learning experiences that are stimulating, challenging and rewarding Obtaining feedback on teaching is indispensable to enhancing the quality of learning design, facilitating personal and/or professional development and maximising student learning outcomes. Peer review of teaching has the potential to improve the quality of teaching at tertiary level, by encouraging critical reflection on teaching, innovation in teaching practice and scholarship of teaching at all academic levels. However, embedding peer review within the culture of teaching and learning is a significant challenge that requires sustained commitment from senior leadership as well as those in leadership roles within local contexts.

Editors review of 'Research in early childhood education for sustainability : international perspectives and provocations'

Mounting concerns about climate change and unsustainable development, and their current and future impacts on all of us – but particularly on children - provided the impetus for this book. Then, as researchers in early childhood education (ECE) and/or education for sustainability (EfS), we used these concerns to shape and question our thinking. This first-ever research text in Early Childhood Education for Sustainability (ECEfS) was advanced when the chapter authors, almost all of whom participated in one or both Transnational Dialogues in Research in Early Childhood Education for Sustainability (Stavanger, Norway, 2010, and Brisbane, Australia, 2011) met for the first time - a critical mass of researchers from vastly different parts of the globe - Norway, Sweden, Australia and New Zealand at the inaugural meeting, with participants from Korea, Japan and Singapore attending the second. We came together to debate, discuss and share ideas about research and theory in the emerging field of ECEfS. An agreed-upon outcome of the Dialogues was this text.

Using simplified peer review processes to fund research: A prospective study

Objective: To prospectively test two simplified peer review processes, estimate the agreement between the simplified and official processes, and compare the costs of peer review. Design, participants and setting: A prospective parallel study of Project Grant proposals submitted in 2013 to the National Health and Medical Research Council (NHMRC) of Australia. The official funding outcomes were compared with two simplified processes using proposals in Public Health and Basic Science. The two simplified processes were: panels of 7 reviewers who met face-to-face and reviewed only the nine-page research proposal and track record (simplified panel); and 2 reviewers who independently reviewed only the nine-page research proposal (journal panel). The official process used panels of 12 reviewers who met face-to-face and reviewed longer proposals of around 100 pages. We compared the funding outcomes of 72 proposals that were peer reviewed by the simplified and official processes. Main outcome measures: Agreement in funding outcomes; costs of peer review based on reviewers’ time and travel costs. Results: The agreement between the simplified and official panels (72%, 95% CI 61% to 82%), and the journal and official panels (74%, 62% to 83%), was just below the acceptable threshold of 75%. Using the simplified processes would save $A2.1–$A4.9 million per year in peer review costs. Conclusions: Using shorter applications and simpler peer review processes gave reasonable agreement with the more complex official process. Simplified processes save time and money that could be reallocated to actual research. Funding agencies should consider streamlining their application processes.

Radiation therapist peer review: Raising the bar on quality and safety in radiation oncology

Purpose An emerging developmental tool to help radiation therapists achieve better outcomes is 'peer review'. This review of the current literature summarises the challenges and benefits of peer review in both individual and departmental practice. Discussion There is compelling evidence supporting peer review implementation at both individual and department level in many professions. Implementing peer review requires that radiation therapists and other radiation oncology professionals embrace a culture that supports safety. Peer review can identify trends and barriers associated with quality radiotherapy and share best practice or recommend changes accordingly. Support for peer review must come from pre-registration educational systems as well as clinical managers. Continuing professional development in the workplace is nurtured by peer review of radiotherapy practice and an aptitude for this should be viewed as important to the profession as technical and clinical skills. Conclusion It is clear that peer review has the potential to facilitate reflective practice, improve staff motivation and help foster a culture of quality and safety in radiation oncology. To drive the issues of quality and safety a step further radiation therapists need to accept the challenge of adopting peer review methods in day-to-day practice.

Streamlined research funding using short proposals and accelerated peer review : an observational study

Background Despite the widely recognised importance of sustainable health care systems, health services research remains generally underfunded in Australia. The Australian Centre for Health Services Innovation (AusHSI) is funding health services research in the state of Queensland. AusHSI has developed a streamlined protocol for applying and awarding funding using a short proposal and accelerated peer review. Method An observational study of proposals for four health services research funding rounds from May 2012 to November 2013. A short proposal of less than 1,200 words was submitted using a secure web-based portal. The primary outcome measures are: time spent preparing proposals; a simplified scoring of grant proposals (reject, revise or accept for interview) by a scientific review committee; and progressing from submission to funding outcomes within eight weeks. Proposals outside of health services research were deemed ineligible. Results There were 228 eligible proposals across 4 funding rounds: from 29% to 79% were shortlisted and 9% to 32% were accepted for interview. Success rates increased from 6% (in 2012) to 16% (in 2013) of eligible proposals. Applicants were notified of the outcomes within two weeks from the interview; which was a maximum of eight weeks after the submission deadline. Applicants spent 7 days on average preparing their proposal. Applicants with a ranking of reject or revise received written feedback and suggested improvements for their proposals, and resubmissions composed one third of the 2013 rounds. Conclusions The AusHSI funding scheme is a streamlined application process that has simplified the process of allocating health services research funding for both applicants and peer reviewers. The AusHSI process has minimised the time from submission to notification of funding outcomes.

On the time spent preparing grant proposals : an observational study of Australian researchers

Objective: To estimate the time spent by the researchers for preparing grant proposals, and to examine whether spending more time increase the chances of success. Design: Observational study. Setting: The National Health and Medical Research Council (NHMRC) of Australia. Participants: Researchers who submitted one or more NHMRC Project Grant proposals in March 2012. Main outcome measures: Total researcher time spent preparing proposals; funding success as predicted by the time spent. Results: The NHMRC received 3727 proposals of which 3570 were reviewed and 731 (21%) were funded. Among our 285 participants who submitted 632 proposals, 21% were successful. Preparing a new proposal took an average of 38 working days of researcher time and a resubmitted proposal took 28 working days, an overall average of 34 days per proposal. An estimated 550 working years of researchers' time (95% CI 513 to 589) was spent preparing the 3727 proposals, which translates into annual salary costs of AU$66 million. More time spent preparing a proposal did not increase the chances of success for the lead researcher (prevalence ratio (PR) of success for 10 day increase=0.91, 95% credible interval 0.78 to 1.04) or other researchers (PR=0.89, 95% CI 0.67 to 1.17). Conclusions: Considerable time is spent preparing NHMRC Project Grant proposals. As success rates are historically 20–25%, much of this time has no immediate benefit to either the researcher or society, and there are large opportunity costs in lost research output. The application process could be shortened so that only information relevant for peer review, not administration, is collected. This would have little impact on the quality of peer review and the time saved could be reinvested into research.

The utility and challenges of using ICD codes in child maltreatment research : a review of existing literature

Objective: The objectives of this article are to explore the extent to which the International Statistical Classification of Diseases and Related Health Problems (ICD) has been used in child abuse research, to describe how the ICD system has been applied and to assess factors affecting the reliability of ICD coded data in child abuse research.----- Methods: PubMed, CINAHL, PsychInfo and Google Scholar were searched for peer reviewed articles written since 1989 that used ICD as the classification system to identify cases and research child abuse using health databases. Snowballing strategies were also employed by searching the bibliographies of retrieved references to identify relevant associated articles. The papers identified through the search were independently screened by two authors for inclusion, resulting in 47 studies selected for the review. Due to heterogeneity of studies metaanalysis was not performed.----- Results: This paper highlights both utility and limitations of ICD coded data. ICD codes have been widely used to conduct research into child maltreatment in health data systems. The codes appear to be used primarily to determine child maltreatment patterns within identified diagnoses or to identify child maltreatment cases for research.----- Conclusions: A significant impediment to the use of ICD codes in child maltreatment research is the under-ascertainment of child maltreatment by using coded data alone. This is most clearly identified and, to some degree, quantified, in research where data linkage is used. Practice Implications: The importance of improved child maltreatment identification will assist in identifying risk factors and creating programs that can prevent and treat child maltreatment and assist in meeting reporting obligations under the CRC.

Thinking Collaboratively about the Peer-Review Process for Journal- Article Publication

In this conversation, Kevin K. Kumashiro shares his reflections on challenges to publishing anti-oppressive research in educational journals. He then invites eight current and former editors of leading educational research journals--William F. Pinar, Elizabeth Graue, Carl A. Grant, Maenette K. P. Benham, Ronald H. Heck, James Joseph Scheurich, Allan Luke, and Carmen Luke--to critique and expand on his analysis. Kumashiro begins the conversation by describing his own experiences submitting manuscripts to educational research journals and receiving comments by anonymous reviewers and journal editors. He suggests three ways to rethink the collaborative potential of the peer-review process: as constructive, as multilensed, and as situated. The eight current and former editors of leading educational research journals then critique and expand Kumashiro's analysis. Kumashiro concludes the conversation with additional reflections on barriers and contradictions involved in advancing anti-oppressive educational research in educational journals. (Contains 3 notes.)

A literature review on research methodologies of gross pollutant traps

This paper presents a comprehensive review of scientific and grey literature on gross pollutant traps (GPTs). GPTs are designed with internal screens to capture gross pollutants—organic matter and anthropogenic litter. Their application involves professional societies, research organisations, local city councils, government agencies and the stormwater industry—often in partnership. In view of this, the 113 references include unpublished manuscripts from these bodies along with scientific peer-reviewed conference papers and journal articles. The literature reviewed was organised into a matrix of six main devices and nine research areas (testing methodologies) which include: design appraisal study, field monitoring/testing, experimental flow fields, gross pollutant capture/retention characteristics, residence time calculations, hydraulic head loss, screen blockages, flow visualisations and computational fluid dynamics (CFD). When the fifty-four item matrix was analysed, twenty-eight research gaps were found in the tabulated literature. It was also found that the number of research gaps increased if only the scientific literature was considered. It is hoped, that in addition to informing the research community at QUT, this literature review will also be of use to other researchers in this field.