Preparing for Research Assessment: Co-evolution and Gamesmanship

In this paper we adopt a complex systems perspective to examine the perturbations caused by the introduction of the Research Quality Framework (RQF) at a research-intensive Australian university. This case is instructive as it 1) presents a Federal policy initiative that attempted to fundamentally alter the recognition and reward mechanism within a regulated funding environment, 2) analyses the strategies of an institution and its research groups as they sought to not only comply with the implementation of the RQF but to maximise their outcome,and 3) it reveals the ways that some actors used this perturbation to advance their own interests. In short, this case represents an instrumental study into the dynamics of how information systems, organisations, and individuals co-evolve in practice as they seek to navigate a complex problem scenario.

Rank and file : assessing research quality in Australia

In this paper, the author describes recent developments in the assessment of research activity and publication in Australia. Of particular interest to readers will be the move to rank academic journals. Educational Philosophy and Theory (EPAT) received the highest possible ranking, however, the process is far from complete. Some implications for the field, for this journal and particularly, for the educational foundations are discussed.

Assessment of applicability and transferability of evidence-based antenatal interventions to the Australian indigenous setting

There is a need for public health interventions to be based on the best available evidence. Unfortunately, well-conducted studies from settings similar to that in which an intervention is to be implemented are often not available. Therefore, health practitioners are forced to make judgements about proven effective interventions in one setting and their suitability to make a difference in their own setting. The framework of Wang et al. has been proposed to help with this process. This paper provides a case study on the application of the framework to a decision-making process regarding antenatal care in Aboriginal and Torres Strait Islander communities in Queensland. This method involved undertaking a systematic search of the current available evidence, then conducting a second literature search to determine factors that may affect the applicability and transferability of these interventions into these communities. Finally, in consideration of these factors, clinical judgement decisions on the applicability and transferability of these interventions were made. This method identified several interventions or strategies for which there was evidence of improving antenatal care or outcomes. By using the framework, we concluded that several of these effective interventions would be feasible in Aboriginal and Torres Strait Islander communities within Queensland.

An investigation of the impacts of Excellence in Research for Australia : a case study on accounting for research

This is the first study to explore the way Excellence in Research for Australia (ERA), a research assessment exercise introduced in the Australian higher education sector in 2010, fostered the development of strategically oriented Management Accounting technologies in the form of Performance Management Systems (PMS) to achieve research excellence within an Australian university. It identifies ERA's intended and unintended consequences. While ERA enabled the creation of tighter controls in the PMS of faculties, departments and individual academics within the university, enhancing its reported research performance, the impact on academics was low job satisfaction, increased workload and a higher focus on research than teaching.

Implementing an enhanced repository statistics system for QUT ePrints

Queensland University of Technology (QUT) was one of the first universities in Australia to establish an institutional repository. Launched in November 2003, the repository (QUT ePrints) uses the EPrints open source repository software (from Southampton) and has enjoyed the benefit of an institutional deposit mandate since January 2004. Currently (April 2012), the repository holds over 36,000 records, including 17,909 open access publications with another 2,434 publications embargoed but with mediated access enabled via the ‘Request a copy’ button which is a feature of the EPrints software. At QUT, the repository is managed by the library.QUT ePrints (http://eprints.qut.edu.au) The repository is embedded into a number of other systems at QUT including the staff profile system and the University’s research information system. It has also been integrated into a number of critical processes related to Government reporting and research assessment. Internally, senior research administrators often look to the repository for information to assist with decision-making and planning. While some statistics could be drawn from the advanced search feature and the existing download statistics feature, they were rarely at the level of granularity or aggregation required. Getting the information from the ‘back end’ of the repository was very time-consuming for the Library staff. In 2011, the Library funded a project to enhance the range of statistics which would be available from the public interface of QUT ePrints. The repository team conducted a series of focus groups and individual interviews to identify and prioritise functionality requirements for a new statistics ‘dashboard’. The participants included a mix research administrators, early career researchers and senior researchers. The repository team identified a number of business criteria (eg extensible, support available, skills required etc) and then gave each a weighting. After considering all the known options available, five software packages (IRStats, ePrintsStats, AWStats, BIRT and Google Urchin/Analytics) were thoroughly evaluated against a list of 69 criteria to determine which would be most suitable. The evaluation revealed that IRStats was the best fit for our requirements. It was deemed capable of meeting 21 out of the 31 high priority criteria. Consequently, IRStats was implemented as the basis for QUT ePrints’ new statistics dashboards which were launched in Open Access Week, October 2011. Statistics dashboards are now available at four levels; whole-of-repository level, organisational unit level, individual author level and individual item level. The data available includes, cumulative total deposits, time series deposits, deposits by item type, % fulltexts, % open access, cumulative downloads, time series downloads, downloads by item type, author ranking, paper ranking (by downloads), downloader geographic location, domains, internal v external downloads, citation data (from Scopus and Web of Science), most popular search terms, non-search referring websites. The data is displayed in charts, maps and table format. The new statistics dashboards are a great success. Feedback received from staff and students has been very positive. Individual researchers have said that they have found the information to be very useful when compiling a track record. It is now very easy for senior administrators (including the Deputy Vice Chancellor-Research) to compare the full-text deposit rates (i.e. mandate compliance rates) across organisational units. This has led to increased ‘encouragement’ from Heads of School and Deans in relation to the provision of full-text versions.

Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal

Developing an effective impact evaluation framework, managing and conducting rigorous impact evaluations, and developing a strong research and evaluation culture within development communication organisations presents many challenges. This is especially so when both the community and organisational context is continually changing and the outcomes of programs are complex and difficult to clearly identify.----- This paper presents a case study from a research project being conducted from 2007-2010 that aims to address these challenges and issues, entitled Assessing Communication for Social Change: A New Agenda in Impact Assessment. Building on previous development communication projects which used ethnographic action research, this project is developing, trailing and rigorously evaluating a participatory impact assessment methodology for assessing the social change impacts of community radio programs in Nepal. This project is a collaboration between Equal Access – Nepal (EAN), Equal Access – International, local stakeholders and listeners, a network of trained community researchers, and a research team from two Australian universities. A key element of the project is the establishment of an organisational culture within EAN that values and supports the impact assessment process being developed, which is based on continuous action learning and improvement. The paper describes the situation related to monitoring and evaluation (M&E) and impact assessment before the project began, in which EAN was often reliant on time-bound studies and ‘success stories’ derived from listener letters and feedback. We then outline the various strategies used in an effort to develop stronger and more effective impact assessment and M&E systems, and the gradual changes that have occurred to date. These changes include a greater understanding of the value of adopting a participatory, holistic, evidence-based approach to impact assessment. We also critically review the many challenges experienced in this process, including:----- • Tension between the pressure from donors to ‘prove’ impacts and the adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations.----- • Resistance from the content teams to changing their existing M&E practices and to the perceived complexity of the approach.----- • Lack of meaningful connection between the M&E and content teams.----- • Human resource problems and lack of capacity in analysing qualitative data and reporting results.----- • The contextual challenges, including extreme poverty, wide cultural and linguistic diversity, poor transport and communications infrastructure, and political instability.----- • A general lack of acceptance of the importance of evaluation within Nepal due to accepting everything as fate or ‘natural’ rather than requiring investigation into a problem.

Deliberative assessment of surrogate consent in dementia research

Background Research involving incapacitated persons with dementia entails complex scientific, legal, and ethical issues, making traditional surveys of layperson views on the ethics of such research challenging. We therefore assessed the impact of democratic deliberation (DD), involving balanced, detailed education and peer deliberation, on the views of those responsible for persons with dementia. Methods One hundred and seventy-eight community-recruited caregivers or primary decision-makers for persons with dementia were randomly assigned to either an all-day DD session group or a control group. Educational materials used for the DD session were vetted for balance and accuracy by an interdisciplinary advisory panel. We assessed the acceptability of family-surrogate consent for dementia research (“surrogate-based research”) from a societal policy perspective as well as from the more personal perspectives of deciding for a loved one or for oneself (surrogate and self-perspectives), assessed at baseline, immediately post-DD session, and 1 month after DD date, for four research scenarios of varying risk-benefit profiles. Results At baseline, a majority in both the DD and control groups supported a policy of family consent for dementia research in all research scenarios. The support for a policy of family consent for surrogate-based research increased in the DD group, but not in the control group. The change in the DD group was maintained 1 month later. In the DD group, there were transient changes in attitudes from surrogate or self-perspectives. In the control group, there were no changes from baseline in attitude toward surrogate consent from any perspective. Conclusions Intensive, balanced, and accurate education, along with peer deliberation provided by democratic deliberation, led to a sustained increase in support for a societal policy of family consent in dementia research among those responsible for dementia patients.

The Australian Grey Nomads : are they who we think they are? Enhancing formative research through the quantitative assessment of psychological constructs

Issue addressed: Measures of 'social identity' and 'psychological sense of community' were included within a broader formative research inquiry to gain insight into the identity characteristics and level of connectedness among older recreational road travellers (commonly known as Grey Nomads). The research sought to gain insights on how best to reach or speak to this growing driver cohort. ----- ----- Method: Participants included 631 older recreational road travellers ranging in age from 50 years to over 80 years. Data were obtained through three scales which were incorporated into a larger formative research survey; an identity hierarchy, the Three Factor Model of Social Identity and the Sense of Community Index. ----- ----- Results: Older recreational road travellers see themselves principally as couples, with social group identity being secondary. Although many identified to some degree with the Grey Nomad identity, when asked to self categorise as either members of the Broad Network of Recreational Vehicle Travellers or as Grey Nomads, the majority categorised themselves as the former. Those identifying as Grey Nomads, however, reported significantly higher levels of 'social identification' and 'sense of community'. ----- ----- Conclusion: The Grey Nomad identity may not be the best identity at which to target road safety messages for this cohort. Targeting travelling 'couples' may be more efficacious. Using the 'Grey Nomad' identity is likely to reap at least some success, however, given that many identified to some degree with this group identity. Those identifying as Grey Nomads may be more open to community participation or behaviour change given their significantly higher levels of 'social identity' and 'sense of community'.

Motorcycle Safety Research Project Interim Report 2 : Review of the Consistent Assessment Process (CAP)

Motorcycle trauma is a serious road safety issue in Queensland and throughout Australia. In 2009, Queensland Transport (later Transport and Main Roads or TMR) appointed CARRS-Q to provide a three-year program of Road Safety Research Services for Motorcycle Rider Safety. Funding for this research originated from the Motor Accident Insurance Commission. This program of research was undertaken to produce knowledge to assist TMR to improve motorcycle safety by further strengthening the licensing and training system to make learner riders safer by developing a pre-learner package (Deliverable 1), and by evaluating the Q-Ride CAP program to ensure that it is maximally effective and contributes to the best possible training for new riders (Deliverable 2), which is the focus of this report. Deliverable 3 of the program identified potential new licensing components that will reduce the incidence of risky riding and improve higher-order cognitive skills in new riders. This report provides a summary of Deliverables 2.1 through to 2.4.

Current research: Neurocognitive risk assessment

The current research began from the starting point that what we are grappling with when we are dealing with violent extremists by and large is essentially ‘normal people’. What follows in this third major section of this research paper is the theoretical and conceptual search for making researchable the following question: ‘How do you assess someone who is normal?’

Research update: LANDMark study: Longitudinal assessment of neuropathy in diabetes using novel ophthalmic MARKers

The eye is a simple, non-invasive location for screening, diagnosing and follow up of diabetic peripheral neuropathy.

The Scored Patient-Generated Subjective Global Assessment (PG-SGA) and its Association with Quality of Life in Ambulatory Patients Receiving Radiotherapy

OBJECTIVE: To evaluate the scored Patient-generated Subjective Global Assessment (PG-SGA) tool as an outcome measure in clinical nutrition practice and determine its association with quality of life (QoL). DESIGN: A prospective 4 week study assessing the nutritional status and QoL of ambulatory patients receiving radiation therapy to the head, neck, rectal or abdominal area. SETTING: Australian radiation oncology facilities. SUBJECTS: Sixty cancer patients aged 24-85 y. INTERVENTION: Scored PG-SGA questionnaire, subjective global assessment (SGA), QoL (EORTC QLQ-C30 version 3). RESULTS: According to SGA, 65.0% (39) of subjects were well-nourished, 28.3% (17) moderately or suspected of being malnourished and 6.7% (4) severely malnourished. PG-SGA score and global QoL were correlated (r=-0.66, P<0.001) at baseline. There was a decrease in nutritional status according to PG-SGA score (P<0.001) and SGA (P<0.001); and a decrease in global QoL (P<0.001) after 4 weeks of radiotherapy. There was a linear trend for change in PG-SGA score (P<0.001) and change in global QoL (P=0.003) between those patients who improved (5%) maintained (56.7%) or deteriorated (33.3%) in nutritional status according to SGA. There was a correlation between change in PG-SGA score and change in QoL after 4 weeks of radiotherapy (r=-0.55, P<0.001). Regression analysis determined that 26% of the variation of change in QoL was explained by change in PG-SGA (P=0.001). CONCLUSION: The scored PG-SGA is a nutrition assessment tool that identifies malnutrition in ambulatory oncology patients receiving radiotherapy and can be used to predict the magnitude of change in QoL.