Preparing for Research Assessment: Co-evolution and Gamesmanship

In this paper we adopt a complex systems perspective to examine the perturbations caused by the introduction of the Research Quality Framework (RQF) at a research-intensive Australian university. This case is instructive as it 1) presents a Federal policy initiative that attempted to fundamentally alter the recognition and reward mechanism within a regulated funding environment, 2) analyses the strategies of an institution and its research groups as they sought to not only comply with the implementation of the RQF but to maximise their outcome,and 3) it reveals the ways that some actors used this perturbation to advance their own interests. In short, this case represents an instrumental study into the dynamics of how information systems, organisations, and individuals co-evolve in practice as they seek to navigate a complex problem scenario.

KE Workshop report: Report Knowledge Exchange workshop Making data count - Research Data and Research Assessment

The possibilities of digital research have altered the production, publication and use of research results. Academic research practice and culture are changing or have already been transformed, but to a large degree the system of academic recognition has not yet adapted to the practices and possibilities of digital research. This applies especially to research data, which are increasingly produced, managed, published and archived, but play hardly a role yet in practices of research assessment. The aim of the workshop was to bring experts and stakeholders from research institutions, universities, scholarly societies and funding agencies together in order to review, discuss and build on possibilities to implement the culture of sharing and to integrate publication of data into research assessment procedures. The report 'The Value of Research Data - Metrics for datasets from a cultural and technical point of view' was presented and discussed. Some of the key finding were that data sharing should be considered normal research practice, in fact not sharing should be considered malpractice. Research funders and universities should support and encourage data sharing. There are a number of important aspects to consider when making data count in research and evaluation procedures. Metrics are a necessary tool in monitoring the sharing of data sets. However, data metrics are at present not very well developed and there is not yet enough experience in what these metrics actually mean. It is important to implement the culture of sharing through codes of conducts in the scientific communities. For further key findings please read the report.

A work-based research assessment of the impact of 'lean manufacturing' on health and safety education within an SME

Globalisation has had a major impact on the engineering industry as pacific Rim countries undercut manufacturing costs and provide a more cost-effective location for many businesses. Engineering in Nortehrn Ireland has mostly declined owing to increased competition from these countries. Engineering companies are now forced to streamline their production processes and employ cost-reducing practices in order to meet customer demands at reduced prices. This article aims to analyse the effects of one such streamlining endeavour which was first introduced after World War II in Japan- 'lean manufacturing' . 'Lean manufacturing' aims to reduce all wasteful activities within the production process in order to improve productivity, while reducing manufacturing costs. The work-based project under consideration was concerned with the impact 'lean manufacturing' may have on health and safety performance and education within an engineering company. The focus of the project was to determine through work-based research, and quantitative analysis, the employee perception on health and safety: has it changed (either positively or negatively), as a consequence of implementing 'lean manufacturing'.

Health, education and community: Assets and needs of a small urban community in north east Tamaulipas, Mexico: A community based participatory research assessment

This study explored the health, education, social assets, needs, attitudes, and behaviors of residents of Ferrocarril #4, a small urban community in Tamaulipas, Mexico. A collaborative Participatory Action Research approach was used to emphasize community involvement. Using Triangulation to ensure validity, qualitative methods included key informant in depth interviews, participant observation and participatory discussion groups with women and men. A personal interview with a probability sample of women was done. The median age of interviewees was 37 years. The majority was married or had a partner. Over half of respondents completed grades 6-9. Employed women (25%) earned a median weekly salary equivalent to ∼56 USD. Women with health insurance (67.7%) were covered mainly through Social Security and Seguro Popular. One in 5 reported bad health. Barriers to care were primarily money and transportation. To improve health care, women wanted a full service clinic in or close to the community and affordable health care. Socially, 28% of respondents had no close friends in the community and most did not participate in beneficial community activities. Many women did not socialize with others and help from neighbors was situational. Primary school teachers lacked parental support and it interfered with classroom efforts. Healthy community discussion groups focused on personal and environmental hygiene and safety. Valuable assets exist in the community. To date, collaborative efforts resulted in a school First Aid station, a school nurse visit weekly, posting of emergency contact phone numbers in the school and community center, and development of a student health information form. ^

Counting Highly Cited Papers for University Research Assessment: Conceptual and Technical Issues

A Kuhnian approach to research assessment requires us to consider that the important scientific breakthroughs that drive scientific progress are infrequent and that the progress of science does not depend on normal research. Consequently, indicators of research performance based on the total number of papers do not accurately measure scientific progress. Similarly, those universities with the best reputations in terms of scientific progress differ widely from other universities in terms of the scale of investments made in research and in the higher concentrations of outstanding scientists present, but less so in terms of the total number of papers or citations. This study argues that indicators for the 1% high-citation tail of the citation distribution reveal the contribution of universities to the progress of science and provide quantifiable justification for the large investments in research made by elite research universities. In this tail, which follows a power low, the number of the less frequent and highly cited important breakthroughs can be predicted from the frequencies of papers in the upper part of the tail. This study quantifies the false impression of excellence produced by multinational papers, and by other types of papers that do not contribute to the progress of science. Many of these papers are concentrated in and dominate lists of highly cited papers, especially in lower-ranked universities. The h-index obscures the differences between higher- and lower-ranked universities because the proportion of h-core papers in the 1% high-citation tail is not proportional to the value of the h-index.

The state of the field in UK management research: reflections of the Research Assessment Exercise (RAE) panel

This paper reviews the state of the field of the sub-disciplines within UK management research, based upon the submissions of 94 UK higher education institutions to the Business and Management Studies Panel in the UK's 2001 Research Assessment Exercise (RAE). It offers observations on the UK model of the assessment of quality in, and funding of, research conducted in publicly funded higher education institutions.

Making sense of research quality assessment

Purpose – For many academics in UK universities the nature and orientation of their research is overwhelmingly determined by considerations of how that work will be graded in research assessment exercises (RAEs). The grades awarded to work in a particular subject area can have a considerable impact on the individual and their university. There is a need to better understand those factors which may influence these grades. The paper seeks to address this issue. Design/methodology/approach – The paper considers relationships between the grades awarded and the quantitative information provided to the assessment panels for the 1996 and 2001 RAEs for two subject areas, built environment and town and country planning, and for three other subject areas, civil engineering, geography and archaeology, in the 2001 RAE. Findings – A simple model demonstrating strong and consistent relationships is established. RAE performance relates to numbers of research active staff, the production of books and journal papers, numbers of research studentships and graduations, and research income. Important differences between subject areas are identified. Research limitations/implications – Important issues are raised about the extent to which the new assessment methodology to be adopted for the 2008 RAE will capture the essence of good quality research in architecture and built environment. Originality/value – The findings provide a developmental perspective of RAEs and show how, despite a changed methodology, various research activities might be valued in the 2008 RAE. The basis for a methodology for reviewing the credibility of the judgements of panels is proposed.

Rank and file : assessing research quality in Australia

In this paper, the author describes recent developments in the assessment of research activity and publication in Australia. Of particular interest to readers will be the move to rank academic journals. Educational Philosophy and Theory (EPAT) received the highest possible ranking, however, the process is far from complete. Some implications for the field, for this journal and particularly, for the educational foundations are discussed.