Outdated journal rankings and the ERA exercise

As the results of the latest Excellence in Research Australia (ERA) exercise come closer to being announced, universities around Australia are holding their collective breaths. The ERA claims to be an assessment of research strengths and quality at Australian universities. While it is not supposed to produce a set of league tables, ultimately that is what tends to happen...

Multi-dimensional examination of marketing journal rankings by North American academics

This paper extends the journal ranking debate by developing weighted multi-dimensional perceptual rankings based on respondents' evaluation of a journals' prestige, contribution to theory, contribution to practice and contribution to teaching. Spearman correlations indicate that journal rankings based on each of the evaluative criteria are statistically significantly correlated for all pairs of criteria, other than teaching and theory. The multi-dimensional evaluation is also statistically significantly correlated to all individual criteria, other than the teaching dimension. A cluster analysis of the journals using the four evaluative criteria identified that there are three groupings of journals: A, B, and C. In most cases, A journals are evaluated as performing better than B journals on each of evaluative criteria other than for the contribution to teaching where there are no difference in perceptions between groups. B journals are viewed to perform better than C journals on prestige, contribution to practice and contribution to teaching, but not based on their prestige of weighted evaluation. This suggests that there are differences in how academics view each group of journals, although these differences do seem to vary based on the grouping being considered (i.e., A, B, or C).

Journal rankings : does one size fit all?

The question of ranking journals, institutions, individual academics and specific papers is regularly visited within business and management disciplines. The Mort, et al. (2004) study advances the discussion by examining this theme within the context of academic marketing in Australia and New Zealand. In this commentary I comment on three aspects of the study that are left partially unanswered: why rank journals in the first place, how should journals be evaluated, and to what extent are the aspirations of academics in the region met?

Citation patterns within the leading top-tier finance journals implications for journal rankings and other issues

This paper examines all citations and self-citations to a list of 95 finance journals appearing in the Journal of Finance, Journal of Financial Economics and Review of Financial Studies from 1995 through 2005. Additionally, the publication profile of 100 prolific authors in top-tier finance journals is tabulated for these 95 journals. Citations to non-finance journals in economics and accounting are also tabulated for comparison with their finance counterpart along with working papers. Six ranking schemes are constructed with each scheme identifying the top fifty finance journals. Citations to finance journals are highly concentrated within ten journals and similarly for self-citations. Authors of papers appearing in top-tier finance journals pay scant attention to the bulk of research published in other finance journals. Furthermore, these authors cite other economic journals with greater frequency than their counterpart in finance. Of the top fifty finance journals identified in this paper, only 21 are listed in Social Sciences Citation Index (SSCI), and this compares to approximately 500 listed economic journals. Some glaring omissions from SSCI are identified, but most notably the Journal of Applied Corporate Finance, Journal of Financial Research, Journal of Empirical Finance and Journal of Fixed Income. An analysis of 2006 citations patterns is also presented. The top-tier mantra assigned to finance journals has a void with the decision by the Journal of Business to cease publication with the November 2006 issue. This paper identifies five finance journals anyone of which could potentially fill the void.

Citation based journal rankings : an application of fuzzy measures

Here we investigate the use of fuzzy measures and averaging aggregation functions for understanding the behavior and tendencies of decision-makers in an ordinal classification problem. Using the Aotools package to approximate the data, we classify each journal based on aggregation of the ISI Web of knowledge indices and discuss the results.

Explicit and implicit subject bias in the ABS journal quality guide

This paper addresses issues raised in two recent papers published in this journal about the UK Association of Business Schools' Journal Quality Guide (ABS Guide). While much of the debate about journal rankings in general, and the ABS Guide in particular, has focused on the construction, power and (mis)use of these rankings, this paper differs in that it explains and provides evidence about explicit and implicit biases in the ABS Guide. In so doing, it poses potentially difficult questions that the editors of the ABS Guide need to address and urgently rectify if the ABS Guide seeks to build and retain legitimacy. In particular, the evidence in this paper shows explicit bias in the ABS Guide against several subject areas, including accounting and finance. It also shows implicit bias against accounting and finance when comparing journal rankings in sub-areas shared between accounting and finance and the broader business management subject areas.

Discovering implicit criteria preferences reflected in tourism journal ranking lists using the Choquet Integral

Variations between journal rankings may cause confusion. As such, prior attempts were made to compare and evaluate journal ranking criteria for obtaining insightful knowledge on how different research communities have ranked journals. However, existing approaches are unable to model the journal ranking process closely enough as they are incapable of considering the relationship between multiple criteria simultaneously. In this paper, we address the challenges by introducing the Choquet Integral (CI) for evaluating journal ranking criteria. The new approach is able to account for interactions between criteria in relation to overall ranking score, using a fuzzy measure in its computation. Its properties, the Shapley value and the Interaction index, allow for good representations of importance and interactions between criteria. We demonstrate the efficiency of the CI through a case study of journal ranking lists in tourism and service journals.

Rising Publication Delays Inflate Journal Impact Factors

Journal impact factors have become an important criterion to judge the quality of scientific publications over the years, influencing the evaluation of institutions and individual researchers worldwide. However, they are also subject to a number of criticisms. Here we point out that the calculation of a journal’s impact factor is mainly based on the date of publication of its articles in print form, despite the fact that most journals now make their articles available online before that date. We analyze 61 neuroscience journals and show that delays between online and print publication of articles increased steadily over the last decade. Importantly, such a practice varies widely among journals, as some of them have no delays, while for others this period is longer than a year. Using a modified impact factor based on online rather than print publication dates, we demonstrate that online-to-print delays can artificially raise a journal’s impact factor, and that this inflation is greater for longer publication lags. We also show that correcting the effect of publication delay on impact factors changes journal rankings based on this metric. We thus suggest that indexing of articles in citation databases and calculation of citation metrics should be based on the date of an article’s online appearance, rather than on that of its publication in print.

Exploring diversity in accounting through Faculty journal perceptions

The accounting research community has frequently been described as being both diverse and focused on local issues. At the same time, increasing pressure is being placed on researchers to publish in internationally highly regarded journals. Since faculty evaluations depend on journal rankings, such rankings need to take into account the diversity of the research community. Therefore, this study examines how contextual factors such as a researcher's location and research orientation may influence journal quality perceptions and readership patterns based on an international sample of 1,230 accounting academics. The perceived quality of journals is measured across a number of dimensions, including journal familiarity, average rank position, percent of respondents who classify a journal as top tier, and readership. The results support that a significant variation in journal quality perceptions exists based on a researcher's geographic origin, research orientation, and affiliation with a journal.

Insights into factors affecting production and operations management (POM) journal evaluation

The paper extends the current literature on peer review journal evaluations by providing a number of insights based on the diversity of Production and Operations Management (POM) research. We provide peer review evaluations for POM research outlets, based on a sampling frame that includes a large number of POM researchers worldwide. More specifically, the paper develops and tests various hypotheses as to whether the perceived quality and relevance of a journal is affected by such factors as: (i) nature of research work (empiricists versus modelers), (ii) society membership, (iii) research productivity, (iv) geographical location, and (v) seniority. Our findings suggest that caution must be exercised when utilizing existing POM journal rankings, as some factors, particularly the difference between empiricists and modelers, may influence journal evaluation. These must be considered when addressing issues such as faculty promotions, tenure, and salary. © 2006 Elsevier B.V. All rights reserved.

Rising Publication Delays Inflate Journal Impact Factors

Journal impact factors have become an important criterion to judge the quality of scientific publications over the years, influencing the evaluation of institutions and individual researchers worldwide. However, they are also subject to a number of criticisms. Here we point out that the calculation of a journal’s impact factor is mainly based on the date of publication of its articles in print form, despite the fact that most journals now make their articles available online before that date. We analyze 61 neuroscience journals and show that delays between online and print publication of articles increased steadily over the last decade. Importantly, such a practice varies widely among journals, as some of them have no delays, while for others this period is longer than a year. Using a modified impact factor based on online rather than print publication dates, we demonstrate that online-to-print delays can artificially raise a journal’s impact factor, and that this inflation is greater for longer publication lags. We also show that correcting the effect of publication delay on impact factors changes journal rankings based on this metric. We thus suggest that indexing of articles in citation databases and calculation of citation metrics should be based on the date of an article’s online appearance, rather than on that of its publication in print.

Rising Publication Delays Inflate Journal Impact Factors

Journal impact factors have become an important criterion to judge the quality of scientific publications over the years, influencing the evaluation of institutions and individual researchers worldwide. However, they are also subject to a number of criticisms. Here we point out that the calculation of a journal’s impact factor is mainly based on the date of publication of its articles in print form, despite the fact that most journals now make their articles available online before that date. We analyze 61 neuroscience journals and show that delays between online and print publication of articles increased steadily over the last decade. Importantly, such a practice varies widely among journals, as some of them have no delays, while for others this period is longer than a year. Using a modified impact factor based on online rather than print publication dates, we demonstrate that online-to-print delays can artificially raise a journal’s impact factor, and that this inflation is greater for longer publication lags. We also show that correcting the effect of publication delay on impact factors changes journal rankings based on this metric. We thus suggest that indexing of articles in citation databases and calculation of citation metrics should be based on the date of an article’s online appearance, rather than on that of its publication in print.

What are we measuring when we evaluate journals?

This article undertakes two studies to examine issues related to journal rankings. Study 1 examines the consistency between journal rankings reported in past studies. It finds that while there is consistency when comparing these studies, this consistency does not always occur outside the top-ranked journals. Study 2 explores whether individuals believe that the weighting of four underlying evaluative criteria—that is, prestige, contribution to theory, contribution to practice, and contribution to teaching—vary, based on (1) whose criteria are used (individual or individuals’ perception of their institutions weighting), (2) the geographic region in which the individuals teach (North America, Europe, and Asia-Pacific), and (3) whether or not an individual works at an institution offering a Ph.D./D.B.A. The results suggest that some differences in criteria weighting exist. Implications are discussed, with it being suggested that it may not be possible to develop a universally applicable set of journal rankings.

The downside of relying on research outputs to assess business faculty performance : comments from down under regarding facilitating and creating synergies between teaching and research : the role of the academic administrator

This commentary presents an Australian perspective on Balkin and Mello’s “Facilitating and Creating Synergies between Teaching and Research: The Role of the Academic Administrator.” It addresses one particularly important aspect of the separation of teaching and research in business schools; namely, the increasing dominance of discipline-based research output measures in overall business faculty performance, coupled with a reliance on journal rankings as de facto quality measures for research output. Some possible alternate approaches to research performance/impact measurement in our rapidly changing academic environment are also considered.

An insight into research performance through a citation counting analysis

Prior studies on academic performance predominantly concentrated on ranking universities and geographical regions using publications in selected journals. Despite general agreement on journal rankings based on the number of citations, no extant articles analyze universities or countries on the basis of citations from publications in leading hospitality and tourism journals. This paper examined the number of citations that published articles from six leading journals in hospitality and tourism received during the 10-year period from 1996 to 2005. The affiliated universities, countries/regions, and geographical continents were then ranked to determine their level of academic performance. This paper provides an alternative insight into academic performance of research universities and countries.© 2014 The Authors.