Appraising qualitative research for inclusion in systematic reviews:A quantitative and qualitative comparison of three methods

Objective: Qualitative research is increasingly valued as part of the evidence for policy and practice, but how it should be appraised is contested. Various appraisal methods, including checklists and other structured approaches, have been proposed but rarely evaluated. We aimed to compare three methods for appraising qualitative research papers that were candidates for inclusion in a systematic review of evidence on support for breast-feeding. Method: A sample of 12 research papers on support for breast-feeding was appraised by six qualitative reviewers using three appraisal methods: unprompted judgement, based on expert opinion; a UK Cabinet Office quality framework; and CASP, a Critical Appraisal Skills Programme tool. Papers were assigned, following appraisals, to 1 of 5 categories, which were dichotomized to indicate whether or not papers should be included in a systematic review. Patterns of agreement in categorization of papers were assessed quantitatively using κ statistics, and qualitatively using cross-case analysis. Results: Agreement in categorizing papers across the three methods was slight (κ =0.13; 95% CI 0.06-0.24). Structured approaches did not appear to yield higher agreement than that by unprompted judgement. Qualitative analysis revealed reviewers' dilemmas in deciding between the potential impact of findings and the quality of the research execution or reporting practice. Structured instruments appeared to make reviewers more explicit about the reasons for their judgements. Conclusions: Structured approaches may not produce greater consistency of judgements about whether to include qualitative papers in a systematic review. Future research should address how appraisals of qualitative research should be incorporated in systematic reviews. © The Royal Society of Medicine Press Ltd 2007.

A discussion of ethnographic research methods and their relevance for the translation process

In recent years, technologically advanced methodologies such as Translog have gained a lot of ground in translation process research. However, in this paper it will be argued that quantitative research methods can be supplemented by ethnographic qualitative ones so as to enhance our understanding of what underlies the translation process. Although translation studies scholars have sometimes applied an ethnographic approach to the study of translation, this paper offers a different perspective and considers the potential of ethnographic research methods for tapping cognitive and behavioural aspects of the translation process. A number of ethnographic principles are discussed and it is argued that process researchers aiming to understand translators’ perspectives and intentions, how these shape their behaviours, as well as how translators reflect on the situations they face and how they see themselves, would undoubtedly benefit from adopting an ethnographic framework for their studies on translation processes.

The problem of appraising qualitative research

Qualitative research can make a valuable contribution to the study of quality and safety in health care. Sound ways of appraising qualitative research are needed, but currently there are many different proposals with few signs of an emerging consensus. One problem has been the tendency to treat qualitative research as a unified field. We distinguish universal features of quality from those specific to methodology and offer a set of minimally prescriptive prompts to assist with the assessment of generic features of qualitative research. In using these, account will need to be taken of the particular method of data collection and methodological approach being used. There may be a need for appraisal criteria suited to the different methods of qualitative data collection and to different methodological approaches. These more specific criteria would help to distinguish fatal flaws from more minor errors in the design, conduct, and reporting of qualitative research. There will be difficulties in doing this because some aspects of qualitative research, particularly those relating to quality of insight and interpretation, will remain difficult to appraise and will rely largely on subjective judgement.

Finding qualitative research: an evaluation of search strategies

Background Qualitative research makes an important contribution to our understanding of health and healthcare. However, qualitative evidence can be difficult to search for and identify, and the effectiveness of different types of search strategies is unknown. Methods Three search strategies for qualitative research in the example area of support for breast-feeding were evaluated using six electronic bibliographic databases. The strategies were based on using thesaurus terms, free-text terms and broad-based terms. These strategies were combined with recognised search terms for support for breast-feeding previously used in a Cochrane review. For each strategy, we evaluated the recall (potentially relevant records found) and precision (actually relevant records found). Results A total yield of 7420 potentially relevant records was retrieved by the three strategies combined. Of these, 262 were judged relevant. Using one strategy alone would miss relevant records. The broad-based strategy had the highest recall and the thesaurus strategy the highest precision. Precision was generally poor: 96% of records initially identified as potentially relevant were deemed irrelevant. Searching for qualitative research involves trade-offs between recall and precision. Conclusions These findings confirm that strategies that attempt to maximise the number of potentially relevant records found are likely to result in a large number of false positives. The findings also suggest that a range of search terms is required to optimise searching for qualitative evidence. This underlines the problems of current methods for indexing qualitative research in bibliographic databases and indicates where improvements need to be made.

New business histories! Plurality in business history research methods

We agree with de Jong et al.'s argument that business historians should make their methods more explicit and welcome a more general debate about the most appropriate methods for business historical research. But rather than advocating one ‘new business history’, we argue that contemporary debates about methodology in business history need greater appreciation for the diversity of approaches that have developed in the last decade. And while the hypothesis-testing framework prevalent in the mainstream social sciences favoured by de Jong et al. should have its place among these methodologies, we identify a number of additional streams of research that can legitimately claim to have contributed novel methodological insights by broadening the range of interpretative and qualitative approaches to business history. Thus, we reject privileging a single method, whatever it may be, and argue instead in favour of recognising the plurality of methods being developed and used by business historians – both within their own field and as a basis for interactions with others.

The experiences of con sultations on oral anticoagula-tion for AF:the importance of qualitative research incardiovascular sciences

Objectives: Multiple-perspective qualitative designs can aid researchersto develop a more multifaceted account of a phenomenon and as aform of triangulation of data. Two interlinking studies aimed toexplore patients’ and physicians’ experiences of atrial fibrillation (AF)and warfarin.Methods: Audio-recorded semistructured individual interviews wereused. Study 1: Three AF patient subgroups were interviewed (n = 11);accepted, refused, or discontinued warfarin. Study 2: Four physiciansubgroups (n = 16): consultant cardiologists, consultant general physi-cians, general practitioners, and cardiology registrars. Data was ana-lyzed using interpretative phenomenological analysis, a qualitativemethodology.Results: Study 1: Three overarching themes comprised patients’ experi-ences: the initial consultation, life after the consultation, and patients’reflections. Patients commented on the reassurance experienced duringthe consultation, but they perceived the decision-making processmostly led by the physician. Lack of education and take-home materi-als during the initial consultation were highlighted. Patients’ uptake ofinformation was influenced by past experiences and knowledge ofstroke and/or bleeding. Study 2: Two overarching themes covered phy-sicians’ experiences: communicating information and challenges withwarfarin prescription for AF. Physicians’ approach to the consultationstyle shifted through a continuum of compliance-adherence-concor-dance during the consultation. Time and the perceived patient trust inthem as the expert led to physicians to adopt a paternalistic approach.Guideline adherence and the need to adopt a multidisciplinaryapproach were pointed out as current challenges.Conclusion: There is a need to target patients’ and physicians’ abilityto communicate with each other in a comprehensible way. This projecthas illustrated the benefit of using a qualitative approach to under-stand the lived experience of the physician–patient consultation.Disclosure of Interest: None declare

"It's no skin off my nose":Why people take part in qualitative research

In this article, the authors analyze participants' accounts of why they took part in a repeat-interview study exploring newly diagnosed patients' perceptions of diabetes service provision in Lothian, Scotland. The study involved three semistructured in-depth interviews with each patient (N = 40), which spanned a year. The authors provide a thematic discursive analysis of responses to the question, Can I ask you what made you decide to part in the study and why you've stayed involved over the past year? The main themes are (a) recruitment within health contexts ("the nurse said it would help"), (b) altruism ("if it can help somebody"), (c) qualitative research being seen as inherently innocuous ("nothing to lose"), and (d) therapeutic aspects of interviewing ("getting it off my chest"). The analysis contributes both to the qualitative literature about generic research participation and to a germinal literature exploring qualitative health research participation. © 2006 Sage Publications.

How can systematic reviews incorporate qualitative research? A critical perspective

Systematic review has developed as a specific methodology for searching for, appraising and synthesizing findings of primary studies, and has rapidly become a cornerstone of the evidence-based practice and policy movement. Qualitative research has traditionally been excluded from systematic reviews, and much effort is now being invested in resolving the daunting methodological and epistemological challenges associated with trying to move towards more inclusive forms of review. We describe our experiences, as a very diverse multidisciplinary group, in attempting to incorporate qualitative research in a systematic review of support for breastfeeding. We show how every stage of the review process, from asking the review question through to searching for and sampling the evidence, appraising the evidence and producing a synthesis, provoked profound questions about whether a review that includes qualitative research can remain consistent with the frame offered by current systematic review methodology. We conclude that more debate and dialogue between the different communities that wish to develop review methodology is needed, and that attempts to impose dominant views about the appropriate means of conducting reviews of qualitative research should be resisted so that innovation can be fostered. Copyright © 2006 SAGE Publications.

What do we know about the experience of age related macular degeneration? A systematic review and meta-synthesis of qualitative research

Age Related Macular Degeneration (AMD) is the leading cause of registerable blindness with a high medical and societal cost burden. Much of the research examining experiences of living with AMD has been conducted independently with small sample sizes and has failed to impact on practice. Meta-synthesis of qualitative research can improve the understanding of the experience of living with AMD by drawing together findings of qualitative studies. This article presents a systematic review and meta-synthesis of qualitative studies investigating the experience of AMD (literature searched up to April 2012; published studies identified range from 1996 to 2009). The review highlights themes relating to: functional limitations, adaptation and independence; feelings about the future with vision impairment; interaction with the health service; social engagement; disclosure; and the emotional impacts of living with AMD. Attention to the experience of living with AMD can help us to better understand the needs of patients. This meta-synthesis aimed to bring together the findings of qualitative research studies and highlights important areas for consideration when caring for patients with AMD. Our findings suggest that a holistic approach to service provision and support for AMD is needed which takes into account individuals' needs and experiences when coping with and adjusting to living with AMD. This support should aim to reduce stigma, increase social engagement, and develop the psychological resources of patients with AMD.

Identifying frames:a comparison of research methods

Framing plays an important role in public policy. Interest groups strategically highlight some aspects of a policy proposal while downplaying others in order to steer the policy debate in a favorable direction. Despite the importance of framing, we still know relatively little about the framing strategies of interest groups due to methodological difficulties that have prevented scholars from systematically studying interest group framing across a large number of interest groups and multiple policy debates. This article therefore provides an overview of three novel research methods that allow researchers to systematically measure interest group frames. More specifically, this article introduces a word-based quantitative text analysis technique, a manual, computer-assisted content analysis approach and face-to-face interviews designed to systematically identify interest group frames. The results generated by all three techniques are compared on the basis of a case study of interest group framing in an environmental policy debate in the European Union.

The computerisation of community pharmacy

Computers have, over the past 10 to 15 years, become an integral part of many activities carried out by British community pharmacists. This thesis employs quantitative and qualitative research methods to explore the use of computers and other forms of information technology (IT) in a number of these activities. Mail questionnaires were used to estimate the level of IT use among British community pharmacists in 1989 and 1990. Comparison of the results suggests that the percentage of community pharmacists using computers and other forms of IT is increasing, and that the range of applications to which pharmacy computers are put is expanding. The use of an electronic, on-line information service, PINS, by community pharmacists was investigated using mail questionnaires. The majority of community pharmacists who subscribed to the service, and who responded to the questionnaire, claimed to use PINS less than they had expected to. In addition, most did not find it user-friendly. A computer program to aid pharmacists when responding to their patients' symptoms was investigated using interviews and direct observation. The aid was not found to help pharmacists in responding to patients' symptoms because of impracticalities involved in its operation. Use of the same computer program by members of the public without the involvement of a pharmacist was also studied. In this setting, the program was favourably accepted by the majority of those who used it. Provision of computer generated information leaflets from pharmacies was investigated using mail questionnaires and interviews. The leaflets were found to be popular with the majority of recipients interviewed. Since starting to give out the leaflets, 27 out of 55 pharmacists who responded to the questionnaire had experienced an increase in the numbers of prescriptions they dispensed. 46 had experienced an increase in the number of patient enquiries they received.