The eight-item modified Medical Outcomes Study Social Support Survey: psychometric evaluation showed excellent performance

Evaluation and validation of the psychometric properties of the eight-item modified Medical Outcomes Study Social Support Survey (mMOS-SS).

New methods can extend the use of minimal important difference units in meta-analyses of continuous outcome measures

For continuous outcomes measured using instruments with an established minimally important difference (MID), pooled estimates can be usefully reported in MID units. Approaches suggested thus far omit studies that used instruments without an established MID. We describe an approach that addresses this limitation.

Bivariate meta-analysis of predictive values of diagnostic tests can be an alternative to bivariate meta-analysis of sensitivity and specificity

Meta-analysis of predictive values is usually discouraged because these values are directly affected by disease prevalence, but sensitivity and specificity sometimes show substantial heterogeneity as well. We propose a bivariate random-effects logitnormal model for the meta-analysis of the positive predictive value (PPV) and negative predictive value (NPV) of diagnostic tests.

ASSESSING THE UNRELIABILITY OF THE MEDICAL LITERATURE: A RESPONSE TO "WHY MOST PUBLISHED RESEARCH FINDINGS ARE FALSE"

A recent article in this journal (Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124) argued that more than half of published research findings in the medical literature are false. In this commentary, we examine the structure of that argument, and show that it has three basic components: 1)An assumption that the prior probability of most hypotheses explored in medical research is below 50%. 2)Dichotomization of P-values at the 0.05 level and introduction of a “bias” factor (produced by significance-seeking), the combination of which severely weakens the evidence provided by every design. 3)Use of Bayes theorem to show that, in the face of weak evidence, hypotheses with low prior probabilities cannot have posterior probabilities over 50%. Thus, the claim is based on a priori assumptions that most tested hypotheses are likely to be false, and then the inferential model used makes it impossible for evidence from any study to overcome this handicap. We focus largely on step (2), explaining how the combination of dichotomization and “bias” dilutes experimental evidence, and showing how this dilution leads inevitably to the stated conclusion. We also demonstrate a fallacy in another important component of the argument –that papers in “hot” fields are more likely to produce false findings. We agree with the paper’s conclusions and recommendations that many medical research findings are less definitive than readers suspect, that P-values are widely misinterpreted, that bias of various forms is widespread, that multiple approaches are needed to prevent the literature from being systematically biased and the need for more data on the prevalence of false claims. But calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that “most research findings are false for most research designs and for most fields” must be considered as yet unproven.

Matched-pair study showed higher quality of placebo-controlled trials in Western phytotherapy than conventional medicine

OBJECTIVES: Herbal medicine (phytotherapy) is widely used, but the evidence for its effectiveness is a matter of ongoing debate. We compared the quality and results of trials of Western phytotherapy and conventional medicine. STUDY DESIGN AND SETTING: A random sample of placebo-controlled trials of Western phytotherapy was identified in a comprehensive literature search (19 electronic databases). Conventional medicine trials matched for condition and type of outcome were selected from the Cochrane Central Controlled Trials Register (issue 1, 2003). Data were extracted in duplicate. Trials described as double-blind, with adequate generation of allocation sequence and adequate concealment of allocation were assumed to be of higher methodological quality. RESULTS: Eighty-nine herbal medicine and 89 matched conventional medicine trials were analyzed. Studies of Western herbalism were smaller, less likely to be published in English, and less likely to be indexed in MEDLINE than their counterparts from conventional medicine. Nineteen (21%) herbal and four (5%) conventional medicine trials were of higher quality. In both groups, smaller trials showed more beneficial treatment effects than larger trials. CONCLUSIONS: Our findings challenge the widely held belief that the quality of the evidence on the effectiveness of herbal medicine is generally inferior to the evidence available for conventional medicine.

Bibliographic study showed improving methodology of meta-analyses published in leading journals 1993-2002

OBJECTIVE: To assess the methodology of meta-analyses published in leading general and specialist medical journals over a 10-year period. STUDY DESIGN AND SETTING: Volumes 1993-2002 of four general medicine journals and four specialist journals were searched by hand for meta-analyses including at least five controlled trials. Characteristics were assessed using a standardized questionnaire. RESULTS: A total of 272 meta-analyses, which included a median of 11 trials (range 5-195), were assessed. Most (81%) were published in general medicine journals. The median (range) number of databases searched increased from 1 (1-9) in 1993/1994 to 3.5 (1-21) in 2001/2002, P<0.0001. The proportion of meta-analyses including searches by hand (10% in 1993/1994, 25% in 2001/2002, P=0.005), searches of the grey literature (29%, 51%, P=0.010 by chi-square test), and of trial registers (10%, 32%, P=0.025) also increased. Assessments of the quality of trials also became more common (45%, 70%, P=0.008), including whether allocation of patients to treatment groups had been concealed (24%, 60%, P=0.001). The methodological and reporting quality was consistently higher in general medicine compared to specialist journals. CONCLUSION: Many meta-analyses published in leading journals have important methodological limitations. The situation has improved in recent years but considerable room for further improvements remains.

A review identifies and classifies reasons for ordering diagnostic tests

OBJECTIVE: To consider the reasons and context for test ordering by doctors when faced with an undiagnosed complaint in primary or secondary care. STUDY DESIGN AND SETTING: We reviewed any study of any design that discussed factors that may affect a doctor's decision to order a test. Articles were located through searches of electronic databases, authors' files on diagnostic methodology, and reference lists of relevant studies. We extracted data on: study design, type of analysis, setting, topic area, and any factors reported to influence test ordering. RESULTS: We included 37 studies. We carried out a thematic analysis to synthesize data. Five key groupings arose from this process: diagnostic factors, therapeutic and prognostic factors, patient-related factors, doctor-related factors, and policy and organization-related factors. To illustrate how the various factors identified may influence test ordering we considered the symptom low back pain and the diagnosis multiple sclerosis as examples. CONCLUSIONS: A wide variety of factors influence a doctor's decision to order a test. These are integral to understanding diagnosis in clinical practice. Traditional diagnostic accuracy studies should be supplemented with research into the broader context in which doctors perform their work.

An empirical comparison of methods for meta-analysis of diagnostic accuracy showed hierarchical models are necessary

OBJECTIVE: Meta-analysis of studies of the accuracy of diagnostic tests currently uses a variety of methods. Statistically rigorous hierarchical models require expertise and sophisticated software. We assessed whether any of the simpler methods can in practice give adequately accurate and reliable results. STUDY DESIGN AND SETTING: We reviewed six methods for meta-analysis of diagnostic accuracy: four simple commonly used methods (simple pooling, separate random-effects meta-analyses of sensitivity and specificity, separate meta-analyses of positive and negative likelihood ratios, and the Littenberg-Moses summary receiver operating characteristic [ROC] curve) and two more statistically rigorous approaches using hierarchical models (bivariate random-effects meta-analysis and hierarchical summary ROC curve analysis). We applied the methods to data from a sample of eight systematic reviews chosen to illustrate a variety of patterns of results. RESULTS: In each meta-analysis, there was substantial heterogeneity between the results of different studies. Simple pooling of results gave misleading summary estimates of sensitivity and specificity in some meta-analyses, and the Littenberg-Moses method produced summary ROC curves that diverged from those produced by more rigorous methods in some situations. CONCLUSION: The closely related hierarchical summary ROC curve or bivariate models should be used as the standard method for meta-analysis of diagnostic accuracy.

Multivariable adjustments counteract spectrum and test review bias in accuracy studies

OBJECTIVES: The STAndards for Reporting studies of Diagnostic accuracy (STARD) for investigators and editors and the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) for reviewers and readers offer guidelines for the quality and reporting of test accuracy studies. These guidelines address and propose some solutions to two major threats to validity: spectrum bias and test review bias. STUDY DESIGN AND SETTING: Using a clinical example, we demonstrate that these solutions fail and propose an alternative solution that concomitantly addresses both sources of bias. We also derive formulas that prove the generality of our arguments. RESULTS: A logical extension of our ideas is to extend STARD item 23 by adding a requirement for multivariable statistical adjustment using information collected in QUADAS items 1, 2, and 12 and STARD items 3-5, 11, 15, and 18. CONCLUSION: We recommend reporting not only variation of diagnostic accuracy across subgroups (STARD item 23) but also the effects of the multivariable adjustments on test performance. We also suggest that the QUADAS be supplemented by an item addressing the appropriateness of statistical methods, in particular whether multivariable adjustments have been included in the analysis.

Hierarchical modeling gave plausible estimates of associations between metabolic syndrome and components of antiretroviral therapy

OBJECTIVE: Hierarchical modeling has been proposed as a solution to the multiple exposure problem. We estimate associations between metabolic syndrome and different components of antiretroviral therapy using both conventional and hierarchical models. STUDY DESIGN AND SETTING: We use discrete time survival analysis to estimate the association between metabolic syndrome and cumulative exposure to 16 antiretrovirals from four drug classes. We fit a hierarchical model where the drug class provides a prior model of the association between metabolic syndrome and exposure to each antiretroviral. RESULTS: One thousand two hundred and eighteen patients were followed for a median of 27 months, with 242 cases of metabolic syndrome (20%) at a rate of 7.5 cases per 100 patient years. Metabolic syndrome was more likely to develop in patients exposed to stavudine, but was less likely to develop in those exposed to atazanavir. The estimate for exposure to atazanavir increased from hazard ratio of 0.06 per 6 months' use in the conventional model to 0.37 in the hierarchical model (or from 0.57 to 0.81 when using spline-based covariate adjustment). CONCLUSION: These results are consistent with trials that show the disadvantage of stavudine and advantage of atazanavir relative to other drugs in their respective classes. The hierarchical model gave more plausible results than the equivalent conventional model.

Cohort study of trials submitted to ethics committee identified discrepant reporting of outcomes in publications

OBJECTIVES To identify factors associated with discrepant outcome reporting in randomized drug trials. STUDY DESIGN AND SETTING Cohort study of protocols submitted to a Swiss ethics committee 1988-1998: 227 protocols and amendments were compared with 333 matching articles published during 1990-2008. Discrepant reporting was defined as addition, omission, or reclassification of outcomes. RESULTS Overall, 870 of 2,966 unique outcomes were reported discrepantly (29.3%). Among protocol-defined primary outcomes, 6.9% were not reported (19 of 274), whereas 10.4% of reported outcomes (30 of 288) were not defined in the protocol. Corresponding percentages for secondary outcomes were 19.0% (284 of 1,495) and 14.1% (334 of 2,375). Discrepant reporting was more likely if P values were <0.05 compared with P ≥ 0.05 [adjusted odds ratio (aOR): 1.38; 95% confidence interval (CI): 1.07, 1.78], more likely for efficacy compared with harm outcomes (aOR: 2.99; 95% CI: 2.08, 4.30) and more likely for composite than for single outcomes (aOR: 1.48; 95% CI: 1.00, 2.20). Cardiology (aOR: 2.34; 95% CI: 1.44, 3.79) and infectious diseases (aOR: 1.77; 95% CI: 1.01, 3.13) had more discrepancies compared with all specialties combined. CONCLUSION Discrepant reporting was associated with statistical significance of results, type of outcome, and specialty area. Trial protocols should be made freely available, and the publications should describe and justify any changes made to protocol-defined outcomes.