ASSESSING THE UNRELIABILITY OF THE MEDICAL LITERATURE: A RESPONSE TO "WHY MOST PUBLISHED RESEARCH FINDINGS ARE FALSE"

A recent article in this journal (Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124) argued that more than half of published research findings in the medical literature are false. In this commentary, we examine the structure of that argument, and show that it has three basic components: 1)An assumption that the prior probability of most hypotheses explored in medical research is below 50%. 2)Dichotomization of P-values at the 0.05 level and introduction of a “bias” factor (produced by significance-seeking), the combination of which severely weakens the evidence provided by every design. 3)Use of Bayes theorem to show that, in the face of weak evidence, hypotheses with low prior probabilities cannot have posterior probabilities over 50%. Thus, the claim is based on a priori assumptions that most tested hypotheses are likely to be false, and then the inferential model used makes it impossible for evidence from any study to overcome this handicap. We focus largely on step (2), explaining how the combination of dichotomization and “bias” dilutes experimental evidence, and showing how this dilution leads inevitably to the stated conclusion. We also demonstrate a fallacy in another important component of the argument –that papers in “hot” fields are more likely to produce false findings. We agree with the paper’s conclusions and recommendations that many medical research findings are less definitive than readers suspect, that P-values are widely misinterpreted, that bias of various forms is widespread, that multiple approaches are needed to prevent the literature from being systematically biased and the need for more data on the prevalence of false claims. But calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that “most research findings are false for most research designs and for most fields” must be considered as yet unproven.

Low prevalence of SV40 in Swiss mesothelioma patients after elimination of false-positive PCR results

The association of simian virus 40 (SV40) with malignant pleural mesothelioma is currently under debate. In some malignancies of viral aetiology, viral DNA can be detected in the patients' serum or plasma. To characterize the prevalence of SV40 in Swiss mesothelioma patients, we optimized a real-time PCR for quantitative detection of SV40 DNA in plasma, and used a monoclonal antibody for immunohistochemical detection of SV40 in mesothelioma tissue microarrays. Real-time PCR was linear over five orders of magnitude, and sensitive to a single gene copy. Repeat PCR determinations showed excellent reproducibility. However, SV40 status varied for independent DNA isolates of single samples. We noted that SV40 detection rates by PCR were drastically reduced by the implementation of strict room compartmentalization and decontamination procedures. Therefore, we systematically addressed common sources of contamination and found no cross-reactivity with DNA of other polyomaviruses. Contamination during PCR was rare and plasmid contamination was infrequent. SV40 DNA was reproducibly detected in only 4 of 78 (5.1%) plasma samples. SV40 DNA levels were low and not consistently observed in paired plasma and tumour samples from the same patient. Immunohistochemical analysis revealed a weak but reproducible SV40 staining in 16 of 341 (4.7%) mesotheliomas. Our data support the occurrence of non-reproducible SV40 PCR amplifications and underscore the importance of proper sample handling and analysis. SV40 DNA and protein were found at low prevalence (5%) in plasma and tumour tissue, respectively. This suggests that SV40 does not appear to play a major role in the development of mesothelioma.

Screening for carriage of methicillin-resistant Staphylococcus aureus shortly after exposure may lead to false-negative results

We evaluated a double screening strategy for carriage of methicillin-resistant Staphylococcus aureus (MRSA) in patients exposed to a newly detected MRSA carrier. If the first screening of the exposed patient yielded negative results, screening was repeated 4 days later. This strategy detected 12 (28%) of the 43 new MRSA carriers identified during the study period. The results suggest that there is an incubation period before MRSA carriage is detectable.

The Role of Consciousness in Memory

Conscious events interact with memory systems in learning, rehearsal and retrieval (Ebbinghaus 1885/1964; Tulving 1985). Here we present hypotheses that arise from the IDA computional model (Franklin, Kelemen and McCauley 1998; Franklin 2001b) of global workspace theory (Baars 1988, 2002). Our primary tool for this exploration is a flexible cognitive cycle employed by the IDA computational model and hypothesized to be a basic element of human cognitive processing. Since cognitive cycles are hypothesized to occur five to ten times a second and include interaction between conscious contents and several of the memory systems, they provide the means for an exceptionally fine-grained analysis of various cognitive tasks. We apply this tool to the small effect size of subliminal learning compared to supraliminal learning, to process dissociation, to implicit learning, to recognition vs. recall, and to the availability heuristic in recall. The IDA model elucidates the role of consciousness in the updating of perceptual memory, transient episodic memory, and procedural memory. In most cases, memory is hypothesized to interact with conscious events for its normal functioning. The methodology of the paper is unusual in that the hypotheses and explanations presented are derived from an empirically based, but broad and qualitative computational model of human cognition.

Exploring "fringe" consciousness: the subjective experience of perceptual fluency and its objective bases

Perceptual fluency is the subjective experience of ease with which an incoming stimulus is processed. Although perceptual fluency is assessed by speed of processing, it remains unclear how objective speed is related to subjective experiences of fluency. We present evidence that speed at different stages of the perceptual process contributes to perceptual fluency. In an experiment, figure-ground contrast influenced detection of briefly presented words, but not their identification at longer exposure durations. Conversely, font in which the word was written influenced identification, but not detection. Both contrast and font influenced subjective fluency. These findings suggest that speed of processing at different stages condensed into a unified subjective experience of perceptual fluency.