Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review.

BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. METHODOLOGY/PRINCIPAL FINDINGS: In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

Bewertung des Risikos für Bias in kontrollierten Studien [Assessment of risk of bias in controlled studies].

BACKGROUND: Practicing physicians are faced with many medical decisions daily. These are mainly influenced by personal experience but should also consider patient preferences and the scientific evidence reflected by a constantly increasing number of medical publications and guidelines. With the objective of optimal medical treatment, the concept of evidence-based medicine is founded on these three aspects. It should be considered that there is a high risk of misinterpreting evidence, leading to medical errors and adverse effects without knowledge of the methodological background. OBJECTIVES: This article explains the concept of systematic error (bias) and its importance. Causes and effects as well as methods to minimize bias are discussed. This information should impart a deeper understanding, leading to a better assessment of studies and implementation of its recommendations in daily medical practice. CONCLUSION: Developed by the Cochrane Collaboration, the risk of bias (RoB) tool is an assessment instrument for the potential of bias in controlled trials. Good handling, short processing time, high transparency of judgements and a graphical presentation of findings that is easily comprehensible are among its strengths. Attached to this article the German translation of the RoB tool is published. This should facilitate the applicability for non-experts and moreover, support evidence-based medical decision-making.

Risk aversion and embedding bias

In Selten (1967) ?Strategy Method,? the second mover in the game submits a complete strategy. This basic idea has been exported to nonstrategic experiments, where a participant reports a complete list of contingent decisions, one for each situation or state in a given sequence, out of which one and only one state, randomly selected, will be implemented.In general, the method raises the following concern. If S0 and S1 are two differentsequences of states, and state s is in both S0 and S1, would the participant make the same decision in state s when confronted with S0 as when confronted with S1? If not, the experimental results are suspect of suffering from an ?embedding bias.?We check for embedding biases in elicitation methods of Charles Holt and Susan Laury(Laury and Holt, 2000, and Holt and Laury, 2002), and of the present authors (Bosch-Dom?nech and Silvestre, 1999, 2002, 2006a, b) by appropriately chosen replications of the original experiments. We find no evidence of embedding bias in our work. But in Holt and Laury?s method participants tend to switch earlier to the riskier option when later pairs of lotteries are eliminated from the sequence, suggesting the presence of some embedding bias.

Does uncertainty cause inertia in decision making? An experimental study of the role of regret aversion and indecisiveness

Previous research has shown that often there is clear inertia in individual decision making---that is, a tendency for decision makers to choose a status quo option. I conduct a laboratory experiment to investigate two potential determinants of inertia in uncertain environments: (i) regret aversion and (ii) ambiguity-driven indecisiveness. I use a between-subjects design with varying conditions to identify the effects of these two mechanisms on choice behavior. In each condition, participants choose between two simple real gambles, one of which is the status quo option. I find that inertia is quite large and that both mechanisms are equally important.

A demonstration of attentional bias, using a novel dual task paradigm, towards clinically salient material in recovering alcohol abuse patients?

Background. This study examined whether alcohol abuse patients are characterized either by enhanced schematic processing of alcohol related cues or by an attentional bias towards the processing of alcohol cues. Method. Abstinent alcohol abusers (N = 25) and non-clinical control participants (N = 24) performed a dual task paradigm in which they had to make an odd/even decision to a centrally presented number while performing a peripherally presented lexical decision task. Stimuli on the lexical decision task comprised alcohol words, neutral words and non-words. In addition, participants completed an incidental recall task for the words presented in the lexical decision task. Results. It was found that, in the presence of alcohol related words, the performance of patients on the odd/even decision task was poorer than in the presence of other stimului. In addition, patients displayed slower lexical decision times for alcohol related words. Both groups displayed better recall for alcohol words than for other stimuli. Conclusions. These results are interpreted as supporting neither model of drug cravings. Rather, it is proposed that, in the presence of alcohol stimuli, alcohol abuse patients display a breakdown in the ability to focus attention.

Entropic risk analysis by a high level decision support system (e-FDSS) for construction SMEs

The method of entropy has been useful in evaluating inconsistency on human judgments. This paper illustrates an entropy-based decision support system called e-FDSS to the solution of multicriterion risk and decision analysis in projects of construction small and medium enterprises (SMEs). It is optimized and solved by fuzzy logic, entropy, and genetic algorithms. A case study demonstrated the use of entropy in e-FDSS on analyzing multiple risk criteria in the predevelopment stage of SME projects. Survey data studying the degree of impact of selected project risk criteria on different projects were input into the system in order to evaluate the preidentified project risks in an impartial environment. Without taking into account the amount of uncertainty embedded in the evaluation process; the results showed that all decision vectors are indeed full of bias and the deviations of decisions are finally quantified providing a more objective decision and risk assessment profile to the stakeholders of projects in order to search and screen the most profitable projects.

Bias or rationality? The case of UK commercial real estate investment

Purpose – The purpose of this paper is to test the hypothesis that investment decision making in the UK direct property market does not conform to the assumption of economic rationality underpinning portfolio theory. Design/methodology/approach – The developing behavioural real estate paradigm is used to challenge the idea that investor “man” is able to perform with economic rationality, specifically with reference to the analysis of the spatial dispersion of the entire UK “investible stock” and “investible locations” against observed spatial patterns of institutional investment. Location quotients are derived, combining different data sets. Findings – Considerably greater variation in institutional property holdings is found across the UK than would be expected given the economic and stock characteristics of local areas. This appears to provide evidence of irrationality (in the strict traditional economic sense) in the behaviour of institutional investors, with possible herding underpinning levels of investment that cannot be explained otherwise. Research limitations/implications – Over time a lack of distinction has developed between the cause and effect of comparatively low levels of development and institutional property investment across the regions. A critical examination of decision making and behaviour in practice could break this cycle, and could in turn promote regional economic growth. Originality/value – The entire “population” of observations is used to demonstrate the relationships between economic theory and investor performance exploring, for the first time, stock and local area characteristics.

The influence of a decision support tool on real estate valuations

An appraisal task involves the rendering of market value, an unobservable and hypothetical construct. Direct feedback against this objective is typically not possible, so alternative feedback such as confirmation of previous appraised values may be employed. This may alter the appraiser’s perception of the valuation objective leading to divergence from the appraisal normative model. The real estate literature suggests appraisers have been susceptible to the influence of previous appraised values, often resulting in biased valuations. This research focuses on the efficacy of a decision support tool in eliminating or subduing this bias in the appraisal process.

Present bias predicts participation in payments for environmental services: evidence from a behavioral experiment in Uganda

Farmers are necessary agents in global efforts to conserve the environment now that croplands and pastures together constitute the largest terrestrial system on Earth – covering some 48% of ice-free land surface. Whereas standard economic models predict that farmers will participate in conservation programs so long as they are profitable, empirical findings from behavioral economics point to a number of normally unobservable preferences that may influence the decision-making process. This study tests, for the first time, whether heterogeneity in behavioral preferences correlates with decisions to participate in Payments for Environmental Services (PES) programs. We elicit individual trust and time preferences using economic experiments and link resulting measures to household survey data and participation decisions in a Ugandan PES program. We find that farmers who exhibit a preference for proximate gains – present-biased preferences – are 47.7% more likely to participate in the program than those who show time-consistent or future-biased preferences. This result has implications for ongoing and planned PES programs involving farmers, particularly in Africa, by highlighting a potential relationship between payment timing and participation, and further validates the use of behavioral experiments in explaining real-world decisions.

The ratio bias phenomenon: fact or artifact?

The ratio bias—according to which individuals prefer to bet on probabilities expressed as a ratio of large numbers to normatively equivalent or superior probabilities expressed as a ratio of small numbers—has recently gained momentum, with researchers especially in health economics emphasizing the policy importance of the phenomenon. Although the bias has been replicated several times, some doubts remain about its economic significance. Our two experiments show that the bias disappears once order effects are excluded, and once salient and dominant incentives are provided. This holds true for both choice and valuation tasks. Also, adding context to the decision problem does not alter this finding. No ratio bias could be found in between-subject tests either, which leads us to the conclusion that the policy relevance of the phenomenon is doubtful at best.

Decision Support Systems And Promoting Socially Just Environmental Management

The Mauri Model DMF is unique in its approach to the management of water resources as the framework offers a transparent and inclusive approach to considering the environmental, economic, social and cultural aspects of the decisions being contemplated. The Mauri Model DMF is unique because it is capable of including multiple-worldviews and adopts mauri (intrinsic value or well-being) in the place of the more common monetised assessments of pseudo sustainability using Cost Benefit Analysis. The Mauri Model DMF uses a two stage process that first identifies participants’ worldviews and inherent bias regarding water resource management, and then facilitates transparent assessment of selected sustainability performance indicators. The assessment can then be contemplated as the separate environmental, economic, social and cultural dimensions of the decision, and collectively as an overall result; or the priorities associated with different worldviews can be applied to determine the sensitivity of the result to different cultural contexts or worldviews.

Overconfidence and confirmation bias: are future managers vulnerable?

Decision makers often use ‘rules of thumb’, or heuristics, to help them handling decision situations (Kahneman and Tversky, 1979b). Those cognitive shortcuts are taken by the brain to cope with complexity and time limitation of decisions, by reducing the burden of information processing (Hodgkinson et al, 1999; Newell and Simon, 1972). Although crucial for decision-making, heuristics come at the cost of occasionally sending us off course, that is, make us fall into judgment traps (Tversky and Kahneman, 1974). Over fifty years of psychological research has shown that heuristics can lead to systematic errors, or biases, in decision-making. This study focuses on two particularly impactful biases to decision-making – the overconfidence and confirmation biases. A specific group – top management school students and recent graduates - were subject to classic experiments to measure their level of susceptibility to those biases. This population is bound to take decision positions at companies, and eventually make decisions that will impact not only their companies but society at large. The results show that this population is strongly biased by overconfidence, but less so to the confirmation bias. No significant relationship between the level of susceptibility to the overconfidence and to the confirmation bias was found.

Automatic design of decision-tree induction algorithms tailored to flexible-receptor docking data

Background: This paper addresses the prediction of the free energy of binding of a drug candidate with enzyme InhA associated with Mycobacterium tuberculosis. This problem is found within rational drug design, where interactions between drug candidates and target proteins are verified through molecular docking simulations. In this application, it is important not only to correctly predict the free energy of binding, but also to provide a comprehensible model that could be validated by a domain specialist. Decision-tree induction algorithms have been successfully used in drug-design related applications, specially considering that decision trees are simple to understand, interpret, and validate. There are several decision-tree induction algorithms available for general-use, but each one has a bias that makes it more suitable for a particular data distribution. In this article, we propose and investigate the automatic design of decision-tree induction algorithms tailored to particular drug-enzyme binding data sets. We investigate the performance of our new method for evaluating binding conformations of different drug candidates to InhA, and we analyze our findings with respect to decision tree accuracy, comprehensibility, and biological relevance. Results: The empirical analysis indicates that our method is capable of automatically generating decision-tree induction algorithms that significantly outperform the traditional C4.5 algorithm with respect to both accuracy and comprehensibility. In addition, we provide the biological interpretation of the rules generated by our approach, reinforcing the importance of comprehensible predictive models in this particular bioinformatics application. Conclusions: We conclude that automatically designing a decision-tree algorithm tailored to molecular docking data is a promising alternative for the prediction of the free energy from the binding of a drug candidate with a flexible-receptor.

A partial dependence factorial analysis to deal with selection bias in observational studis

This thesis presents a creative and practical approach to dealing with the problem of selection bias. Selection bias may be the most important vexing problem in program evaluation or in any line of research that attempts to assert causality. Some of the greatest minds in economics and statistics have scrutinized the problem of selection bias, with the resulting approaches – Rubin’s Potential Outcome Approach(Rosenbaum and Rubin,1983; Rubin, 1991,2001,2004) or Heckman’s Selection model (Heckman, 1979) – being widely accepted and used as the best fixes. These solutions to the bias that arises in particular from self selection are imperfect, and many researchers, when feasible, reserve their strongest causal inference for data from experimental rather than observational studies. The innovative aspect of this thesis is to propose a data transformation that allows measuring and testing in an automatic and multivariate way the presence of selection bias. The approach involves the construction of a multi-dimensional conditional space of the X matrix in which the bias associated with the treatment assignment has been eliminated. Specifically, we propose the use of a partial dependence analysis of the X-space as a tool for investigating the dependence relationship between a set of observable pre-treatment categorical covariates X and a treatment indicator variable T, in order to obtain a measure of bias according to their dependence structure. The measure of selection bias is then expressed in terms of inertia due to the dependence between X and T that has been eliminated. Given the measure of selection bias, we propose a multivariate test of imbalance in order to check if the detected bias is significant, by using the asymptotical distribution of inertia due to T (Estadella et al. 2005) , and by preserving the multivariate nature of data. Further, we propose the use of a clustering procedure as a tool to find groups of comparable units on which estimate local causal effects, and the use of the multivariate test of imbalance as a stopping rule in choosing the best cluster solution set. The method is non parametric, it does not call for modeling the data, based on some underlying theory or assumption about the selection process, but instead it calls for using the existing variability within the data and letting the data to speak. The idea of proposing this multivariate approach to measure selection bias and test balance comes from the consideration that in applied research all aspects of multivariate balance, not represented in the univariate variable- by-variable summaries, are ignored. The first part contains an introduction to evaluation methods as part of public and private decision process and a review of the literature of evaluation methods. The attention is focused on Rubin Potential Outcome Approach, matching methods, and briefly on Heckman’s Selection Model. The second part focuses on some resulting limitations of conventional methods, with particular attention to the problem of how testing in the correct way balancing. The third part contains the original contribution proposed , a simulation study that allows to check the performance of the method for a given dependence setting and an application to a real data set. Finally, we discuss, conclude and explain our future perspectives.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias

BACKGROUND: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. METHODOLOGY/PRINCIPAL FINDINGS: We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. CONCLUSIONS: Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.