Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

OBJECTIVES To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ  =  0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ  = 0.19), ranging from κ  =  -0.04 ("Other bias") to κ  =  0.62 ("Sequence generation"). There was also no agreement (κ  =  -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.

Intergroup bias in third-party punishment stems from both ingroup favoritism and outgroup discrimination

Social norms pervade almost every aspect of social interaction. If they are violated, not only legal institutions, but other members of society as well, punish, i.e., inflict costs on the wrongdoer. Sanctioning occurs even when the punishers themselves were not harmed directly and even when it is costly for them. There is evidence for intergroup bias in this third-party punishment: third-parties, who share group membership with victims, punish outgroup perpetrators more harshly than ingroup perpetrators. However, it is unknown whether a discriminatory treatment of outgroup perpetrators (outgroup discrimination) or a preferential treatment of ingroup perpetrators (ingroup favoritism) drives this bias. To answer this question, the punishment of outgroup and ingroup perpetrators must be compared to a baseline, i.e., unaffiliated perpetrators. By applying a costly punishment game, we found stronger punishment of outgroup versus unaffiliated perpetrators and weaker punishment of ingroup versus unaffiliated perpetrators. This demonstrates that both ingroup favoritism and outgroup discrimination drive intergroup bias in third-party punishment of perpetrators that belong to distinct social groups.

Negative bias of processing ambiguously cued emotional stimuli.

Daily we cope with upcoming potentially disadvantageous events. Therefore, it makes sense to be prepared for the worst case. Such a 'pessimistic' bias is reflected in brain activation during emotion processing. Healthy individuals underwent functional neuroimaging while viewing emotional stimuli that were earlier cued ambiguously or unambiguously concerning their emotional valence. Presentation of ambiguously announced pleasant pictures compared with unambiguously announced pleasant pictures resulted in increased activity in the ventrolateral prefrontal, premotor and temporal cortex, and in the caudate nucleus. This was not the case for the respective negative conditions. This indicates that pleasant stimuli after ambiguous cueing provided 'unexpected' emotional input, resulting in the adaptation of brain activity. It strengthens the hypothesis of a 'pessimistic' bias of brain activation toward ambiguous emotional events.

Dopaminergic stimulation increases selfish behavior in the absence of punishment threat.

RATIONALE People often face decisions that pit self-interested behavior aimed at maximizing personal reward against normative behavior such as acting cooperatively, which benefits others. The threat of social sanctions for defying the fairness norm prevents people from behaving overly selfish. Thus, normative behavior is influenced by both seeking rewards and avoiding punishment. However, the neurochemical processes mediating the impact of these influences remain unknown. Several lines of evidence link the dopaminergic system to reward and punishment processing, respectively, but this evidence stems from studies in non-social contexts. OBJECTIVES The present study investigates dopaminergic drug effects on individuals' reward seeking and punishment avoidance in social interaction. METHODS Two-hundred one healthy male participants were randomly assigned to receive 300 mg of L-3,4-dihydroxyphenylalanine (L-DOPA) or a placebo before playing an economic bargaining game. This game involved two conditions, one in which unfair behavior could be punished and one in which unfair behavior could not be punished. RESULTS In the absence of punishment threats, L-DOPA administration led to more selfish behavior, likely mediated through an increase in reward seeking. In contrast, L-DOPA administration had no significant effect on behavior when faced with punishment threats. CONCLUSIONS The results of this study broaden the role of the dopaminergic system in reward seeking to human social interactions. We could show that even a single dose of a dopaminergic drug may bring selfish behavior to the fore, which in turn may shed new light on potential causal relationships between the dopaminergic system and norm abiding behaviors in certain clinical subpopulations.

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Can eye movements bias memory recall?

A large body of research suggests that when we retrieve visual information from memory, we look back to the location where we encoded these objects. It has been proposed that the oculomotor trace we act out during encoding is stored in long-term memory, along other contents of the episodic representation. If memory recall triggers the eyes to revisit the location where the stimulus was encoded, is there also an effect in the reverse direction? Can eye movements trigger memory recall? In Experiment 1 participants encoded two faces at two different locations on the computer screen. Then, the average face (morph) of these two faces appeared in either of the two encoding locations and participants had to indicate whether it resembles more the first or second face. In Experiment 2 the morph appeared in a new location, but participants had to repeat one of the oculomotor traces that was used during encoding. Participants’ morph perception was influenced both by the location and the eye-movement it was presented with. Our results suggest that eye-movements can bias memory recall, but only in a short-lasting and rather fragile way.

School-age children show a bias toward fantasy classifications after playing a platform game

We investigated the influence of playing a video game on children’s ability to distinguish between fantasy and reality. School-age children played a platform game for 15 min and then performed a fantasy/reality distinction task in which they were to judge whether images (from the platform game and from other games) were fantasy images or reality images. Unlike those in the control group (who played a memory game), the children in the experimental group showed a response bias toward mistakenly classifying reality images from the video game as fantasy images (as determined by means of an analysis based on signal detection theory). We conclude that playing the video game exerted a short-term influence on children’s ability to distinguish between fantasy and reality.

Implicit threat, self-regulatory resources, and cognitive performance

The color red has been considered to indicate threat in achievement contexts. Recent studies have used brief confrontations with red — either as the color or as the word red — to prime for implicit threat, and have found a related impairment of cognitive performance. In another line of research, it has been shown that initial self-regulatory efforts cause diminished investment of self-regulatory resources afterwards, leading to a relative shift from a controlled to an automatic mode of information processing. We assume that activation of implicit threat via the color or the word red impairs cognitive performance more strongly during automatic compared to controlled processing of information. To test this hypothesis, we manipulated undergraduates’ (n = 78) momentary processing mode (automatic vs. controlled) by an initial task that required either high or low self-regulatory effort. Afterwards, participants were briefly confronted with red or gray stimuli and were then asked to complete a standardized intelligence measure. As expected, confrontation with red, as opposed to gray, impaired intellectual performance when participants were in an automatic processing mode. In contrast, no color effect emerged when participants were in a relatively controlled processing mode. In a second study, we replicated this finding in a sample of secondary school students (n = 130), using the black-printed word red or gray to experimentally manipulate implicit threat. Among others, the present findings may help to explain occasional difficulties in replicating findings of priming research.

8p23 beta-defensin copy number determination by single-locus pseudogene-based paralog ratio tests risk bias due to low-frequency sequence variations.

BACKGROUND The copy number variation (CNV) in beta-defensin genes (DEFB) on human chromosome 8p23 has been proposed to contribute to the phenotypic differences in inflammatory diseases. However, determination of exact DEFB CN is a major challenge in association studies. Quantitative real-time PCR (qPCR), paralog ratio tests (PRT) and multiplex ligation-dependent probe amplification (MLPA) have been extensively used to determine DEFB CN in different laboratories, but inter-method inconsistencies were observed frequently. In this study we asked which one is superior among the three methods for DEFB CN determination. RESULTS We developed a clustering approach for MLPA and PRT to statistically correlate data from a single experiment. Then we compared qPCR, a newly designed PRT and MLPA for DEFB CN determination in 285 DNA samples. We found MLPA had the best convergence and clustering results of the raw data and the highest call rate. In addition, the concordance rates between MLPA or PRT and qPCR (32.12% and 37.99%, respectively) were unacceptably low with underestimated CN by qPCR. Concordance rate between MLPA and PRT (90.52%) was high but PRT systematically underestimated CN by one in a subset of samples. In these samples a sequence variant which caused complete PCR dropout of the respective DEFB cluster copies was found in one primer binding site of one of the targeted paralogous pseudogenes. CONCLUSION MLPA is superior to PRT and even more to qPCR for DEFB CN determination. Although the applied PRT provides in most cases reliable results, such a test is particularly sensitive to low-frequency sequence variations preferably accumulating in loci like pseudogenes which are most likely not under selective pressure. In the light of the superior performance of multiplex assays, the drawbacks of such single PRTs could be overcome by combining more test markers.

Threat of alveolar echinococcosis to public health - a challenge for Europe

Alveolar echinococcosis (AE) is a neglected 'malignant' parasitic disease. The European endemic area of Echinococcus multilocularis in foxes is larger than previously anticipated, and there is new evidence that both fox populations and the prevalence of E. multilocularis have increased in many areas, indicating increased pressure for infection with E. multilocularis eggs in intermediate and accidental hosts, including humans. This may result in more human AE cases within the next decades. Current numbers of both immunocompetent and immunocompromised AE patients, and the anticipated future increase, call for scaling-up research to rapidly improve the development and implementation of prevention measures, early diagnosis, and curative treatment of human AE.