Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials

OBJECTIVES To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. METHODS Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). RESULTS In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ  =  0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ  = 0.19), ranging from κ  =  -0.04 ("Other bias") to κ  =  0.62 ("Sequence generation"). There was also no agreement (κ  =  -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. CONCLUSIONS Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.

Intergroup bias in third-party punishment stems from both ingroup favoritism and outgroup discrimination

Social norms pervade almost every aspect of social interaction. If they are violated, not only legal institutions, but other members of society as well, punish, i.e., inflict costs on the wrongdoer. Sanctioning occurs even when the punishers themselves were not harmed directly and even when it is costly for them. There is evidence for intergroup bias in this third-party punishment: third-parties, who share group membership with victims, punish outgroup perpetrators more harshly than ingroup perpetrators. However, it is unknown whether a discriminatory treatment of outgroup perpetrators (outgroup discrimination) or a preferential treatment of ingroup perpetrators (ingroup favoritism) drives this bias. To answer this question, the punishment of outgroup and ingroup perpetrators must be compared to a baseline, i.e., unaffiliated perpetrators. By applying a costly punishment game, we found stronger punishment of outgroup versus unaffiliated perpetrators and weaker punishment of ingroup versus unaffiliated perpetrators. This demonstrates that both ingroup favoritism and outgroup discrimination drive intergroup bias in third-party punishment of perpetrators that belong to distinct social groups.

Negative bias of processing ambiguously cued emotional stimuli.

Daily we cope with upcoming potentially disadvantageous events. Therefore, it makes sense to be prepared for the worst case. Such a 'pessimistic' bias is reflected in brain activation during emotion processing. Healthy individuals underwent functional neuroimaging while viewing emotional stimuli that were earlier cued ambiguously or unambiguously concerning their emotional valence. Presentation of ambiguously announced pleasant pictures compared with unambiguously announced pleasant pictures resulted in increased activity in the ventrolateral prefrontal, premotor and temporal cortex, and in the caudate nucleus. This was not the case for the respective negative conditions. This indicates that pleasant stimuli after ambiguous cueing provided 'unexpected' emotional input, resulting in the adaptation of brain activity. It strengthens the hypothesis of a 'pessimistic' bias of brain activation toward ambiguous emotional events.

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Can eye movements bias memory recall?

A large body of research suggests that when we retrieve visual information from memory, we look back to the location where we encoded these objects. It has been proposed that the oculomotor trace we act out during encoding is stored in long-term memory, along other contents of the episodic representation. If memory recall triggers the eyes to revisit the location where the stimulus was encoded, is there also an effect in the reverse direction? Can eye movements trigger memory recall? In Experiment 1 participants encoded two faces at two different locations on the computer screen. Then, the average face (morph) of these two faces appeared in either of the two encoding locations and participants had to indicate whether it resembles more the first or second face. In Experiment 2 the morph appeared in a new location, but participants had to repeat one of the oculomotor traces that was used during encoding. Participants’ morph perception was influenced both by the location and the eye-movement it was presented with. Our results suggest that eye-movements can bias memory recall, but only in a short-lasting and rather fragile way.

School-age children show a bias toward fantasy classifications after playing a platform game

We investigated the influence of playing a video game on children’s ability to distinguish between fantasy and reality. School-age children played a platform game for 15 min and then performed a fantasy/reality distinction task in which they were to judge whether images (from the platform game and from other games) were fantasy images or reality images. Unlike those in the control group (who played a memory game), the children in the experimental group showed a response bias toward mistakenly classifying reality images from the video game as fantasy images (as determined by means of an analysis based on signal detection theory). We conclude that playing the video game exerted a short-term influence on children’s ability to distinguish between fantasy and reality.

8p23 beta-defensin copy number determination by single-locus pseudogene-based paralog ratio tests risk bias due to low-frequency sequence variations.

BACKGROUND The copy number variation (CNV) in beta-defensin genes (DEFB) on human chromosome 8p23 has been proposed to contribute to the phenotypic differences in inflammatory diseases. However, determination of exact DEFB CN is a major challenge in association studies. Quantitative real-time PCR (qPCR), paralog ratio tests (PRT) and multiplex ligation-dependent probe amplification (MLPA) have been extensively used to determine DEFB CN in different laboratories, but inter-method inconsistencies were observed frequently. In this study we asked which one is superior among the three methods for DEFB CN determination. RESULTS We developed a clustering approach for MLPA and PRT to statistically correlate data from a single experiment. Then we compared qPCR, a newly designed PRT and MLPA for DEFB CN determination in 285 DNA samples. We found MLPA had the best convergence and clustering results of the raw data and the highest call rate. In addition, the concordance rates between MLPA or PRT and qPCR (32.12% and 37.99%, respectively) were unacceptably low with underestimated CN by qPCR. Concordance rate between MLPA and PRT (90.52%) was high but PRT systematically underestimated CN by one in a subset of samples. In these samples a sequence variant which caused complete PCR dropout of the respective DEFB cluster copies was found in one primer binding site of one of the targeted paralogous pseudogenes. CONCLUSION MLPA is superior to PRT and even more to qPCR for DEFB CN determination. Although the applied PRT provides in most cases reliable results, such a test is particularly sensitive to low-frequency sequence variations preferably accumulating in loci like pseudogenes which are most likely not under selective pressure. In the light of the superior performance of multiplex assays, the drawbacks of such single PRTs could be overcome by combining more test markers.

How narcissistic self-construction sabotages their relationships: Accuracy and bias in partner perception

Trapped by their superordinate goal to maintain a grandiose self, narcissists navigate their relationships wishing to re-assure themselves of their specialness and superiority, using strategies bound to fail. Dyadic analyses of diary data (14 days, 83 couples) showed that narcissists focus on their partners’ shortcomings, becoming experts at detecting negative forms of support, while simultaneously minimizing their partner’s positive intentions. Partners of narcissists, in contrast, appear to have a positive bias in ongoing relationships and seem to compensate for some of the narcissists’ shortcomings. Narcissists’ focus on their own selfconstruction purposes also extends to their parenting roles. In a retrospective study, narcissistic mothers were found to demand superior performance, desiring to show-off their child, yet concurrently attributing their child’s success to themselves. They also exercise psychological control and role-reversal, requiring the child to cater to their needs, and to admire them, while simultaneously oblivious to their child’s needs and desires.

Brain systems underlying encounter expectancy bias in spider phobia.

Spider-phobic individuals are characterized by exaggerated expectancies to be faced with spiders (so-called encounter expectancy bias). Whereas phobic responses have been linked to brain systems mediating fear, little is known about how the recruitment of these systems relates to exaggerated expectancies of threat. We used fMRI to examine spider-phobic and control participants while they imagined visiting different locations in a forest after having received background information about the likelihood of encountering different animals (spiders, snakes, and birds) at these locations. Critically, imagined encounter expectancies modulated brain responses differently in phobics as compared with controls. Phobics displayed stronger negative modulation of activity in the lateral prefrontal cortex, precuneus, and visual cortex by encounter expectancies for spiders, relative to snakes or birds (within-participants analysis); these effects were not seen in controls. Between-participants correlation analyses within the phobic group further corroborated the hypothesis that these phobia-specific modulations may underlie irrationality in encounter expectancies (deviations of encounter expectancies from objective background information) in spider phobia; the greater the negative modulation a phobic participant displayed in the lateral prefrontal cortex, precuneus, and visual cortex, the stronger was her bias in encounter expectancies for spiders. Interestingly, irrationality in expectancies reflected in frontal areas relied on right rather than left hemispheric deactivations. Our data accord with the idea that expectancy biases in spider phobia may reflect deficiencies in cognitive control and contextual integration that are mediated by right frontal and parietal areas.

Varying expectancies and attention bias in phobic and non-phobic individuals.

Phobic individuals display an attention bias to phobia-related information and biased expectancies regarding the likelihood of being faced with such stimuli. Notably, although attention and expectancy biases are core features in phobia and anxiety disorders, these biases have mostly been investigated separately and their causal impact has not been examined. We hypothesized that these biases might be causally related. Spider phobic and low spider fearful control participants performed a visual search task in which they specified whether the deviant animal in a search array was a spider or a bird. Shorter reaction times (RTs) for spiders than for birds in this task reflect an attention bias toward spiders. Participants' expectancies regarding the likelihood of these animals being the deviant in the search array were manipulated by presenting verbal cues. Phobics were characterized by a pronounced and persistent attention bias toward spiders; controls displayed slower RTs for birds than for spiders only when spider cues had been presented. More important, we found RTs for spider detections to be virtually unaffected by the expectancy cues in both groups, whereas RTs for bird detections showed a clear influence of the cues. Our results speak to the possibility that evolution has formed attentional systems that are specific to the detection of phylogenetically salient stimuli such as threatening animals; these systems may not be as penetrable to variations in (experimentally induced) expectancies as those systems that are used for the detection of non-threatening stimuli. In sum, our findings highlight the relation between expectancies and attention engagement in general. However, expectancies may play a greater role in attention engagement in safe environments than in threatening environments.

Evidence for an encounter expectancy bias in fear of spiders.

Whereas research has demonstrated that phobic or fearful individuals overestimate the likelihood of incurring aversive consequences from an encounter with feared stimuli, it has not yet been systematically investigated whether these individuals also overestimate the likelihood (i.e., the frequency) of such encounters. In the current study, spider-fearful and control participants were presented with background information that allowed them to estimate the overall likelihood that different kinds of animals (spiders, snakes, or birds) would be encountered. Spider-fearful participants systematically overestimated the likelihood of encountering a spider with respect to the likelihood of encountering a snake or a bird. No such expectancy bias was observed in control participants. The results thus strengthen our idea that there indeed exist two different types of expectancy bias in high fear and phobia that can be related to different components of the fear response. A conscientious distinction and examination of these two types of expectancy bias are of potential interest for therapeutic applications.

Controlling for culture-specific response bias using ipsatization and response style indicators: Family orientation in seventeen cultures and two generations

Within-subject standardization (ipsatization) has been advocated as a possible means to control for culture-specific responding (e.g., Fisher, 2004). However, the consequences of different kinds of ipsatization procedures for the interpretation of mean differences remain unclear. The current study compared several ipsatization procedures with ANCOVA-style procedures using response style indicators for the construct of family orientation with data from 14 cultures and two generations from the Value-of-Children-(VOC)-Study (4135 dyads). Results showed that within-subject centering/standardizing across all Likert-scale items of the comprehensive VOC-questionnaire removed most of the original cross-cultural variation in family orientation and lead to a non-interpretable pattern of means in both generations. Within-subject centering/standardizing using a subset of 19 unrelated items lead to a decrease to about half of the original effect size and produced a theoretically meaningful pattern of means. A similar effect size and similar mean differences were obtained when using a measure of acquiescent responding based on the same set of items in an ANCOVA-style analysis. Additional models controlling for extremity and modesty performed worse, and combinations did not differ from the acquiescence-only model. The usefulness of different approaches to control for uniform response styles (scalar equivalence not given) in cross- cultural comparisons is discussed.