Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Co-activation of JAK/STAT and p38 MAPK pathways by type I interferon enhances apoptosis in human basophils

It has previously been published that interferon-α (type I IFN) improves clinical symptoms of asthma patients. Since human basophils are major inflammatory cells in maintaining chronic allergic asthma we investigate whether type I IFN affect human blood basophils. Furthermore, previous studies have shown that spontaneous apoptosis of human basophils is slow due to constitutive expression of anti-apoptotic BCL-2 family members. In addition, IL-3 exceptionally promotes survival of basophils by enhancing constitutive expression of BCL-2 family members and by inducing de-novo expression of Pim-1 kinase. Thus, we also assessed whether type I IFN might overcome IL-3-induced survival of human basophils. Our data show that type I IFN enhances apoptosis in purified human blood basophils compared to spontaneous apoptosis of controls or type II IFN treated cells. Furthermore, we demonstrate that both type I IFN and FasL enhance apoptosis in human basophils with similar efficiency in a rather additive than synergistic way. Analyses of signaling pathways reveal that type I IFN promote prolonged phosphorylation of STAT1/STAT2. By using a pan-JAK inhibitor the phosphorylation of STAT1/STAT2 is inhibited and most importantly the pro-apoptotic effect of type I IFN is abolished. On the other hand, type I IFN do not reduce IL-3-induced de novo expression of Pim-1 and BCL-2. This is in line with our observation that IL-3-induced survival is dominant over type I IFN-enhanced apoptosis. In addition, phosphorylation of p38 MAPK in type I IFN treated cells is comparable to non-treated cells. Particularly however, inhibition of this p-p38 activity abrogates apoptosis as well. We conclude that type I IFN-enhanced apoptosis is tightly regulated by the cooperation of JAK/STAT and p38 MAPK pathways. Our study identifies a so far unknown effect of type I IFN and may explain the improved clinical symptoms of asthma patients treated with type I IFN.