The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

Objectives

A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes.

Study Design and Setting

A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result.

Results

The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up.

Conclusion

The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results. 

Ensuring the statistical soundness of competitive gene set approaches: gene filtering and genome-scale coverage are essential

In this article, we focus on the analysis of competitive gene set methods for detecting the statistical significance of pathways from gene expression data. Our main result is to demonstrate that some of the most frequently used gene set methods, GSEA, GSEArot and GAGE, are severely influenced by the filtering of the data in a way that such an analysis is no longer reconcilable with the principles of statistical inference, rendering the obtained results in the worst case inexpressive. A possible consequence of this is that these methods can increase their power by the addition of unrelated data and noise. Our results are obtained within a bootstrapping framework that allows a rigorous assessment of the robustness of results and enables power estimates. Our results indicate that when using competitive gene set methods, it is imperative to apply a stringent gene filtering criterion. However, even when genes are filtered appropriately, for gene expression data from chips that do not provide a genome-scale coverage of the expression values of all mRNAs, this is not enough for GSEA, GSEArot and GAGE to ensure the statistical soundness of the applied procedure. For this reason, for biomedical and clinical studies, we strongly advice not to use GSEA, GSEArot and GAGE for such data sets.

A Multicenter, Randomised Open Study of Early Corticosteroid Treatment (OSECT) in Preterm Infants With Respiratory Illness: Comparison of Early and Late Treatment and of Dexamethasone and Inhaled Budesonide

AIM: To compare early (15 days) steroid therapy and dexamethasone with inhaled budesonide in very preterm infants at risk of developing chronic lung disease. METHODS: Five hundred seventy infants from 47 neonatal intensive care units were enrolled. Criteria for enrollment included gestational age 30%. Infants were randomly allocated to 1 of 4 treatment groups in a factorial design: early (15 days) dexamethasone, and delayed selective budesonide. Dexamethasone was given in a tapering course beginning with 0.50 mg/kg/day in 2 divided doses for 3 days reducing by half until 12 days of therapy had elapsed. Budesonide was administered by metered dose inhaler and a spacing chamber in a dose of 400 microg/kg twice daily for 12 days. Delayed selective treatment was started if infants needed mechanical ventilation and >30% oxygen for >15 days. The factorial design allowed 2 major comparisons: early versus late treatment and systemic dexamethasone versus inhaled budesonide. The primary outcome was death or oxygen dependency at 36 weeks and analysis was on an intention-to-treat basis. Secondary outcome measures included death or major cerebral abnormality, duration of oxygen treatment, and complications of prematurity. Adverse effects were also monitored daily. RESULTS: There were no significant differences among the groups for the primary outcome. Early steroid treatment was associated with a lower primary outcome rate (odds ratio [OR]: 0.85; 95% confidence interval [CI]: 0.61,1.18) but even after adjustment for confounding variables the difference remained nonsignificant. Dexamethasone-treated infants also had a lower primary outcome rate (OR: 0.86; 95% CI: 0.62,1.20) but again this difference remained not significant after adjustment. For death before discharge, dexamethasone and early treatment had worse outcomes than budesonide and delayed selective treatment (OR: 1.42; 95% CI: 0.93,2.16; OR: 1.51; 95% CI: 0.99,2.30 after adjustment, respectively) with the results not quite reaching significance. Duration of supplementary oxygen was shorter in the early dexamethasone group (median: 31 days vs 40-44 days). Early dexamethasone was also associated with increased weight loss during the first 12 days of treatment (52 g vs 3 g) compared with early budesonide, but over 30 days there was no difference. In the early dexamethasone group, there was a reduced incidence of persistent ductus arteriosus (34% vs 52%-59%) and an increased risk of hyperglycemia (55% vs 29%-34%) compared with the other 3 groups. Dexamethasone was associated with an increased risk of hypertension and gastrointestinal problems compared with budesonide but only the former attained significance. CONCLUSIONS: Infants given early treatment and dexamethasone therapy had improved survival without chronic lung disease at 36 weeks compared with those given delayed selective treatment and inhaled budesonide, respectively, but results for survival to discharge were in the opposite direction; however, none of these findings attained statistical significance. Early dexamethasone treatment reduced the risk of persistent ductus arteriosus. Inhaled budesonide may be safer than dexamethasone, but there is no clear evidence that it is more or less effective

Malignancy and mortality in a population-based cohort of patients with coeliac disease or 'gluten sensitivity'

Aim: To determine the risk of malignancy and mortality in patients with a positive endomysial or anti-gliadin antibody test in Northern Ireland.

Methods: A population-based retrospective cohort study design was used. Laboratory test results used in the diagnosis of coeliac disease were obtained from the Regional Immunology Laboratory, cancer statistics from the Northern Ireland Cancer Registry and mortality statistics from the General Registrar Office, Northern Ireland. Age standardized incidence ratios of malignant neoplasms and standardized mortality ratios of all-cause and cause-specific mortality were calculated.

Results: A total of 13 338 people had an endomysial antibody and/or an anti-gliadin antibody test in Northern Ireland between 1993 and 1996. There were 490 patients who tested positive for endomysial antibodies and they were assumed to have coeliac disease. There were 1133 patients who tested positive for anti-gliadin anti-bodies and they were defined as gluten sensitive. Malignant neoplasms were not significantly associated with coeliac disease; however, all-cause mortality was significantly increased following diagnosis. The standardized incidence and mortality ratios for non-Hodgkin's lymphoma were increased in coeliac disease patients but did not reach statistical significance. Lung and breast cancer incidence were significantly lower and all-cause mortality, mortality from malignant neoplasms, non-Hodgkin's lymphoma and digestive system disorders were significantly higher in gluten sensitive patients compared to the Northern Ireland population.

Conclusion: Patients with coeliac disease or gluten sensitivity had higher mortality rates than the Northern Ireland population. This association persists more than one year after diagnosis in patients testing positive for anti-gliadin antibodies. Breast cancer is significantly reduced in the cohort of patients with gluten sensitivity. © 2007 The WJG Press. All rights reserved.

A meta-analysis of the association between Caesarean section and childhood asthma

Background: Children born by Caesarean section have modified intestinal bacterial colonization and consequently may have an increased risk of developing asthma under the hygiene hypothesis. The results of previous studies that have investigated the association between Caesarean section and asthma have been conflicting.

Objective: To review published literature and perform a meta-analysis summarizing the evidence in support of an association between children born by Caesarean section and asthma.

Methods: MEDLINE, Web Science, Google Scholar and PubMed were searched to identify relevant studies. Odds ratio (OR) and 95% confidence interval (CI) were calculated for each study from the reported prevalence of asthma in children born by Caesarean section and in control children. Meta-analysis was then used to derive a combined OR and test for heterogeneity in the findings between studies.

Results: Twenty-three studies were identified. The overall meta-analysis revealed an increase in the risk of asthma in children delivered by Caesarean section (OR=1.22, 95% CI 1.14, 1.29). However, in this analysis, there was evidence of heterogeneity (I2=46%) that was statistically significant (P<0.001). Restricting the analysis to childhood studies, this heterogeneity was markedly decreased (I2=32%) and no longer attained statistical significance (P=0.08). In these studies, there was also evidence of an increase (P<0.001) in the risk of asthma after Caesarean section (OR=1.20, 95% CI 1.14, 12.6).

Conclusion: In this meta-analysis, we found a 20% increase in the subsequent risk of asthma in children who had been delivered by Caesarean section.

A simple and robust method for connecting small-molecule drugs using gene-expression signatures

Background

Interaction of a drug or chemical with a biological system can result in a gene-expression profile or signature characteristic of the event. Using a suitably robust algorithm these signatures can potentially be used to connect molecules with similar pharmacological or toxicological properties by gene expression profile. Lamb et al first proposed the Connectivity Map [Lamb et al (2006), Science 313, 1929–1935] to make successful connections among small molecules, genes, and diseases using genomic signatures.

Results

Here we have built on the principles of the Connectivity Map to present a simpler and more robust method for the construction of reference gene-expression profiles and for the connection scoring scheme, which importantly allows the valuation of statistical significance of all the connections observed. We tested the new method with two randomly generated gene signatures and three experimentally derived gene signatures (for HDAC inhibitors, estrogens, and immunosuppressive drugs, respectively). Our testing with this method indicates that it achieves a higher level of specificity and sensitivity and so advances the original method.

Conclusion

The method presented here not only offers more principled statistical procedures for testing connections, but more importantly it provides effective safeguard against false connections at the same time achieving increased sensitivity. With its robust performance, the method has potential use in the drug development pipeline for the early recognition of pharmacological and toxicological properties in chemicals and new drug candidates, and also more broadly in other 'omics sciences.

Immune activation in Retinal Aging: A Gene Expression Study.

PURPOSE
To investigate changes in gene expression during aging of the retina in the mouse.

METHODS
Total RNA was extracted from the neuroretina of young (3-month-old) and old (20-month-old) mice and processed for microarray analysis. Age-related, differentially expressed genes were assessed by the empiric Bayes shrinkagemoderated t-statistics method. Statistical significance was based on dual criteria of a ratio of change in gene expression >2 and a P < 0.01. Differential expression in 11 selected genes was further verified by real-time PCR. Functional pathways involved in retinal ageing were analyzed by an online software package (DAVID-2008) in differentially expressed gene lists. Age-related changes in differential expression in the identified retinal molecular pathways were further confirmed by immunohistochemical staining of retinal flat mounts and retinal cryosections.

RESULTS
With ageing of the retina, 298 genes were upregulated and 137 genes were downregulated. Functional annotation showed that genes linked to immune responses (Ir genes) and to tissue stress/injury responses (TS/I genes) were most likely to be modified by ageing. The Ir genes affected included those regulating leukocyte activation, chemotaxis, endocytosis, complement activation, phagocytosis, and myeloid cell differentiation, most of which were upregulated, with only a few downregulated. Increased microglial and complement activation in the aging retina was further confirmed by confocal microscopy of retinal tissues. The most strongly upregulated gene was the calcitonin receptor (Calcr; >40-fold in old versus young mice).

CONCLUSIONS
The results suggest that retinal ageing is accompanied by activation of gene sets, which are involved in local inflammatory responses. A modified form of low-grade chronic inflammation (para-inflammation) characterizes these aging changes and involves mainly the innate immune system. The marked upregulation of Calcr in ageing mice most likely reflects this chronic inflammatory/stress response, since calcitonin is a known systemic biomarker of inflammation/sepsis. © Association for Research in Vision and Ophthalmology.

A randomized clinical trial of hydroxymethylglutaryl-Coenzyme A reductase inhibition for Acute Lung Injury (The HARP study)

Rationale: There is no effective pharmacological treatment for acute lung injury (ALI). Statins are a potential new therapy because they modify many of the underlying processes important in ALI.

Objectives: To test whether simvastatin improves physiological and biological outcomes in ALI.

Methods: We conducted a randomized, double-blinded, placebo-controlled trial in patients with ALI. Patients received 80 mg simvastatin or placebo until cessation of mechanical ventilation or up to 14 days. Extravascular lung water was measured using thermodilution. Measures of pulmonary and nonpulmonary organ function were assessed daily. Pulmonary and systemic inflammation was assessed by bronchoalveolar lavage fluid and plasma cytokines. Systemic inflammation was also measured by plasma C-reactive protein.

Measurements and Main Results: Sixty patients were recruited. Baseline characteristics, including demographics and severity of illness scores, were similar in both groups. At Day 7, there was no difference in extravascular lung water. By Day 14, the simvastatin-treated group had improvements in nonpulmonary organ dysfunction. Oxygenation and respiratory mechanics improved, although these parameters failed to reach statistical significance. Intensive care unit mortality was 30% in both groups. Simvastatin was well tolerated, with no increase in adverse events. Simvastatin decreased bronchoalveolar lavage IL-8 by 2.5-fold (P = 0.04). Plasma C-reactive protein decreased in both groups but failed to achieve significance in the placebo-treated group.

Conclusions: Treatment with simvastatin appears to be safe and may be associated with an improvement in organ dysfunction in ALI. These clinical effects may be mediated by a reduction in pulmonary and systemic inflammation.

Maternal Glucose at 28 Weeks of Gestation Is Not Associated With Obesity in 2-Year-Old Offspring:The Belfast Hyperglycemia and Adverse Pregnancy Outcome (HAPO) Family Study

OBJECTIVE:Diabetes during pregnancy is a strong risk factor for obesity in the offspring, but the age at which this association becomes apparent is unknown. The purpose of this study was to examine the relation of glycemia during pregnancy with anthropometry in offspring of nondiabetic pregnant women from the Belfast U.K. center of the multinational Hyperglycemia and Adverse Pregnancy Outcome (HAPO) Study.
RESEARCH DESIGN AND METHODS: Women from the HAPO Study were invited to participate in follow-up of their offspring aged 2 years. Measurements included height, weight, and thickness of triceps, subscapular, and suprailiac skinfolds. RESULTS: A total of 1,165 offspring (73% of eligible children; 598 boys and 567 girls) were seen from ages 22-30 completed months. The only association that reached statistical significance was between categories of maternal 1-h glucose and BMI Z score =85th percentile at 2 years (P = 0.017). Overall the correlations between maternal glucose during pregnancy and BMI Z score at age 2 years were weak (fasting glucose r = 0.05, P = 0.08; 1-h glucose r = 0.04, P = 0.22; 2-h glucose r = 0.03, P = 0.36; and area under the curve for glucose r = 0.04, P = 0.18).
CONCLUSIONS: This study found little association between maternal glucose during pregnancy and obesity in the offspring at this young age. These findings are not unexpected given that study results for young offspring whose mothers had diabetes during pregnancy were indistinguishable from those for normal offspring at this age. It will be interesting to see whether, as these children age, maternal glucose during pregnancy in the ranges included in the HAPO Study will be associated with obesity in their children. © 2010 by the American Diabetes Association.

Differences in mouse models of diabetes mellitus in studies of male reproduction

Diabetes Mellitus (DM) has been found to have subtle yet profound effects on the metabolic status of the testis, the expression of numerous spermatogenic genes and is associated with increased numbers of sperm with nuclear DNA damage. The precise mechanism causing these detrimental effects remains unknown. The presence of increased levels of the most prominent member (carboxymethyllysine - CML) of the advanced glycation end product adducts and their receptor (RAGE) in the reproductive tract of DM men has provided a new avenue for research. As there are suspicions that the antibiotic (streptozotocin - STZ) employed to induce DM is also capable of causing oxidative stress and DNA damage, we compared CML and RAGE levels in the reproductive tract and sperm nDNA status of STZ mice with the levels in the Ins(2Akita) mouse to determine which more closely mimics the situation described in the human diabetic. CML was observed in the testes, epididymes and sperm of all animals. Sperm from DM mice showed particularly strong CML immunolocalization in the acrosomal cap, the equatorial region and whenever present, cytoplasmic droplets. Although increased, the level of CML on the sperm of the STZ and Ins(2Akita) DM mice did not reach statistical significance. RAGE was present on the developing acrosome and epididymal sperm of all animals and in discrete regions of the epididymes of the DM models. Only the epididymal sperm of the Ins(2Akita) mice were found to have significantly increased (p < 0.0001) nDNA damage. The Ins(2Akita) mouse therefore appears to more accurately reflect the conditions found in the human and, as such, is a more representative model for the study of diabetes and glycation's influence on male fertility.

Replication of association between schizophrenia and ZNF804A in the Irish Case-Control Study of Schizophrenia sample

A recent genome-wide association study reported association between schizophrenia and the ZNF804A gene on chromosome 2q32.1. We attempted to replicate these findings in our Irish Case-Control Study of Schizophrenia (ICCSS) sample (N=1021 cases, 626 controls). Following consultation with the original investigators, we genotyped three of the most promising single-nucleotide polymorphisms (SNPs) from the Cardiff study. We replicate association with rs1344706 (trend test one-tailed P=0.0113 with the previously associated A allele) in ZNF804A. We detect no evidence of association with rs6490121 in NOS1 (one-tailed P=0.21), and only a trend with rs9922369 in RGRIP1L (one-tailed P=0.0515). On the basis of these results, we completed genotyping of 11 additional linkage disequilibrium-tagging SNPs in ZNF804A. Of 12 SNPs genotyped, 11 pass quality control criteria and 4 are nominally associated, with our most significant evidence of association at rs7597593 (P=0.0013) followed by rs1344706. We observe no evidence of differential association in ZNF804A on the basis of family history or sex of case. The associated SNP rs1344706 lies in approximately 30 bp of conserved mammalian sequence, and the associated A allele is predicted to maintain binding sites for the brain-expressed transcription factors MYT1l and POU3F1/OCT-6. In controls, expression is significantly increased from the A allele of rs1344706 compared with the C allele. Expression is increased in schizophrenic cases compared with controls, but this difference does not achieve statistical significance. This study replicates the original reported association of ZNF804A with schizophrenia and suggests that there is a consistent link between the A allele of rs1344706, increased expression of ZNF804A and risk for schizophrenia.

Progressive loss of epidermal growth factor receptor in a subpopulation of breast cancers: implications in target-directed therapeutics

Understanding the molecular etiology and heterogeneity of disease has a direct effect on cancer therapeutics. To identify novel molecular changes associated with breast cancer progression, we conducted phosphoproteomics of the MCF10AT model comprising isogenic, ErbB2- and ErbB3-positive, xenograft-derived cell lines that mimic different stages of breast cancer. Using in vitro animal model and clinical breast samples, our study revealed a marked reduction of epidermal growth factor receptor (EGFR) expression with breast cancer progression. Such diminution of EGFR expression was associated with increased resistance to Gefitinib/Iressa in vitro. Fluorescence in situ hybridization showed that loss of EGFR gene copy number was one of the key mechanisms behind the low/null expression of EGFR in clinical breast tumors. Statistical analysis on the immunohistochemistry data of EGFR expression from 93 matched normal and breast tumor samples showed that (a) diminished EGFR expression could. be detected as early as in the preneoplastic lesion (ductal carcinoma in situ) and this culminated in invasive carcinomas; (b) EGFR expression levels could distinguish between normal tissue versus carcinoma in situ and invasive carcinoma with high statistical significance (P

Implications for Powering Biomarker Discovery Studies

This study examined variations in gene expression between FFPE blocks within tumors of individual patients. Microarray data were used to measure tumor heterogeneity within and between patients and disease states. Data were used to determine the number of samples needed to power biomarker discovery studies. Bias and variation in gene expression were assessed at the intrapatient and interpatient levels and between adenocarcinoma and squamous samples. A mixed-model analysis of variance was fitted to gene expression data and model signatures to assess the statistical significance of observed variations within and between samples and disease states. Sample size analysis, adjusted for sample heterogeneity, was used to determine the number of samples required to support biomarker discovery studies. Variation in gene expression was observed between blocks taken from a single patient. However, this variation was considerably less than differences between histological characteristics. This degree of block-to-block variation still permits biomarker discovery using either macrodissected tumors or whole FFPE sections, provided that intratumor heterogeneity is taken into account. Failure to consider intratumor heterogeneity may result in underpowered biomarker studies that may result in either the generation of longer gene signatures or the inability to identify a viable biomarker. Moreover, the results of this study indicate that a single biopsy sample is suitable for applying a biomarker in nonsmall-cell lung cancer. © 2012 American Society for Investigative Pathology and the Association for Molecular Pathology.

Association Testing of Previously Reported Variants in a Large Case-Control Meta-analysis of Diabetic Nephropathy

We formed the GEnetics of Nephropathy–an International Effort (GENIE) consortium to examine previously reported genetic associations with diabetic nephropathy (DN) in type 1 diabetes. GENIE consists of 6,366 similarly ascertained participants of European ancestry with type 1 diabetes, with and without DN, from the All Ireland-Warren 3-Genetics of Kidneys in Diabetes U.K. and Republic of Ireland (U.K.-R.O.I.) collection and the Finnish Diabetic Nephropathy Study (FinnDiane), combined with reanalyzed data from the Genetics of Kidneys in Diabetes U.S. Study (U.S. GoKinD). We found little evidence for the association of the EPO promoter polymorphism, rs161740, with the combined phenotype of proliferative retinopathy and end-stage renal disease in U.K.-R.O.I. (odds ratio [OR] 1.14, P = 0.19) or FinnDiane (OR 1.06, P = 0.60). However, a fixed-effects meta-analysis that included the previously reported cohorts retained a genome-wide significant association with that phenotype (OR 1.31, P = 2 × 10-9). An expanded investigation of the ELMO1 locus and genetic regions reported to be associated with DN in the U.S. GoKinD yielded only nominal statistical significance for these loci. Finally, top candidates identified in a recent meta-analysis failed to reach genome-wide significance. In conclusion, we were unable to replicate most of the previously reported genetic associations for DN, and significance for the EPO promoter association was attenuated.