The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index

Objectives

A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes.

Study Design and Setting

A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result.

Results

The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up.

Conclusion

The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results. 

An efficacy and mechanism evaluation study of Levosimendan for the Prevention of Acute oRgan Dysfunction in Sepsis (LeoPARDS): Protocol for a randomized controlled trial

Background

Organ dysfunction consequent to infection (‘severe sepsis’) is the leading cause of admission to an intensive care unit (ICU). In both animal models and early clinical studies the calcium channel sensitizer levosimendan has been demonstrated to have potentially beneficial effects on organ function. The aims of the Levosimendan for the Prevention of Acute oRgan Dysfunction in Sepsis (LeoPARDS) trial are to identify whether a 24-hour infusion of levosimendan will improve organ dysfunction in adults who have septic shock and to establish the safety profile of levosimendan in this group of patients.

This is a multicenter, randomized, double-blind, parallel group, placebo-controlled trial. Adults fulfilling the criteria for systemic inflammatory response syndrome due to infection, and requiring vasopressor therapy, will be eligible for inclusion in the trial. Within 24 hours of meeting these inclusion criteria, patients will be randomized in a 1:1 ratio stratified by the ICU to receive either levosimendan (0.05 to 0.2 μg.kg^-1.min^-1 or placebo for 24 hours in addition to standard care. The primary outcome measure is the mean Sequential Organ Failure Assessment (SOFA) score while in the ICU. Secondary outcomes include: central venous oxygen saturations and cardiac output; incidence and severity of renal failure using the Acute Kidney Injury Network criteria; duration of renal replacement therapy; serum bilirubin; time to liberation from mechanical ventilation; 28-day, hospital, 3 and 6 month survival; ICU and hospital length-of-stay; and days free from catecholamine therapy. Blood and urine samples will be collected on the day of inclusion, at 24 hours, and on days 4 and 6 post-inclusion for investigation of the mechanisms by which levosimendan might improve organ function. Eighty patients will have additional blood samples taken to measure levels of levosimendan and its active metabolites OR-1896 and OR-1855. A total of 516 patients will be recruited from approximately 25 ICUs in the United Kingdom.

This trial will test the efficacy of levosimendan to reduce acute organ dysfunction in adult patients who have septic shock and evaluate its biological mechanisms of action.

Research Protocol: A Randomized Controlled Trial of Functional Family Therapy: An Early Intervention Foundation (EIF) Partnership between Croydon Council and Queen's University Belfast

The paper presents a protocol for ‘A Randomized Controlled Trial of Functional Family Therapy (FFT): An Early Intervention Foundation (EIF) Partnership between Croydon Council and Queen's University Belfast’. The protocol describes a trial that uses FFT as an alternative intervention to current use of the youth justice system and local authority care with the aim of reducing crime/recidivism in young people referred to Croydon Council. The trial will take place over a period of 36 months and will involve up to 154 families. Croydon Council will employ a team of five Functional Family Therapists who will work with families to promote effective outcomes. The Centre for Effective Education at Queen’s University Belfast will act as independent evaluators of outcomes for families and young people. The work is supported from the United Kingdom Economic & Social Research Council/Early Intervention Foundation Grant Number ES/M006921/1.

Use of the child health utility and strengths and difficulties outcome measures in economic evaluations of school-based interventions: data from a cluster-randomised controlled trial in Northern Ireland

Background There is growing evidence linking early social and emotional wellbeing to later academic performance and various health outcomes including mental health. An economic evaluation was designed alongside the Roots of Empathy cluster-randomised trial evaluation, which is a school-based intervention for improving pupils’ social and emotional wellbeing. Exploration of the relevance of the Strengths and Diffi culties Questionnaire (SDQ) and Child Health Utility 9D (CHU9D) in school-based health economic evaluations is warranted. The SDQ is a behavioural screening questionnaire for 4–17-year-old children, consisting of a total diffi culties score, and also prosocial behaviour,
which aims to identify positive aspects of behaviour. The CHU9D is a generic preference-based health-related quality of life instrument for 7–17-year-old children.

Different frequencies of acupuncture for treatment for chronic low back pain: an-assessor blinded pilot randomised controlled trial.

OBJECTIVES:
To examine the feasibility of a main RCT to compare the effectiveness of two frequencies (2 versus 5 times/week) of acupuncture treatment for chronic low back pain (LBP).
METHODS:
Participants (n=30) with chronic LBP were randomised into two groups to receive 10 acupuncture treatments: Low Frequency Group, 2 times/week for five weeks (n=15); High Frequency Group, 5 times/week for two weeks (n=15). The following outcomes were measured blindly at baseline, 2 weeks, 5 weeks, 3 months and 1 year: pain on a VAS, functional disability using the RMDQ, quality of life using the Measure Yourself Medical Outcome Profile (MYMOP-2), psychological impact with the Coping Strategies Questionnaire (CSQ) and Pain Locus of Control (PLC) questionnaire. Two objective outcomes, the Shuttle Walk Test (SWT) and Lateral Trunk Flexibility (LTF), were also measured.
RESULTS:
The compliance rate was 100% for each group. Some of the measurements were shown to be sensitive (VAS, RMDQ, MYMOP-2 Wellbeing). 66-330 participants would be required for a fully powered non-inferiority trial. The groups were balanced at baseline for LBP and demographic characteristics. There were no significant differences between the groups in terms of any of the outcomes, at each follow-up time point. It was notable however that the clinically important improvement in terms of pain, functional disability, quality of life, and SWT in both groups was achieved within the first two weeks, which was maintained at one year follow-up.
CONCLUSIONS:
It is feasible to conduct a main RCT, to compare different frequencies of acupuncture for LBP, using sensitive measurements. Also the trend for early clinically important improvement within a minimum of four measurements is worthy of further study.

Supervised walking in comparison with fitness training for chronic back pain in physiotherapy: results of the SWIFT single-blinded randomized controlled trial (ISRCTN17592092)

Effectiveness of brief/minimal contact self-activation interventions that encourage participation in physical activity (PA) for chronic low back pain (CLBP >12 weeks) is unproven. The primary objective of this assessor-blinded randomized controlled trial was to investigate the difference between an individualized walking programme (WP), group exercise class (EC), and usual physiotherapy (UP, control) in mean change in functional disability at 6 months. A sample of 246 participants with CLBP aged 18 to 65 years (79 men and 167 women; mean age ± SD: 45.4 ± 11.4 years) were recruited from 5 outpatient physiotherapy departments in Dublin, Ireland. Consenting participants completed self-report measures of functional disability, pain, quality of life, psychosocial beliefs, and PA were randomly allocated to the WP (n = 82), EC (n = 83), or UP (n = 81) and followed up at 3 (81%; n = 200), 6 (80.1%; n = 197), and 12 months (76.4%; n = 188). Cost diaries were completed at all follow-ups. An intention-to-treat analysis using a mixed between-within repeated-measures analysis of covariance found significant improvements over time on the Oswestry Disability Index (Primary Outcome), the Numerical Rating Scale, Fear Avoidance-PA scale, and the EuroQol EQ-5D-3L Weighted Health Index (P < 0.05), but no significant between-group differences and small between-group effect sizes (WP: mean difference at 6 months, 6.89 Oswestry Disability Index points, 95% confidence interval [CI] -3.64 to -10.15; EC: -5.91, CI: -2.68 to -9.15; UP: -5.09, CI: -1.93 to -8.24). The WP had the lowest mean costs and the highest level of adherence. Supervised walking provides an effective alternative to current forms of CLBP management.

Latanoprost for open-angle glaucoma (UKGTS): a randomised, multicentre, placebo-controlled trial

Background
Treatments for open-angle glaucoma aim to prevent vision loss through lowering of intraocular pressure, but to our knowledge no placebo-controlled trials have assessed visual function preservation, and the observation periods of previous (unmasked) trials have typically been at least 5 years. We assessed vision preservation in patients given latanoprost compared with those given placebo.

Methods
In this randomised, triple-masked, placebo-controlled trial, we enrolled patients with newly diagnosed open-angle glaucoma at ten UK centres (tertiary referral centres, teaching hospitals, and district general hospitals). Eligible patients were randomly allocated (1:1) with a website-generated randomisation schedule, stratified by centre and with a permuted block design, to receive either latanoprost 0·005% (intervention group) or placebo (control group) eye drops. Drops were administered from identical bottles, once a day, to both eyes. The primary outcome was time to visual field deterioration within 24 months. Analyses were done in all individuals with follow-up data. The Data and Safety Monitoring Committee (DSMC) recommended stopping the trial on Jan 6, 2011 (last patient visit July, 2011), after an interim analysis, and suggested a change in primary outcome from the difference in proportions of patients with incident progression between groups to time to visual field deterioration within 24 months. This trial is registered, number ISRCTN96423140.

Findings
We enrolled 516 individuals between Dec 1, 2006, and March 16, 2010. Baseline mean intraocular pressure was 19·6 mm Hg (SD 4·6) in 258 patients in the latanoprost group and 20·1 mm Hg (4·8) in 258 controls. At 24 months, mean reduction in intraocular pressure was 3·8 mm Hg (4·0) in 231 patients assessed in the latanoprost group and 0·9 mm Hg (3·8) in 230 patients assessed in the placebo group. Visual field preservation was significantly longer in the latanoprost group than in the placebo group: adjusted hazard ratio (HR) 0·44 (95% CI 0·28–0·69; p=0·0003). We noted 18 serious adverse events, none attributable to the study drug.

Interpretation
This is the first randomised placebo-controlled trial to show preservation of the visual field with an intraocular-pressure-lowering drug in patients with open-angle glaucoma. The study design enabled significant differences in vision to be assessed in a relatively short observation period

A cluster randomised controlled trial evaluation and cost-effectiveness analysis of the Roots of Empathy schools-based programme for improving social and emotional wellbeing outcomes among 8-9 year olds in Northern Ireland

A substantial body of evidence suggest that well designed school based prevention programmes can be effective in improving a variety of social, health and academic outcomes for children and young people. This poster presents the methodology for evaluating the Roots of Empathy (ROE) programme. ROE is a universal programme delivered on a whole-class basis for one academic year. It consists of 27 lessons that run over a school year and is based around a monthly classroom visit by an infant and parent, typically recruited from the local community, whom the class 'adopts' at the start of the school year. The evaluation aims to evaluate the immediate and longer term impact of ROE on social and emotional wellbeing outcomes among 8-9 year old pupils, as well as evaluate the cost-effectiveness of the programme.