Where there is smoke, there is (the potential for) fire: soft indicators of research and policy impact

There is a growing literature examining the impact of research on informing policy, and of research and policy on practice. Research and policy do not have the same types of impact on practice but can be evaluated using similar approaches. Sometimes the literature provides a platform for methodological debate but mostly it is concerned with how research can link to improvements in the process and outcomes of education, how it can promote innovative policies and practice, and how it may be successfully disseminated. Whether research-informed or research-based, policy and its implementation is often assessed on such 'hard' indicators of impact as changes in the number of students gaining five or more A to C grades in national examinations or a percentage fall in the number of exclusions in inner city schools. Such measures are necessarily crude, with large samples smoothing out errors and disguising instances of significant success or failure. Even when 'measurable' in such a fashion, however, the impact of any educational change or intervention may require a period of years to become observable. This paper considers circumstances in which short-term change may be implausible or difficult to observe. It explores how impact is currently theorized and researched and promotes the concept of 'soft' indicators of impact in circumstances in which the pursuit of conventional quantitative and qualitative evidence is rendered impractical within a reasonable cost and timeframe. Such indicators are characterized by their avowedly subjective, anecdotal and impressionistic provenance and have particular importance in the context of complex community education issues where the assessment of any impact often faces considerable problems of access. These indicators include the testimonies of those on whom the research intervention or policy focuses (for example, students, adult learners), the formative effects that are often reported (for example, by head teachers, community leaders) and media coverage. The collation and convergence of a wide variety of soft indicators (Where there is smoke …) is argued to offer a credible means of identifying subtle processes that are often neglected as evidence of potential and actual impact (… there is fire).

Effect of tailored practice and patient care plans on secondary prevention of heart disease in general practice: cluster randomised controlled trial

Objective: To test the effectiveness of a complex intervention designed, within a theoretical framework, to improve outcomes for patients with coronary heart disease. Design: Cluster randomised controlled multicentre trial. Setting: General practices in Northern Ireland and the Republic of Ireland, regions with different healthcare systems. Participants: 903 patients with established coronary heart disease registered with one of 48 practices. Intervention: Tailored care plans for practices (practice based training in prescribing and behaviour change, administrative support, quarterly newsletter), and tailored care plans for patients (motivational interviewing, goal identification, and target setting for lifestyle change) with reviews every four months at the practices. Control practices provided usual care. Main outcome measures: The proportion of patients at 18 month follow-up above target levels for blood pressure and total cholesterol concentration, and those admitted to hospital, and changes in physical and mental health status (SF-12). Results: At baseline the numbers (proportions) of patients above the recommended limits were: systolic blood pressure greater than 140 mm Hg (305/899; 33.9%, 95% confidence interval 30.8% to 33.9%), diastolic blood pressure greater than 90 mm Hg (111/901; 12.3%, 10.2% to 14.5%), and total cholesterol concentration greater than 5 mmol/l (188/860; 20.8%, 19.1% to 24.6%). At the 18 month follow-up there were no significant differences between intervention and control groups in the numbers (proportions) of patients above the recommended limits: systolic blood pressure, intervention 98/360 (27.2%) v control, 133/405 (32.8%), odds ratio 1.51 (95% confidence interval 0.99 to 2.30; P=0.06); diastolic blood pressure, intervention 32/360 (8.9%) v control, 40/405 (9.9%), 1.40 (0.75 to 2.64; P=0.29); and total cholesterol concentration, intervention 52/342 (15.2%) v control, 64/391 (16.4%), 1.13 (0.63 to 2.03; P=0.65). The number of patients admitted to hospital over the 18 month study period significantly decreased in the intervention group compared with the control group: 107/415 (25.8%) v 148/435 (34.0%), 1.56 (1.53 to 2.60; P=0.03). Conclusions: Admissions to hospital were significantly reduced after an intensive 18 month intervention to improve outcomes for patients with coronary heart disease, but no other clinical benefits were shown, possibly because of a ceiling effect related to improved management of the disease. Trial registration: Current Controlled Trials ISRCTN24081411.

Vitamins C and E for prevention of pre-eclampsia in women with type 1 diabetes (DAPIT): a randomised placebo-controlled trial.

Background
Results of several trials of antioxidant use during pregnancy have not shown a reduction in pre-eclampsia, but the eff ect in women with diabetes is unknown. We aimed to assess whether supplementation with vitamins C and E reduced incidence of pre-eclampsia in women with type 1 diabetes.

Methods
We enrolled women from 25 UK antenatal metabolic clinics in a multicentre randomised placebo-controlled trial. Eligibility criteria were type 1 diabetes preceding pregnancy, presentation between 8 weeks’ and 22 weeks’ gestation, singleton pregnancy, and age 16 years or older. Women were randomly allocated in a 1:1 ratio to receive1000 mg vitamin C and 400 IU vitamin E (a-tocopherol) or matched placebo daily until delivery. The randomisation sequence was stratifi ed by centre with balanced blocks of eight patients. All trial personnel and participants were masked to treatment allocation. The primary endpoint was pre-eclampsia, which we defi ned as gestational hypertension with proteinuria. Analysis was by modifi ed intention to treat. This study is registered, ISRCTN27214045.

Findings
Between April, 2003, and June, 2008, 762 women were randomly allocated to treatment groups (379 vitamin supplementation, 383 placebo). The primary endpoint was assessed for 375 women allocated to receive vitamins, and 374 allocated to placebo. Rates of pre-eclampsia did not differ between vitamin (15%, n=57) and placebo (19%, 70)groups (risk ratio 0·81, 95% CI 0·59–1·12). No adverse maternal or neonatal outcomes were reported.

Interpretation
Supplementation with vitamins C and E did not reduce risk of pre-eclampsia in women with type 1 diabetes. However, the possibility that vitamin supplementation might be benefi cial in women with a low antioxidant status at baseline needs further testing.