Realist evaluation - promise, problems and practicalities

Realist evaluation is an innovative, multi-method approach to evaluating the effectiveness of complex health care interventions that is having an increasing impact on the research community. Drawing on their experience doing four realist evaluations in diverse areas of healthcare, the authors offer a comprehensive overview and critique of essential theory and practice. The first paper (Realist review and realist evaluation: an introduction) introduces the main components of the approach and shows how realist review can support realist evaluation. The second paper (Concepts and methodology for realist evaluation: help or hindrance?) provides further detail on the key concepts, shows how they can be operationalised, and discusses the advantages and difficulties of using these ideas. Following these two papers introducing and illustrating the major concepts, the third paper (Realist Evaluation: a critical realist critique) takes a step back to re-consider realist evaluation in relation to its critical realist roots, asking whether it leads to evaluators abandoning the attempt to understand (and if necessary challenge) the underlying values of health care interventions and contenting themselves merely with explicating the factors that help or hinder implementation. The fourth and final paper (Data analysis and theory development in realist evaluation) plunges back into the tangled undergrowth of multiple-method data collection and shows how disparate forms of data can be synthesised for theory development, and the results presented in a form that is useful to practitioners and policy-makers.

Multiplex biotoxin surface plasmon resonance method for marine biotoxins in algal and seawater samples

A multiplex surface plasmon resonance (SPR) biosensor method for the detection of paralytic shellfish poisoning (PSP) toxins, okadaic acid (and analogues) and domoic acid was developed. This method was compared to enzyme-linked immunosorbent assay (ELISA) methods. Seawater samples (n?=?256) from around Europe were collected by the consortia of an EU project MIcroarrays for the Detection of Toxic Algae (MIDTAL) and evaluated using each method. A simple sample preparation procedure was developed which involved lysing and releasing the toxins from the algal cells with glass beads followed by centrifugation and filtering the extract before testing for marine biotoxins by both multi-SPR and ELISA. Method detection limits based on IC20 values for PSP, okadaic acid and domoic acid toxins were 0.82, 0.36 and 1.66 ng/ml, respectively, for the prototype multiplex SPR biosensor. Evaluation by SPR for seawater samples has shown that 47, 59 and 61 % of total seawater samples tested positive (result greater than the IC20) for PSP, okadaic acid (and analogues) and domoic acid toxins, respectively. Toxic samples were received mainly from Spain and Ireland. This work has demonstrated the potential of multiplex analysis for marine biotoxins in algal and seawater samples with results available for 24 samples within a 7 h period for three groups of key marine biotoxins. Multiplex immunological methods could therefore be used as early warning monitoring tools for a variety of marine biotoxins in seawater samples.

Comparative evaluation of 2,3-bis [2-methyloxy-4-nitro-5-sulfophenyl]-2H-tetrazolium-5-carboxanilide (XTT) and 2-(2-methoxy-4-nitrophenyl)-3-(4-nitrophenyl)-5-(2, 4-disulfophenyl)-2H-tetrazolium, monosodium salt (WST-8) rapid colorimetric assays for antimicrobial susceptibility testing of staphylococci and ESBL-producing clinical isolates

A new generation of water soluble tetrazolium salts have recently become available and in this study we compared a colorimetric assay developed using one of these salts, 2-(2-methoxy-4-nitrophenyl)-3-(4-nitrophenyl)-5-(2, 4-disulfophenyl)-2H-tetrazolium, monosodium salt (WST-8), with a previously developed 2,3-bis [2-methyloxy-4-nitro-5-sulfophenyl]-2H-tetrazolium-5-carboxanilide (XTT) colorimetric assay to determine which agent is most suitable for use as a colorimetric indicator in susceptibility testing. The MICs of 6 antibiotics were determined for 33 staphylococci using both colorimetric assays and compared with those obtained using the British Society for Antimicrobial Chemotherapy reference broth microdilution method. Absolute categorical agreement between the reference and test methods ranged from 79% (cefuroxime) to 100% (vancomycin) for both assays. No minor or major errors occurred using either assay with very major errors ranging from zero (vancomycin) to seven (cefuroxime). Analysis of the distribution of differences in the log2 dilution MIC results revealed overall agreement, within the accuracy limits of the standard test (± 1 log2 dilution), using the XTT and WST-8 assays of 98% and 88%, respectively. Further studies on 31 ESBL-producing isolates were performed using the XTT method with absolute categorical agreement ranging from 87% (nitrofurantoin) to 100% (ofloxacin and meropenem). No errors were noted for either ofloxacin or meropenem with overall agreement of 91%. The data suggests that XTT is more reliable and accurate than WST-8 for use in a rapid antimicrobial susceptibility test.

A scaling method for modelling the crashworthiness of novel roadside barrier designs

In this paper, a novel method for modelling a scaled vehicle–barrier crash test similar to the 20◦ angled barrier test specified in EN 1317 is reported. The intended application is for proof-of-concept evaluation of novel roadside barrier designs, and as a cost-effective precursor to full-scale testing or detailed computational modelling. The method is based on the combination of the conservation of energy law and the equation of motion of a spring mass system representing the impact, and shows, for the first time, the feasibility of applying classical scaling theories to evaluation of roadside barrier design. The scaling method is used to set the initial velocity of the vehicle in the scaled test and to provide scaling factors to convert the measured vehicle accelerations in the scaled test to predicted full-scale accelerations. These values can then be used to calculate the Acceleration Severity Index score of the barrier for a full-scale test. The theoretical validity of the method is demonstrated by comparison to numerical simulations of scaled and full-scale angled barrier impacts using multibody analysis implemented in the crash simulation software MADYMO. Results show a maximum error of 0.3% ascribable to the scaling method.

Evaluation of the contribution of theatre attendance to medical undergraduate neuroscience teaching - a pilot study.

Introduction: Medical students often attend the neurosurgical theatre during their clinical neurosciences attachment. However, few studies have been performed to objectively assess the value of this theatre-based learning experience. The main aim of this study was to explore student perceptions on the contribution of neurosurgical theatre attendance to clinical neuroscience teaching.

Materials and Methods: Third-year medical students undergoing their 2-week clinical neurosciences rotation at the Royal Hospitals Belfast were invited to participate in this study. A multi-method strategy was employed using a survey questionnaire comprising of closed and open-ended questions followed by semi-structured interviews to gain a greater 'in-depth' analysis of the potential contribution of neurosurgical theatre attendance to neuroscience teaching.

Results: Based on the completed survey responses of 22 students, the overall experience of neurosurgical theatre-based learning was a positive one. 'In-depth' analysis from semi-structured interviews indicated that students felt that some aspects of their neurosurgical theatre attendance could be improved. Better preparation such as reading up on the case in hand and an introduction to simple theatre etiquette to put the student at ease (in particular, for students who had never attended theatre previously), would improve the learning experience. In addition, having an expectation of what students are expected to learn in theatre making it more learning outcomes-based would probably make it feel a more positive experience by the student.

Conclusions: The vast majority of students acknowledged the positive learning outcomes of neurosurgical theatre attendance and felt that it should be made a mandatory component of the curriculum.

Evaluation of sources of uncertainties in microtensile bond strength of dental adhesive system for different specimen geometries

Objective: The aim of this research is to use finite element analysis (FEA) to quantify the effect of the sample shape and the imperfections induced during the manufacturing process of samples on the bond strength and modes of failure of dental adhesive systems through microtensile test. Using the FEA prediction for individual parameters effect, estimation of expected variation and spread of the microtensile bond strength results for different sample geometries is made. Methods: The estimated stress distributions for three different sample shapes, hourglass, stick and dumbbell predicted by FEA are used to predict the strength for different fracture modes. Parameters such as the adhesive thickness, uneven interface of the adhesive and composite and dentin, misalignment of axis of loading, the existence of flaws such as induced cracks during shaping the samples or bubbles created during application of the adhesive are considered. Microtensile experiments are performed simultaneously to measure bond strength and modes of failure. These are compared with the FEA results. Results: The relative bonding strength and its standard deviation for the specimens with different geometries measured through the microtensile tests confirm the findings of the FEA. The hourglass shape samples show lower tensile bond strength and standard deviation compared to the stick and dumbbell shape samples. ANOVA analysis confirms no significant difference between dumbbell and stick geometry results, and major differences of these two geometries compared to hourglass shape measured values. Induced flaws in the adhesive and misalignment of the angle of application of load have significant effect on the microtensile bond strength. Using adhesive with higher modulus the differences between the bond strength of the three sample geometries increase. Significance: The result of the research clarifies the importance of the sample geometry chosen in measuring the bond strength. It quantifies the effect of the imperfections on the bond strength for each of the sample geometries through a systematic and all embracing study. The results explain the reasons of the large spread of the microtensile test results reported by various researchers working in different labs and the need for standardization of the test method and sample shape used in evaluation of the dentin-adhesive bonding system. © 2007 Academy of Dental Materials.

Evaluation of the effect of wheat aleurone-rich foods on markers of antioxidant status, inflammation and endothelial function in apparently healthy men and women

Observational data show an inverse association between the consumption of whole-grain foods, and inflammation and related diseases. Although the underlying mechanisms are unclear, whole grains, and in particular the aleurone layer, contain a wide range of components with putative antioxidant and anti-inflammatory effects. We evaluated the effects of a diet high in wheat aleurone on plasma antioxidants status, markers of inflammation and endothelial function. In this parallel, participant-blinded intervention, seventy-nine healthy, older, overweight participants (45-65 years, BMI>25 kg/m²) incorporated either aleurone-rich cereal products (27 g aleurone/d), or control products balanced for fibre and macronutrients, into their habitual diets for 4 weeks. Fasting blood samples were taken at baseline and on day 29. Results showed that, compared to control, consumption of aleurone-rich products provided substantial amounts of micronutrients and phytochemicals which may function as antioxidants. Additionally, incorporating these products into a habitual diet resulted in significantly lower plasma concentrations of the inflammatory marker, C-reactive protein (P = 0·035), which is an independent risk factor for CVD. However, no changes were observed in other markers of inflammation, antioxidant status or endothelial function. These results provide a possible mechanism underlying the beneficial effects of longer-term whole-grain intake. However, it is unclear whether this effect is owing to a specific component, or a combination of components in wheat aleurone.

Physical and electrochemical evaluation of ATO supported IrO2 catalyst for proton exchange membrane water electrolyser

Antimony doped tin oxide (ATO) was studied as a support material for IrO₂ in proton exchange membrane water electrolyser (PEMWE). Adams fusion method was used to prepare the IrO₂-ATO catalysts. The physical and electrochemical characterisation of the catalysts were carried out using X-ray diffraction (XRD), scanning electron microscopy (SEM), transmission electron microscopy (TEM), powder conductivity, cyclic voltammetry (CV) and membrane electrode assembly (MEA) polarisation. The BET surface area and electronic conductivity of the supported catalysts were found to be predominantly arisen from the IrO₂. Supported catalyst showed higher active surface area than the pristine IrO₂ in CV analysis with 85% H₃PO₄ as electrolyte. The MEA performance using Nafion®−115 membrane at 80 °C and atmospheric pressure showed a better performance for IrO₂ loading ≥60 wt.% than the pristine IrO₂ with a normalised current density of 1625 mA cm⁻² @1.8 V for the 60% IrO₂-ATO compared to 1341 mA cm⁻² for the pristine IrO₂ under the same condition. The higher performance of the supported catalysts was mainly attributed to better dispersion of active IrO₂ on electrochemically inactive ATO support material, forming smaller IrO₂ crystallites. A 40 wt.% reduction in the IrO₂ was achieved by utilising the support material.

Evaluation of methodologies to determine vegetable oil species present in oil mixtures: Proposition of an approach to meet the EU legislation demands for correct vegetable oils labelling

Refined vegetable oils are widely used in the food industry as ingredients or components in many processed food products in the form of oil blends. To date, the generic term 'vegetable oil' has been used in the labelling of food containing oil blends. With the introduction of new EU Regulation for Food Information (1169/2011) due to take effect in 2014, the oil species used must be clearly identified on the package and there is a need for development of fit for purpose methodology for industry and regulators alike to verify the oil species present in a product. The available methodologies that may be employed to authenticate the botanical origin of a vegetable oil admixture were reviewed and evaluated. The majority of the sources however, described techniques applied to crude vegetable oils such as olive oil due to the lack of refined vegetable oil focused studies. Nevertheless, DNA based typing methods and stable isotopes procedures were found not suitable for this particular purpose due to several issues. Only a small number of specific chromatographic and spectroscopic fingerprinting methods in either targeted or untargeted mode were found to be applicable in potentially providing a solution to this complex authenticity problem. Applied as a single method in isolation, these techniques would be able to give limited information on the oils identity as signals obtained for various oil types may well be overlapping. Therefore, more complex and combined approaches are likely to be needed to identify the oil species present in oil blends employing a stepwise approach in combination with advanced chemometrics. Options to provide such a methodology are outlined in the current study.

Development of a method to quantify the DNA content in DNA- cationic peptide nanoparticles

Gene therapy has the potential to provide safe and targeted therapies for a variety of diseases. A range of intracellular gene delivery vehicles have been proposed for this purpose. Non-viral vectors are a particularly attractive option and among them cationic peptides have emerged as promising candidates. For the pharmaceutical formulation and application to clinical studies it is necessary to quantify the amount of pDNA condensed with the delivery system. There is a severe deficiency in this area, thus far no methods have been reported specifically for pDNA condensed with cationic peptide to form nanoparticles. The current study seeks to address this and describes the evaluation of a range of disruption agents to extract DNA from nanoparticles formed by condensation with cationic fusogenic peptides RALA and KALA. Only proteinase K exhibited efficient and reproducible results and compatibility with the PicoGreen reagent based quantification assay. Thus we report for the first time a simple and reliable method that can quantify the pDNA content in pDNA cationic peptide nanoparticles.

Evaluation of Corrosion Fatigue Behaviour of Laser-welded NiTi Alloys using Bending Rotation Fatigue Test in Simulated Body Fluid

Shape memory NiTi alloys have been used extensively for medical device applications such as orthopedic, dental, vascular and cardiovascular devices on account of their unique shape memory effect (SME) and super-elasticity (SE). Laser welding is found to be the most suitable method used to fabricate NiTi-based medical components. However, the performance of laser-welded NiTi alloys under corrosive environments is not fully understood and a specific focus on understanding the corrosion fatigue behaviour is not evident in the literature. This study reveals a comparison of corrosion fatigue behaviour of laser-welded and bare NiTi alloys using bending rotation fatigue (BRF) test which was integrated with a specifically designed corrosion cell. The testing environment was Hanks’ solution (simulated body fluid) at 37.5oC. Electrochemical impedance spectroscopic (EIS) measurement was carried out to monitor the change of corrosion resistance at different periods during the BRF test. Experiments indicate that the laser-welded NiTi alloy would be more susceptible to the corrosion fatigue attack than the bare NiTi alloy. This finding can serve as a benchmark for the product designers and engineers to determine the factor of safety of NiTi medical devices fabricated using laser welding.

Evaluation of an in situ spatial resolution instrument for fixed beds through the assessment of the invasiveness of probes and a comparison with a micro-kinetic model

This paper reports the detailed description and validation of a fully automated, computer controlled analytical method to spatially probe the gas composition and thermal characteristics in packed bed systems. This method has been designed to limit the invasiveness of the probe, a characteristic assessed using CFD. The thermocouple is aligned with the sampling holes to enable simultaneous recording of the gas composition and temperature profiles. This analysis technique has been validated by studying CO oxidation over a 1% Pt/Al2O3 catalyst. The resultant profiles have been compared with a micro-kinetic model, to further assess the strength of the technique. 

Evaluation of the Factors Contributing to Levonorgestrel Binding in Addition Cure Silicone Elastomer Vaginal Rings

Background: Following progress of the dapivirine (DPV)-releasing silicone elastomer (SE) vaginal ring (VR) into Phase III clinical studies, there is now interest in developing next-generation rings that additionally provide contraception. Levonorgestrel (LNG) is a safe and effective progestin that is being widely considered for use as a hormonal contraceptive agent in future multipurpose prevention technology (MPT) products. Although LNG has previously been incorporated into various controlled release SE devices, minimal attention has focused on its propensity to irreversibly react with addition cure SE systems. Here, for the first time, we investigate this LNG binding phenomenon and outline strategies for overcoming it.
Methods: VRs containing various loadings of DPV and LNG were manufactured and in vitro release assessed. Different LNG-only SE samples were also prepared to assess the following parameters: (i) addition cure vs. condensation cure SEs; (ii) different types of addition cure SEs; (iii) mixing time, (iv) cure temperature, (v) cure time; and (vi) LNG particle size. After manufacture, the LNG-only samples were assayed for total drug content using a solvent extraction method. The SE curing reaction and the LNG binding reaction was probed using nuclear magnetic resonance (NMR) spectroscopy. Results:
Under certain drug/formulation/processing conditions, LNG was not recoverable from VRs. Further studies using non-ring samples showed that: (a) the phenomenon was only observed with addition cure SEs (and not condensation cure SEs); (b) the extent of binding was dependent upon the type of addition cure SE; (c) micronised LNG showed significantly greater binding than non-micronised LNG; (d) the extent of binding correlated with increased mixing time, cure time and cure temperature.
Conclusions: Careful control of the API characteristics, the SE composition, and the manufacturing conditions will be necessary to establish a practical VR formulation for controlled release of LNG.

Evaluation of sampling methods for learning from imbalanced data

The problem of learning from imbalanced data is of critical importance in a large number of application domains and can be a bottleneck in the performance of various conventional learning methods that assume the data distribution to be balanced. The class imbalance problem corresponds to dealing with the situation where one class massively outnumbers the other. The imbalance between majority and minority would lead machine learning to be biased and produce unreliable outcomes if the imbalanced data is used directly. There has been increasing interest in this research area and a number of algorithms have been developed. However, independent evaluation of the algorithms is limited. This paper aims at evaluating the performance of five representative data sampling methods namely SMOTE, ADASYN, BorderlineSMOTE, SMOTETomek and RUSBoost that deal with class imbalance problems. A comparative study is conducted and the performance of each method is critically analysed in terms of assessment metrics. © 2013 Springer-Verlag.

Factors affecting the successful implementation and sustainability of the Liverpool Care Pathway for dying patients – a realist evaluation

Objectives: The Liverpool Care Pathway for the dying patient (LCP) was designed to improve end-of-life care in generalist health care settings. Controversy has led to its withdrawal in some jurisdictions. The main objective of this research was to identify the influences that facilitated or hindered successful LCP implementation.

Method: An organisational case study using realist evaluation in one health and social care trust in Northern Ireland. Two rounds of semi-structured interviews were conducted with two policy makers and twenty two participants with experience and/or involvement in management of the LCP during 2011 and 2012.

Results: Key resource inputs included facilitation with a view to maintaining LCP ‘visibility’, reducing anxiety among nurses and increasing their confidence regarding the delivery of end-of-life care; and nurse and medical education designed to increase professional self-efficacy and reduce misuse and misunderstanding of the LCP. Key enabling contexts were consistent senior management support; ongoing education and training tailored to the needs of each professional group; and an organisational cultural change in the hospital setting that encompassed end-of-life care.

Conclusion: There is a need to appreciate the organizationally complex nature of intervening to improve end-of-life care. Successful implementation of evidence-based interventions for end-of-life care requires commitment to planning, training and ongoing review that takes account of different perspectives, institutional hierarchies and relationships and the educational needs of professional disciplines. There is a need also to recognise that medical consultants require particular support in their role as gatekeepers and as a lead communication channel with patients and their relatives.