Refinement of the Clinical Scenario Evaluation Framework for Assessment of Competing Development Strategies with an Application to Multiple Sclerosis

When competing strategies for development programs, clinical trial designs, or data analysis methods exist, the alternatives need to be evaluated in a systematic way to facilitate informed decision making. Here we describe a refinement of the recently proposed clinical scenario evaluation framework for the assessment of competing strategies. The refinement is achieved by subdividing key elements previously proposed into new categories, distinguishing between quantities that can be estimated from preexisting data and those that cannot and between aspects under the control of the decision maker from those that are determined by external constraints. The refined framework is illustrated by an application to a design project for an adaptive seamless design for a clinical trial in progressive multiple sclerosis.

Defining a framework for the evaluation of information

In any enterprise, decisions need be made during the life cycle of information about its management. This requires information evaluation to take place; a little-understood process. For evaluation support to be both effective and resource efficient, some sort of automatic or semi-automatic evaluation method would be invaluable. Such a method would require an understanding of the diversity of the contexts in which evaluation takes place so that evaluation support can have the necessary context-sensitivity. This paper identifies the dimensions influencing the information evaluation process and defines the elements that characterise them, thus providing the foundations for a context-sensitive evaluation framework.

Evaluation of agri-environmental and forestry schemes with multiple objectives

The evaluation of EU policy in the area of rural land use management often encounters problems of multiple and poorly articulated objectives. Agri-environmental policy has a range of aims, including natural resource protection, biodiversity conservation and the protection and enhancement of landscape quality. Forestry policy, in addition to production and environmental objectives, increasingly has social aims, including enhancement of human health and wellbeing, lifelong learning, and the cultural and amenity value of the landscape. Many of these aims are intangible, making them hard to define and quantify. This article describes two approaches for dealing with such situations, both of which rely on substantial participation by stakeholders. The first is the Agri-Environment Footprint Index, a form of multi-criteria participatory approach. The other, applied here to forestry, has been the development of ‘multi-purpose’ approaches to evaluation, which respond to the diverse needs of stakeholders through the use of mixed methods and a broad suite of indicators, selected through a participatory process. Each makes use of case studies and involves stakeholders in the evaluation process, thereby enhancing their commitment to the programmes and increasing their sustainability. Both also demonstrate more ‘holistic’ approaches to evaluation than the formal methods prescribed in the EU Common Monitoring and Evaluation Framework.

Environmental evaluation of agri-environment schemes using participatory approaches: experiences of testing the Agri-Environmental Footprint Index

The Agri-Environment Footprint Index (AFI) has been developed as a generic methodology to assess changes in the overall environmental impacts from agriculture at the farm level and to assist in the evaluation of European agri-environmental schemes (AES). The methodology is based on multicriteria analysis (MCA) and involves stakeholder participation to provide a locally customised evaluation based on weighted environmental indicators. The methodology was subjected to a feasibility assessment in a series of case studies across the EU. The AFI approach was able to measure significant differences in environmental status between farms that participated in an AES and nonparticipants. Wider environmental concerns, beyond the scheme objectives, were also considered in some case studies and the benefits for identification of unintentional (and often beneficial) impacts of AESs are presented. The participatory approach to AES valuation proved efficient in different environments and administrative contexts. The approach proved to be appropriate for environmental evaluation of complex agri-environment systems and can complement any evaluation conducted under the Common Monitoring and Evaluation Framework. The applicability of the AFI in routine monitoring of AES impacts and in providing feedback to improve policy design is discussed.

Impact evaluation of a Kenyan Eco Business: a pilot framework.

Promoting social and environmental entrepreneurship is suggested by many, including international institutions, national development agencies and non-governmental organizations, as critical in tackling longstanding complex global sustainable development problems. However while interest in this kind of alternative entrepreneurship grows, with a proliferation of claims made about its potential to catalyze societal transformation, research in this field remains nascent and fragmented, particularly in relation to Africa. There are few examples of work systematically examining the impacts of social and environmental enterprises on sustainable development and poverty alleviation, especially research based on rigorous empirical fieldwork. This paper begins addressing these limitations by proposing a framework for mapping the sustainable development and poverty alleviation impacts of social and environmental enterprises in Africa. This framework is then piloted with reference to a Kenyan ecobusiness.

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Land use for England and Wales: evaluation of management options to support 'good ecological status' in surface freshwaters

This paper analyses historic records of agricultural land use and management for England and Wales from 1931 and 1991 and uses export coefficient modelling to hindcast the impact of these practices on the rates of diffuse nitrogen (N) and phosphorus (P) export to water bodies for each of the major geo-climatic regions of England and Wales. Key trends indicate the importance of animal agriculture as a contributor to the total diffuse agricultural nutrient loading on waters, and the need to bring these sources under control if conditions suitable for sustaining 'Good Ecological Status' under the Water Framework Directive are to be generated. The analysis highlights the importance of measuring changes in nutrient loading in relation to the catchment-specific baseline state for different water bodies. The approach is also used to forecast the likely impact of broad regional scale scenarios on nutrient export to waters and highlights the need to take sensitive land out of production, introduce ceilings on fertilizer use and stocking densities, and controls on agricultural practice in higher risk areas where intensive agriculture is combined with a low intrinsic nutrient retention capacity, although the uncertainties associated with the modelling applied at this scale should be taken into account in the interpretation of model output. The paper advocates the need for a two-tiered approach to nutrient management, combining broad regional policies with targeted management in high risk areas at the catchment and farm scale.

Evaluation of two hybrid metric-conceptual models, for simulating phosphorus transfer from agricultural land in the river enborne, a lowland UK catchment

Across Europe, elevated phosphorus (P) concentrations in lowland rivers have made them particularly susceptible to eutrophication. This is compounded in southern and central UK by increasing pressures on water resources, which may be further enhanced by the potential effects of climate change. The EU Water Framework Directive requires an integrated approach to water resources management at the catchment scale and highlights the need for modelling tools that can distinguish relative contributions from multiple nutrient sources and are consistent with the information content of the available data. Two such models are introduced and evaluated within a stochastic framework using daily flow and total phosphorus concentrations recorded in a clay catchment typical of many areas of the lowland UK. Both models disaggregate empirical annual load estimates, derived from land use data, as a function of surface/near surface runoff, generated using a simple conceptual rainfall-runoff model. Estimates of the daily load from agricultural land, together with those from baseflow and point sources, feed into an in-stream routing algorithm. The first model assumes constant concentrations in runoff via surface/near surface pathways and incorporates an additional P store in the river-bed sediments, depleted above a critical discharge, to explicitly simulate resuspension. The second model, which is simpler, simulates P concentrations as a function of surface/near surface runoff, thus emphasising the influence of non-point source loads during flow peaks and mixing of baseflow and point sources during low flows. The temporal consistency of parameter estimates and thus the suitability of each approach is assessed dynamically following a new approach based on Monte-Carlo analysis. (c) 2004 Elsevier B.V. All rights reserved.

What is the new medicines use review ‘patient survey’ attempting to capture in the context of existing patient satisfaction with pharmacy questionnaires and a new conceptual framework?

The community pharmacy service medicines use review (MUR) was introduced in 2005 ‘to improve patient knowledge, concordance and use of medicines’ through a private patient–pharmacist consultation. The MUR presents a fundamental change in community pharmacy service provision. While traditionally pharmacists are dispensers of medicines and providers of medicines advice, and patients as recipients, the MUR considers pharmacists providing consultation-type activities and patients as active participants. The MUR facilitates a two-way discussion about medicines use. Traditional patient–pharmacist behaviours transform into a new set of behaviours involving the booking of appointments, consultation processes and form completion, and the physical environment of the patient–pharmacist interaction moves from the traditional setting of the dispensary and medicines counter to a private consultation room. Thus, the new service challenges traditional identities and behaviours of the patient and the pharmacist as well as the environment in which the interaction takes place. In 2008, the UK government concluded there is at present too much emphasis on the quantity of MURs rather than on their quality.[1] A number of plans to remedy the perceived imbalance included a suggestion to reward ‘health outcomes’ achieved, with calls for a more focussed and scientific approach to the evaluation of pharmacy services using outcomes research. Specifically, the UK government set out the main principal research areas for the evaluation of pharmacy services to include ‘patient and public perceptions and satisfaction’as well as ‘impact on care and outcomes’. A limited number of ‘patient satisfaction with pharmacy services’ type questionnaires are available, of varying quality, measuring dimensions relating to pharmacists’ technical competence, behavioural impressions and general satisfaction. For example, an often cited paper by Larson[2] uses two factors to measure satisfaction, namely ‘friendly explanation’ and ‘managing therapy’; the factors are highly interrelated and the questions somewhat awkwardly phrased, but more importantly, we believe the questionnaire excludes some specific domains unique to the MUR. By conducting patient interviews with recent MUR recipients, we have been working to identify relevant concepts and develop a conceptual framework to inform item development for a Patient Reported Outcome Measure questionnaire bespoke to the MUR. We note with interest the recent launch of a multidisciplinary audit template by the Royal Pharmaceutical Society of Great Britain (RPSGB) in an attempt to review the effectiveness of MURs and improve their quality.[3] This template includes an MUR ‘patient survey’. We will discuss this ‘patient survey’ in light of our work and existing patient satisfaction with pharmacy questionnaires, outlining a new conceptual framework as a basis for measuring patient satisfaction with the MUR. Ethical approval for the study was obtained from the NHS Surrey Research Ethics Committee on 2 June 2008. References 1. Department of Health (2008). Pharmacy in England: Building on Strengths – Delivering the Future. London: HMSO. www. official-documents.gov.uk/document/cm73/7341/7341.pdf (accessed 29 September 2009). 2. Larson LN et al. Patient satisfaction with pharmaceutical care: update of a validated instrument. JAmPharmAssoc 2002; 42: 44–50. 3. Royal Pharmaceutical Society of Great Britain (2009). Pharmacy Medicines Use Review – Patient Audit. London: RPSGB. http:// qi4pd.org.uk/index.php/Medicines-Use-Review-Patient-Audit. html (accessed 29 September 2009).

The automation and evaluation of nested clade phylogeographic analysis

Nested clade phylogeographic analysis (NCPA) is a popular method for reconstructing the demographic history of spatially distributed populations from genetic data. Although some parts of the analysis are automated, there is no unique and widely followed algorithm for doing this in its entirety, beginning with the data, and ending with the inferences drawn from the data. This article describes a method that automates NCPA, thereby providing a framework for replicating analyses in an objective way. To do so, a number of decisions need to be made so that the automated implementation is representative of previous analyses. We review how the NCPA procedure has evolved since its inception and conclude that there is scope for some variability in the manual application of NCPA. We apply the automated software to three published datasets previously analyzed manually and replicate many details of the manual analyses, suggesting that the current algorithm is representative of how a typical user will perform NCPA. We simulate a large number of replicate datasets for geographically distributed, but entirely random-mating, populations. These are then analyzed using the automated NCPA algorithm. Results indicate that NCPA tends to give a high frequency of false positives. In our simulations we observe that 14% of the clades give a conclusive inference that a demographic event has occurred, and that 75% of the datasets have at least one clade that gives such an inference. This is mainly due to the generation of multiple statistics per clade, of which only one is required to be significant to apply the inference key. We survey the inferences that have been made in recent publications and show that the most commonly inferred processes (restricted gene flow with isolation by distance and contiguous range expansion) are those that are commonly inferred in our simulations. However, published datasets typically yield a richer set of inferences with NCPA than obtained in our random-mating simulations, and further testing of NCPA with models of structured populations is necessary to examine its accuracy.

Comparative evaluation of a new effective population size estimator based on approximate Bayesian computation

We describe and evaluate a new estimator of the effective population size (N-e), a critical parameter in evolutionary and conservation biology. This new "SummStat" N-e. estimator is based upon the use of summary statistics in an approximate Bayesian computation framework to infer N-e. Simulations of a Wright-Fisher population with known N-e show that the SummStat estimator is useful across a realistic range of individuals and loci sampled, generations between samples, and N-e values. We also address the paucity of information about the relative performance of N-e estimators by comparing the SUMMStat estimator to two recently developed likelihood-based estimators and a traditional moment-based estimator. The SummStat estimator is the least biased of the four estimators compared. In 32 of 36 parameter combinations investigated rising initial allele frequencies drawn from a Dirichlet distribution, it has the lowest bias. The relative mean square error (RMSE) of the SummStat estimator was generally intermediate to the others. All of the estimators had RMSE > 1 when small samples (n = 20, five loci) were collected a generation apart. In contrast, when samples were separated by three or more generations and Ne less than or equal to 50, the SummStat and likelihood-based estimators all had greatly reduced RMSE. Under the conditions simulated, SummStat confidence intervals were more conservative than the likelihood-based estimators and more likely to include true N-e. The greatest strength of the SummStat estimator is its flexible structure. This flexibility allows it to incorporate any, potentially informative summary statistic from Population genetic data.

Evaluation of probabilistic occupancy map people detection for surveillance systems

In this paper, we evaluate the Probabilistic Occupancy Map (POM) pedestrian detection algorithm on the PETS 2009 benchmark dataset. POM is a multi-camera generative detection method, which estimates ground plane occupancy from multiple background subtraction views. Occupancy probabilities are iteratively estimated by fitting a synthetic model of the background subtraction to the binary foreground motion. Furthermore, we test the integration of this algorithm into a larger framework designed for understanding human activities in real environments. We demonstrate accurate detection and localization on the PETS dataset, despite suboptimal calibration and foreground motion segmentation input.

The SAFE FOODS framework for improved risk analysis of foods

Three main changes to current risk analysis processes are proposed to improve their transparency, openness, and accountability. First, the addition of a formal framing stage would allow interested parties, experts and officials to work together as needed to gain an initial shared understanding of the issue, the objectives of regulatory action, and alternative risk management measures. Second, the scope of the risk assessment is expanded to include the assessment of health and environmental benefits as well as risks, and the explicit consideration of economic- and social-impacts of risk management action and their distribution. Moreover approaches were developed for deriving improved information from genomic, proteomic and metabolomic profiling methods and for probabilistic modelling of health impacts for risk assessment purposes. Third, in an added evaluation stage, interested parties, experts, and officials may compare and weigh the risks, costs, and benefits and their distribution. As part of a set of recommendations on risk communication, we propose that reports on each stage should be made public.

Towards a method for the economic evaluation of environmental indicators in UK Integrated Arable Farming Systems

Integrated Arable Farming Systems (IAFS), which involve a reduction in the use of off-farm inputs, are attracting considerable research interest in the UK. The objectives of these systems experiments are to compare their financial performance with that from conventional or current farming practices. To date, this comparison has taken little account of any environmental benefits (or disbenefits) of the two systems. The objective of this paper is to review the assessment methodologies available for the analysis of environmental impacts. To illustrate the results of this exercise, the methodology and environmental indicators chosen are then applied to data from one of the LINK - Integrated Farming Systems experimental sites. Data from the Pathhead site in Southern Scotland are used to evaluate the use of invertebrates and nitrate loss as environmental indicators within IAFS. The results suggest that between 1992 and 1995 the biomass of earthworms fell by 28 kg per hectare on the integrated rotation and rose by 31 kg per hectare on the conventional system. This led to environmental costs ranging between £2.24 and £13.44 per hectare for the integrated system and gains of between £2.48 and £14.88 for the conventional system. In terms of nitrate, the integrated system had an estimated loss of £72.21 per hectare in comparison to £149.40 per hectare on the conventional system. Conclusions are drawn about the advantages and disadvantages of this type of analytical framework. Keywords: Farming systems; IAFS; Environmental valuation; Economics; Earthworms; Nitrates; Soil fauna

Evaluation of diagnostic molecular markers for DUS phenotypic assessment in the cereal crop, barley (Hordeum vulgare ssp. vulgare L.)

The deployment of genetic markers is of interest in crop assessment and breeding programmes, due to the potential savings in cost and time afforded. As part of the internationally recognised framework for the awarding of Plant Breeders’ Rights (PBR), new barley variety submissions are evaluated using a suite of morphological traits to ensure they are distinct, uniform and stable (DUS) in comparison to all previous submissions. Increasing knowledge of the genetic control of many of these traits provides the opportunity to assess the potential of deploying diagnostic/perfect genetic markers in place of phenotypic assessment. Here, we identify a suite of 25 genetic markers assaying for 14 DUS traits, and implement them using a single genotyping platform (KASPar). Using a panel of 169 UK barley varieties, we show that phenotypic state at three of these traits can be perfectly predicted by genotype. Predictive values for an additional nine traits ranged from 81 to 99 %. Finally, by comparison of varietal discrimination based on phenotype and genotype resulted in correlation of 0.72, indicating that deployment of molecular markers for varietal discrimination could be feasible in the near future. Due to the flexibility of the genotyping platform used, the genetic markers described here can be used in any number or combination, in-house or by outsourcing, allowing flexible deployment by users. These markers are likely to find application where tracking of specific alleles is required in breeding programmes, or for potential use within national assessment programmes for the awarding of PBRs.