Evaluating the use of multimedia authoring with dyslexic learners: a case study

This paper reports on the findings of a case study set up to explore the possible benefits that dyslexic learners might have when engaging in the creation of their own multimedia project. Two children with specific learning difficulties worked with the author to develop a multimedia presentation. The children developed authoring skills (such as planning and drafting, composition, revision and reflection, proof reading and presentation) and became active, motivated learners. It is believed that the open-ended character of a multimedia authoring package can encourage creative thinking and interest for content and style of presentation.

Evaluating the legacies of state-building: success, failure, and the role of responsibility

What impact do international state-building missions have on the domestic politics of states they seek to build, and how can we measure this impact with confidence? This article seeks to address these questions and challenge some existing approaches that often appear to assume that state-builders leave lasting legacies rather than demonstrating such influence with the use of carefully chosen empirical evidence. Too often, domestic conditions that follow in the wake of international state-building are assumed to follow as a result of international intervention, usually due to insufficient attention to the causal processes that link international actions to domestic outcomes. The article calls for greater appreciation of the methodological challenges to establishing causal inferences regarding the legacies of state-building and identifies three qualitative methodological strategies—process tracing, counterfactual analysis, and the use of control cases—that can be used to improve confidence in causal claims about state-building legacies. The article concludes with a case study of international state-building in East Timor, highlighting several flaws of existing evaluations of the United Nations' role in East Timor and identifying the critical role that domestic actors play even in the context of authoritative international intervention

Evaluating raw ensembles with the continuous ranked probability score

The continuous ranked probability score (CRPS) is a frequently used scoring rule. In contrast with many other scoring rules, the CRPS evaluates cumulative distribution functions. An ensemble of forecasts can easily be converted into a piecewise constant cumulative distribution function with steps at the ensemble members. This renders the CRPS a convenient scoring rule for the evaluation of ‘raw’ ensembles, obviating the need for sophisticated ensemble model output statistics or dressing methods prior to evaluation. In this article, a relation between the CRPS score and the quantile score is established. The evaluation of ‘raw’ ensembles using the CRPS is discussed in this light. It is shown that latent in this evaluation is an interpretation of the ensemble as quantiles but with non-uniform levels. This needs to be taken into account if the ensemble is evaluated further, for example with rank histograms.

Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures

Many different performance measures have been developed to evaluate field predictions in meteorology. However, a researcher or practitioner encountering a new or unfamiliar measure may have difficulty in interpreting its results, which may lead to them avoiding new measures and relying on those that are familiar. In the context of evaluating forecasts of extreme events for hydrological applications, this article aims to promote the use of a range of performance measures. Some of the types of performance measures that are introduced in order to demonstrate a six-step approach to tackle a new measure. Using the example of the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble precipitation predictions for the Danube floods of July and August 2002, to show how to use new performance measures with this approach and the way to choose between different performance measures based on their suitability for the task at hand is shown. Copyright © 2008 Royal Meteorological Society

Evaluating the alternatives

In this article, I study the impacts of a specific incentives-based approach to safety regulation, namely the control of quality through sampling and threatening penalties when quality fails to meet some minimum standard. The welfare-improving impacts of this type of scheme seem high and are cogently illustrated in a recent contribution by Segerson, which stimulated many of the ideas in this paper. For this reason, the reader is referred to Segerson for a background on some of the motivation, and throughout, I make an effort to indicate differences between the two approaches. There are three major differences. First, I dispense with the calculus as much as possible, seeking readily interpreted, closedform solutions to illustrate the main ideas. Second, (strategically optimal, symmetric) Nash equilibria are the mainstay of each of the current models. Third, in the uncertainquality- provision equilibria, each of the Nash suppliers chooses the level of the lower bound for quality as a control and offers a draw from its (private) distribution in a contribution to the (public) pool of quality.

A comprehensive benchmarking system for evaluating global vegetation models.

We present a benchmark system for global vegetation models. This system provides a quantitative evaluation of multiple simulated vegetation properties, including primary production; seasonal net ecosystem production; vegetation cover, composition and 5 height; fire regime; and runoff. The benchmarks are derived from remotely sensed gridded datasets and site-based observations. The datasets allow comparisons of annual average conditions and seasonal and inter-annual variability, and they allow the impact of spatial and temporal biases in means and variability to be assessed separately. Specifically designed metrics quantify model performance for each process, 10 and are compared to scores based on the temporal or spatial mean value of the observations and a “random” model produced by bootstrap resampling of the observations. The benchmark system is applied to three models: a simple light-use efficiency and water-balance model (the Simple Diagnostic Biosphere Model: SDBM), and the Lund-Potsdam-Jena (LPJ) and Land Processes and eXchanges (LPX) dynamic global 15 vegetation models (DGVMs). SDBM reproduces observed CO2 seasonal cycles, but its simulation of independent measurements of net primary production (NPP) is too high. The two DGVMs show little difference for most benchmarks (including the interannual variability in the growth rate and seasonal cycle of atmospheric CO2), but LPX represents burnt fraction demonstrably more accurately. Benchmarking also identified 20 several weaknesses common to both DGVMs. The benchmarking system provides a quantitative approach for evaluating how adequately processes are represented in a model, identifying errors and biases, tracking improvements in performance through model development, and discriminating among models. Adoption of such a system would do much to improve confidence in terrestrial model predictions of climate change 25 impacts and feedbacks.

Evaluating the interdependencies between managing business change, IT sourcing and IT governance

Organisations need the right business and IT capabilities in order to achieve future business success. It follows that the sourcing of these capabilities is an important decision. Yet, there is a lack of consensus on the approach to decid-ing where and how to source the core operational capabilities. Furthermore, de-veloping its dynamic capability enables an organisation to effectively manage change its operational capabilities. Recent research has proposed that analysing business capabilities is a key pre-requisite to defining its Information Technology (IT) solutions. This research builds on these findings by considering the interde-pendencies between the dynamic business change capability and the sourcing of IT capabilities. Further it examines the decision-making oversight of these areas as implemented through IT governance. There is a good understanding of the direct impact of IT sourcing decision on operational capabilities However, there is a lack of research on the indirect impact to the capability of managing business change. Through a review of prior research and initial pilot field research, a capability framework and three main propositions are proposed, each examining a two-way interdependency. This paper describes the development of the integrated capa-bility framework and the rationale for the propositions. These respectively cover managing business change, IT sourcing and IT governance. Firstly, the sourcing of IT affects both the operational capabilities and the capability to manage business change. Similarly a business change may result in new or revised operational ca-pabilities, which can influence the IT sourcing decision resulting in a two-way rela-tionship. Secondly, this IT sourcing is directed under IT governance, which pro-vides a decision-making framework for the organisation. At the same time, the IT sourcing can have an impact on the IT governance capability, for example by out-sourcing key capabilities; hence this is potentially again a two-way relationship. Finally, there is a postulated two-way relationship between IT governance and managing business change in that IT governance provides an oversight of manag-ing business change through portfolio management while IT governance is a key element of the business change capability. Given the nature and novelty of this framework, a philosophical paradigm of constructivism is preferred. To illustrate and explore the theoretical perspectives provided, this paper reports on the find-ings of a case study incorporating eight high-level interviews with senior execu-tives in a German bank with 2300 employees. The collected data also include or-ganisational charts, annual reports, project and activity portfolio and benchmark reports for the IT budget. Recommendations are made for practitioners. An understanding of the interdependencies can support professionals in improving business success through effectively managing business change. Additionally, they can be assisted to evaluate the impact of IT sourcing decisions on the organisa-tion’s operational and dynamic capabilities, using an appropriate IT governance framework.

Observational challenges in evaluating climate models

Activities like the Coupled Model Intercomparison Project (CMIP) have revolutionized climate modelling in terms of our ability to compare models and to process information about climate projections and their uncertainties. The evaluation of models against observations is now considered a key component of multi-model studies. While there are a number of outstanding scientific issues surrounding model evaluation, notably the open question of how to link model performance to future projections, here we highlight a specific but growing problem in model evaluation - that of uncertainties in the observational data that are used to evaluate the models. We highlight the problem using an example obtained from studies of the South Asian Monsoon but we believe the problem is a generic one which arises in many different areas of climate model evaluation and which requires some attention by the community.

Evaluating relative impact of virtual reality system variables on architectural design comprehension and presence - A variable-centered approach using fractional factorial experiment

The relative contributions of five variables (Stereoscopy, screen size, field of view, level of realism and level of detail) of virtual reality systems on spatial comprehension and presence are evaluated here. Using a variable-centered approach instead of an object-centric view as its theoretical basis, the contributions of these five variables and their two-way interactions are estimated through a 25-1 fractional factorial experiment (screening design) of resolution V with 84 subjects. The experiment design, procedure, measures used, creation of scales and indices, results of statistical analysis, their meaning and agenda for future research are elaborated.