Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk

This paper describes a framework for evaluation of spoken dialogue systems. Typically, evaluation of dialogue systems is performed in a controlled test environment with carefully selected and instructed users. However, this approach is very demanding. An alternative is to recruit a large group of users who evaluate the dialogue systems in a remote setting under virtually no supervision. Crowdsourcing technology, for example Amazon Mechanical Turk (AMT), provides an efficient way of recruiting subjects. This paper describes an evaluation framework for spoken dialogue systems using AMT users and compares the obtained results with a recent trial in which the systems were tested by locally recruited users. The results suggest that the use of crowdsourcing technology is feasible and it can provide reliable results. Copyright © 2011 ISCA.

Towards a process framework for assessing the potential value of technologies

Current technology valuation literature predominantly focuses on explaining the merits and implications of specific tools, but little research is available that takes a contextual process perspective. The aim of this paper is to further develop an integrative process framework that supports the structuring of the valuation process and the more systematic choice of valuation techniques for new technologies. The paper starts by reviewing key concepts and issues that surround the assessment of technology investments and the evidence of what companies use. Many factors need to be brought into the appraisal process, reflecting technological and market conditions. While there is usually a desire to reduce the assessment to a financial value, it is also widely appreciated that there is long term strategic value in securing a technological lead, which is difficult, or even inappropriate, to assess in purely financial terms. The multiple factors involved in the evaluation activity are identified with respect to the changing nature of the appraisal process as the technology matures and the implications for associated tools. The result of the literature review is a process framework which provides a conceptual basis for integrating valuation techniques. This framework is then populated with the results of industrial case studies on technology valuation to allow conclusions on its applicability to be drawn. © 2011 IEEE.

Towards the development of an integrated design framework using the Blackboard model

Current research into the process of engineering design is extending the use of computers towards the acquisition, representation and application of design process knowledge in addition to the existing storage and manipulation of product-based models of design objects. This is a difficult task because the design of mechanical systems is a complex, often unpredictable process involving ill-structured problem solving skills and large amounts of knowledge, some which may be of an incomplete and subjective nature. Design problems require the integration of a variety of modes of working such as numerical, graphical, algorithmic or heuristic and demand products through synthesis, analysis and evaluation activities.

This report presents the results of a feasibility study into the blackboard approach and discusses the development of an initial prototype system that will enable an alphanumeric design dialogue between a designer and an expert to be analysed in a formal way, thus providing real-life protocol data on which to base the blackboard message structures.

Preliminary evaluation of a nuclear scenario involving innovative gas cooled reactors

In order to guarantee a sustainable supply of future energy demand without compromising the environment, some actions for a substantial reduction of CO 2 emissions are nowadays deeply analysed. One of them is the improvement of the nuclear energy use. In this framework, innovative gas-cooled reactors (both thermal and fast) seem to be very attractive from the electricity production point of view and for the potential industrial use along the high temperature processes (e.g., H 2 production by steam reforming or I-S process). This work focuses on a preliminary (and conservative) evaluation of possible advantages that a symbiotic cycle (EPR-PBMR-GCFR) could entail, with special regard to the reduction of the HLW inventory and the optimization of the exploitation of the fuel resources. The comparison between the symbiotic cycle chosen and the reference one (once-through scenario, i.e., EPR-SNF directly disposed) shows a reduction of the time needed to reach a fixed reference level from ∼170000 years to ∼1550 years (comparable with typical human times and for this reason more acceptable by the public opinion). In addition, this cycle enables to have a more efficient use of resources involved: the total electric energy produced becomes equal to ∼630 TWh/year (instead of only ∼530 TWh/year using only EPR) without consuming additional raw materials. © 2009 Barbara Vezzoni et al.