Expert elicitation and its interface with technology : a review with a view to designing Elicitator

Expert knowledge is valuable in many modelling endeavours, particularly where data is not extensive or sufficiently robust. In Bayesian statistics, expert opinion may be formulated as informative priors, to provide an honest reflection of the current state of knowledge, before updating this with new information. Technology is increasingly being exploited to help support the process of eliciting such information. This paper reviews the benefits that have been gained from utilizing technology in this way. These benefits can be structured within a six-step elicitation design framework proposed recently (Low Choy et al., 2009). We assume that the purpose of elicitation is to formulate a Bayesian statistical prior, either to provide a standalone expert-defined model, or for updating new data within a Bayesian analysis. We also assume that the model has been pre-specified before selecting the software. In this case, technology has the most to offer to: targeting what experts know (E2), eliciting and encoding expert opinions (E4), whilst enhancing accuracy (E5), and providing an effective and efficient protocol (E6). Benefits include: -providing an environment with familiar nuances (to make the expert comfortable) where experts can explore their knowledge from various perspectives (E2); -automating tedious or repetitive tasks, thereby minimizing calculation errors, as well as encouraging interaction between elicitors and experts (E5); -cognitive gains by educating users, enabling instant feedback (E2, E4-E5), and providing alternative methods of communicating assessments and feedback information, since experts think and learn differently; and -ensuring a repeatable and transparent protocol is used (E6).

Use of expert panels in highway safety: A critique

Expert panels have been used extensively in the development of the "Highway Safety Manual" to extract research information from highway safety experts. While the panels have been used to recommend agendas for new and continuing research, their primary role has been to develop accident modification factors—quantitative relationships between highway safety and various highway safety treatments. Because the expert panels derive quantitative information in a “qualitative” environment and because their findings can have significant impacts on highway safety investment decisions, the expert panel process should be described and critiqued. This paper is the first known written description and critique of the expert panel process and is intended to serve professionals wishing to conduct such panels.

Bayesian methodology incorporating expert judgment for ranking countermeasure effectiveness under uncertainty: Example applied to at grade railroad crossings in Korea

This paper describes the formalization and application of a methodology to evaluate the safety benefit of countermeasures in the face of uncertainty. To illustrate the methodology, 18 countermeasures for improving safety of at grade railroad crossings (AGRXs) in the Republic of Korea are considered. Akin to “stated preference” methods in travel survey research, the methodology applies random selection and laws of large numbers to derive accident modification factor (AMF) densities from expert opinions. In a full Bayesian analysis framework, the collective opinions in the form of AMF densities (data likelihood) are combined with prior knowledge (AMF density priors) for the 18 countermeasures to obtain ‘best’ estimates of AMFs (AMF posterior credible intervals). The countermeasures are then compared and recommended based on the largest safety returns with minimum risk (uncertainty). To the author's knowledge the complete methodology is new and has not previously been applied or reported in the literature. The results demonstrate that the methodology is able to discern anticipated safety benefit differences across candidate countermeasures. For the 18 at grade railroad crossings considered in this analysis, it was found that the top three performing countermeasures for reducing crashes are in-vehicle warning systems, obstacle detection systems, and constant warning time systems.

Peak power demand reduction under moving block signalling using an expert system

The concept of moving block signallings (MBS) has been adopted in a few mass transit railway systems. When a dense queue of trains begins to move from a complete stop, the trains can re-start in very close succession under MBS. The feeding substations nearby are likely to be overloaded and the service will inevitably be disturbed unless substations of higher power rating are used. By introducing starting time delays among the trains or limiting the trains’ acceleration rate to a certain extent, the peak energy demand can be contained. However, delay is introduced and quality of service is degraded. An expert system approach is presented to provide a supervisory tool for the operators. As the knowledge base is vital for the quality of decisions to be made, the study focuses on its formulation with a balance between delay and peak power demand.

Restarting trains under moving block signalling : an expert system approach

A high peak power demand at substations will result under Moving Block Signalling (MBS) when a dense queue of trains begins to start from a complete stop at the same time in an electrified railway system. This may cause the power supply interruption and in turn affect the train service substantially. In a recent study, measures of Starting Time Delay (STD) and Acceleration Rate Limit (ARL) are the possible approaches to reduce the peak power demand on the supply system under MBS. Nevertheless, there is no well-defined relationship between the two measures and peak power demand reduction (PDR). In order to attain a lower peak demand at substations on different traffic conditions and system requirements, an expert system is one of the possible approaches to procure the appropriate use of peak demand reduction measures. The main objective of this paper is to study the effect of the train re-starting strategies on the power demand at substations and the time delay suffered by the trains with the aid of computer simulation. An expert system is a useful tool to select various adoptions of STD and ARL under different operational conditions and system requirements.

Attractiveness of venture idea amongst expert entrepreneurs : a conjoint analysis

Principal Topic Venture ideas are at the heart of entrepreneurship (Davidsson, 2004). However, we are yet to learn what factors drive entrepreneurs’ perceptions of the attractiveness of venture ideas, and what the relative importance of these factors are for their decision to pursue an idea. The expected financial gain is one factor that will obviously influence the perceived attractiveness of a venture idea (Shepherd & DeTienne, 2005). In addition, the degree of novelty of venture ideas along one or more dimensions such as new products/services, new method of production, enter into new markets/customer and new method of promotion may affect their attractiveness (Schumpeter, 1934). Further, according to the notion of an individual-opportunity nexus venture ideas are closely associated with certain individual characteristics (relatedness). Shane (2000) empirically identified that individual’s prior knowledge is closely associated with the recognition of venture ideas. Sarasvathy’s (2001; 2008) Effectuation theory proposes a high degree of relatedness between venture ideas and the resource position of the individual. This study examines how entrepreneurs weigh considerations of different forms of novelty and relatedness as well as potential financial gain in assessing the attractiveness of venture ideas. Method I use conjoint analysis to determine how expert entrepreneurs develop preferences for venture ideas which involved with different degrees of novelty, relatedness and potential gain. The conjoint analysis estimates respondents’ preferences in terms of utilities (or part-worth) for each level of novelty, relatedness and potential gain of venture ideas. A sample of 32 expert entrepreneurs who were awarded young entrepreneurship awards were selected for the study. Each respondent was interviewed providing with 32 scenarios which explicate different combinations of possible profiles open them into consideration. Results and Implications Results indicate that while the respondents do not prefer mere imitation they receive higher utility for low to medium degree of newness suggesting that high degrees of newness are fraught with greater risk and/or greater resource needs. Respondents pay considerable weight on alignment with the knowledge and skills they already posses in choosing particular venture idea. The initial resource position of entrepreneurs is not equally important. Even though expected potential financial gain gives substantial utility, result indicate that it is not a dominant factor for the attractiveness of venture idea.

Expertise in software design : novice and expert models

This paper investigates how software designers use their knowledge during the design process. The research is based on the analysis of the observational and verbal data from three software design teams generated during the conceptual stage of the design process. The knowledge captured from the analysis of the mapped design team data is utilized to generate descriptive models of novice and expert designers. These models contribute to a better understanding of the connections between, and integration of, designer variables, and to a better understanding of software design expertise and its development. The models are transferable to other domains.

Combining expert opinions in prior elicitation

A method of eliciting prior distributions for Bayesian models using expert knowledge is proposed. Elicitation is a widely studied problem, from a psychological perspective as well as from a statistical perspective. Here, we are interested in combining opinions from more than one expert using an explicitly model-based approach so that we may account for various sources of variation affecting elicited expert opinions. We use a hierarchical model to achieve this. We apply this approach to two problems. The first problem involves a food risk assessment problem involving modelling dose-response for Listeria monocytogenes contamination of mice. The second concerns the time taken by PhD students to submit their thesis in a particular school.

Elicitator : a user-friendly, interactive tool to support scenario-based elicitation of expert knowledge

Expert elicitation is the process of determining what expert knowledge is relevant to support a quantitative analysis and then eliciting this information in a form that supports analysis or decision-making. The credibility of the overall analysis, therefore, relies on the credibility of the elicited knowledge. This, in turn, is determined by the rigor of the design and execution of the elicitation methodology, as well as by its clear communication to ensure transparency and repeatability. It is difficult to establish rigor when the elicitation methods are not documented, as often occurs in ecological research. In this chapter, we describe software that can be combined with a well-structured elicitation process to improve the rigor of expert elicitation and documentation of the results

Focusing on patients : differences between the practice of expert and non-expert nephrology nurses

Specialisation in nursing enables a nurse to focus, in much greater depth, on the requisite knowledge and skills for providing patients with the best possible care. Nephrology nursing is one such area where specialisation has evolved. The characteristic focus of practice emerged as an important feature during a study into the process of expertise acquisition in nephrology nursing practice. Using grounded theory methodology, this study involved 6 non-expert and 11 expert nurses and took place in one renal unit in New South Wales. Nephrology nursing practice was observed for 103 hours, and this was immediately followed by semi-structured interviews. The characteristic of focus was conceptualised as the nurses' centre of attention or concentration while they were undertaking nursing activities. Focus ranged from inexperienced non-expert nurses concentrating predominantly on the immediate task at hand, experienced non-expert nurses who focussed on the medium term to expert nurses who viewed actions (and their possible consequences) more broadly and in the longer term. Of significance to nursing, is how nephrology nurses alter their focus of practice as they acquire and exercise their developing expertise in this specialty.

Understanding the role of knowledge in the practice of expert nephrology nurses in Australia

This paper, which is abstracted from a larger study into the acquisition and exercise of nephrology nursing expertise, aims to explore the role of knowledge in expert practice. Using grounded theory methodology, the study involved 17 registered nurses who were practicing in a metropolitan renal unit in New South Wales, Australia. Concurrent data collection and analysis was undertaken, incorporating participants' observations and interviews. Having extensive nephrology nursing knowledge was a striking characteristic of a nursing expert. Expert nurses clearly relied on and utilized extensive nephrology nursing knowledge to practice. Of importance for nursing, the results of this study indicate that domain-specific knowledge is a crucial feature of expert practice.

Uncovering the evidence of non-expert nephrology nursing practice

Expertise in nursing has been widely studied although there have been no previous studies into what constitutes expertise in nephrology (renal) nursing. This paper, which is abstracted from a larger study into the acquisition and exercise of nephrology nursing expertise, provides evidence of the characteristics and practices of non-expert nephrology nurses. Using the grounded theory method, the study took place in one renal unit in New South Wales, Australia, and involved six non-expert and 11 expert nurses. Sampling was purposive then theoretical. Simultaneous data collection and analysis using participant observation, review of nursing documentation and semistructured interviews was undertaken. The study revealed a three-stage skills-acquisitive process that was identified as non-expert, experienced non-expert and expert stages. Non-expert nurses showed superficial nephrology nursing knowledge and limited experience; they were acquiring basic nephrology nursing skills and possessed a narrow focus of practice.

Multitask learning with expert advice

We consider the problem of prediction with expert advice in the setting where a forecaster is presented with several online prediction tasks. Instead of competing against the best expert separately on each task, we assume the tasks are related, and thus we expect that a few experts will perform well on the entire set of tasks. That is, our forecaster would like, on each task, to compete against the best expert chosen from a small set of experts. While we describe the “ideal” algorithm and its performance bound, we show that the computation required for this algorithm is as hard as computation of a matrix permanent. We present an efficient algorithm based on mixing priors, and prove a bound that is nearly as good for the sequential task presentation case. We also consider a harder case where the task may change arbitrarily from round to round, and we develop an efficient approximate randomized algorithm based on Markov chain Monte Carlo techniques.