Combining expert opinions in prior elicitation

A method of eliciting prior distributions for Bayesian models using expert knowledge is proposed. Elicitation is a widely studied problem, from a psychological perspective as well as from a statistical perspective. Here, we are interested in combining opinions from more than one expert using an explicitly model-based approach so that we may account for various sources of variation affecting elicited expert opinions. We use a hierarchical model to achieve this. We apply this approach to two problems. The first problem involves a food risk assessment problem involving modelling dose-response for Listeria monocytogenes contamination of mice. The second concerns the time taken by PhD students to submit their thesis in a particular school.

Combining expert opinions in prior elicitation

We consider the problem of combining opinions from different experts in an explicitly model-based way to construct a valid subjective prior in a Bayesian statistical approach. We propose a generic approach by considering a hierarchical model accounting for various sources of variation as well as accounting for potential dependence between experts. We apply this approach to two problems. The first problem deals with a food risk assessment problem involving modelling dose-response for Listeria monocytogenes contamination of mice. Two hierarchical levels of variation are considered (between and within experts) with a complex mathematical situation due to the use of an indirect probit regression. The second concerns the time taken by PhD students to submit their thesis in a particular school. It illustrates a complex situation where three hierarchical levels of variation are modelled but with a simpler underlying probability distribution (log-Normal).

Prior and posterior linear pooling for combining expert opinions: Uses and impact on Bayesian Networks

The use of expert knowledge to quantify a Bayesian Network (BN) is necessary when data is not available. This however raises questions regarding how opinions from multiple experts can be used in a BN. Linear pooling is a popular method for combining probability assessments from multiple experts. In particular, Prior Linear Pooling (PrLP), which pools opinions then places them into the BN is a common method. This paper firstly proposes an alternative pooling method, Posterior Linear Pooling (PoLP). This method constructs a BN for each expert, then pools the resulting probabilities at the nodes of interest. Secondly, it investigates the advantages and disadvantages of using these pooling methods to combine the opinions of multiple experts. Finally, the methods are applied to an existing BN, the Wayfinding Bayesian Network Model, to investigate the behaviour of different groups of people and how these different methods may be able to capture such differences. The paper focusses on 6 nodes Human Factors, Environmental Factors, Wayfinding, Communication, Visual Elements of Communication and Navigation Pathway, and three subgroups Gender (female, male),Travel Experience (experienced, inexperienced), and Travel Purpose (business, personal) and finds that different behaviors can indeed be captured by the different methods.

How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabiesPetrogale penicillata

1. Species' distribution modelling relies on adequate data sets to build reliable statistical models with high predictive ability. However, the money spent collecting empirical data might be better spent on management. A less expensive source of species' distribution information is expert opinion. This study evaluates expert knowledge and its source. In particular, we determine whether models built on expert knowledge apply over multiple regions or only within the region where the knowledge was derived. 2. The case study focuses on the distribution of the brush-tailed rock-wallaby Petrogale penicillata in eastern Australia. We brought together from two biogeographically different regions substantial and well-designed field data and knowledge from nine experts. We used a novel elicitation tool within a geographical information system to systematically collect expert opinions. The tool utilized an indirect approach to elicitation, asking experts simpler questions about observable rather than abstract quantities, with measures in place to identify uncertainty and offer feedback. Bayesian analysis was used to combine field data and expert knowledge in each region to determine: (i) how expert opinion affected models based on field data and (ii) how similar expert-informed models were within regions and across regions. 3. The elicitation tool effectively captured the experts' opinions and their uncertainties. Experts were comfortable with the map-based elicitation approach used, especially with graphical feedback. Experts tended to predict lower values of species occurrence compared with field data. 4. Across experts, consensus on effect sizes occurred for several habitat variables. Expert opinion generally influenced predictions from field data. However, south-east Queensland and north-east New South Wales experts had different opinions on the influence of elevation and geology, with these differences attributable to geological differences between these regions. 5. Synthesis and applications. When formulated as priors in Bayesian analysis, expert opinion is useful for modifying or strengthening patterns exhibited by empirical data sets that are limited in size or scope. Nevertheless, the ability of an expert to extrapolate beyond their region of knowledge may be poor. Hence there is significant merit in obtaining information from local experts when compiling species' distribution models across several regions.

Expert elicitation and its interface with technology : a review with a view to designing Elicitator

Expert knowledge is valuable in many modelling endeavours, particularly where data is not extensive or sufficiently robust. In Bayesian statistics, expert opinion may be formulated as informative priors, to provide an honest reflection of the current state of knowledge, before updating this with new information. Technology is increasingly being exploited to help support the process of eliciting such information. This paper reviews the benefits that have been gained from utilizing technology in this way. These benefits can be structured within a six-step elicitation design framework proposed recently (Low Choy et al., 2009). We assume that the purpose of elicitation is to formulate a Bayesian statistical prior, either to provide a standalone expert-defined model, or for updating new data within a Bayesian analysis. We also assume that the model has been pre-specified before selecting the software. In this case, technology has the most to offer to: targeting what experts know (E2), eliciting and encoding expert opinions (E4), whilst enhancing accuracy (E5), and providing an effective and efficient protocol (E6). Benefits include: -providing an environment with familiar nuances (to make the expert comfortable) where experts can explore their knowledge from various perspectives (E2); -automating tedious or repetitive tasks, thereby minimizing calculation errors, as well as encouraging interaction between elicitors and experts (E5); -cognitive gains by educating users, enabling instant feedback (E2, E4-E5), and providing alternative methods of communicating assessments and feedback information, since experts think and learn differently; and -ensuring a repeatable and transparent protocol is used (E6).

Bayesian methodology incorporating expert judgment for ranking countermeasure effectiveness under uncertainty: Example applied to at grade railroad crossings in Korea

This paper describes the formalization and application of a methodology to evaluate the safety benefit of countermeasures in the face of uncertainty. To illustrate the methodology, 18 countermeasures for improving safety of at grade railroad crossings (AGRXs) in the Republic of Korea are considered. Akin to “stated preference” methods in travel survey research, the methodology applies random selection and laws of large numbers to derive accident modification factor (AMF) densities from expert opinions. In a full Bayesian analysis framework, the collective opinions in the form of AMF densities (data likelihood) are combined with prior knowledge (AMF density priors) for the 18 countermeasures to obtain ‘best’ estimates of AMFs (AMF posterior credible intervals). The countermeasures are then compared and recommended based on the largest safety returns with minimum risk (uncertainty). To the author's knowledge the complete methodology is new and has not previously been applied or reported in the literature. The results demonstrate that the methodology is able to discern anticipated safety benefit differences across candidate countermeasures. For the 18 at grade railroad crossings considered in this analysis, it was found that the top three performing countermeasures for reducing crashes are in-vehicle warning systems, obstacle detection systems, and constant warning time systems.

Combining opinions for use in Bayesian networks: A measurement error approach

Bayesian networks (BNs) are graphical probabilistic models used for reasoning under uncertainty. These models are becoming increasing popular in a range of fields including ecology, computational biology, medical diagnosis, and forensics. In most of these cases, the BNs are quantified using information from experts, or from user opinions. An interest therefore lies in the way in which multiple opinions can be represented and used in a BN. This paper proposes the use of a measurement error model to combine opinions for use in the quantification of a BN. The multiple opinions are treated as a realisation of measurement error and the model uses the posterior probabilities ascribed to each node in the BN which are computed from the prior information given by each expert. The proposed model addresses the issues associated with current methods of combining opinions such as the absence of a coherent probability model, the lack of the conditional independence structure of the BN being maintained, and the provision of only a point estimate for the consensus. The proposed model is applied an existing Bayesian Network and performed well when compared to existing methods of combining opinions.

Towards a rule-based matrix for evaluating distress mechanisms in bridges

The effective management of bridge stock involves making decisions as to when to repair, remedy, or do nothing, taking into account the financial and service life implications. Such decisions require a reliable diagnosis as to the cause of distress and an understanding of the likely future degradation. Such diagnoses are based on a combination of visual inspections, laboratory tests on samples and expert opinions. In addition, the choice of appropriate laboratory tests requires an understanding of the degradation mechanisms involved. Under these circumstances, the use of expert systems or evaluation tools developed from “realtime” case studies provides a promising solution in the absence of expert knowledge. This paper addresses the issues in bridge infrastructure management in Queensland, Australia. Bridges affected by alkali silica reaction and chloride induced corrosion have been investigated and the results presented using a mind mapping tool. The analysis highights that several levels of rules are required to assess the mechanism causing distress. The systematic development of a rule based approach is presented. An example of this application to a case study bridge has been used to demonstrate that preliminary results are satisfactory.

Throwing frame and performance of elite male seated shot-putters

Seated shot-putters rely on a customized assistive device called a throwing frame. Currently, the construction of each individual throwing frame is mainly driven by an empirical approach. One way to refine the conception is to improve the basic understanding of performance of seated shot-putters. The relationship between performance and throwing technique has been well described. Remarkably, the relationship between performance and throwing frame characteristics has received limited attention. The primary objective of this study was to present a cataloguing of characteristics of throwing frames used by seated shot-putters. This cataloguing consisted of defining and grouping 26 characteristics into three main categories (i.e., whole body, foot and upper limb specific characteristics) and seven sub-categories. The secondary objective of this study was to provide raw characterisations of the throwing frames for a group of athletes who participated in a world-class event. The characterisation consisted of describing the characteristics of each throwing frame. Potential relationships between characteristics, performance and classification were also identified. The cataloguing was achieved using a 6-step heuristic approach, involving expert opinions and the analysis of 215 attempts produced by 55 male athletes during the 2006 IPC Athletics World Championships. The distribution of samples across characteristics suggested a relevant level of comprehensiveness for the proposed cataloguing. The raw data, the profile of best athletes and the frequency of characteristics provided key benchmark information for construction of a throwing frame as well as coaching, classification and officiating. Analysis of data sets relating to characteristics, performance and classification were inconclusive.

Peer-led diabetes self-management programme for community-dwelling older people in China : study protocol for a quasi-experimental design

Aim. A protocol for a new peer-led self-management programme for communitydwelling older people with diabetes in Shanghai, China. Background. The increasing prevalence of type 2 diabetes poses major public health challenges. Appropriate education programmes could help people with diabetes to achieve self-management and better health outcomes. Providing education programmes to the fast growing number of people with diabetes present a real challenge to Chinese healthcare system, which is strained for personnel and funding shortages. Empirical literature and expert opinions suggest that peer education programmes are promising. Design. Quasi-experimental. Methods. This study is a non-equivalent control group design (protocol approved in January, 2008). A total of 190 people, with 95 participants in each group, will be recruited from two different, but similar, communities. The programme, based on Social Cognitive Theory, will consist of basic diabetes instruction and social support and self-efficacy enhancing group activities. Basic diabetes instruction sessions will be delivered by health professionals, whereas social support and self-efficacy enhancing group activities will be led by peer leaders. Outcome variables include: self-efficacy, social support, self-management behaviours, depressive status, quality of life and healthcare utilization, which will be measured at baseline, 4 and 12 weeks. Discussion. This theory-based programme tailored to Chinese patients has potential for improving diabetes self-management and subsequent health outcomes. In addition, the delivery mode, through involvement of peer leaders and existing community networks,is especially promising considering healthcare resource shortage in China.

People capabilities for promoting sustainability in facilities management practices

Sustainability needs to be embedded throughout the life-cycle of a construction project. From project conception, planning, design, construction stage, operation and maintenance to demolition, each phase of development should embrace principles of sustainability and the stakeholder involved should be empowered with the necessary skills. Past research explored the importance of ensuring sustainability measures during the occupancy phase based on considerations of Life-Cycle Cost Analysis and a project’s long-term detrimental impact on the environment. Facility managers are in a unique position to promote sustainability over longer periods of project engagement and can apply a high level of influence on the built assets through management and upgrades. There is growing interest among facility managers in incorporating sustainability measures into day-to-day practice. More, however, needs to be done. Previous studies have identified barriers such as the lack of sustainability knowledge and skills, poor access to information, and unwillingness to change among facility management (FM) practitioners and stakeholders. This inhibits proper implementation of sustainable practices in the FM sector. A number of key factors, such as knowledge discrepancy, time constraints, diversity of FM functions and a lack of incentives, require urgent remedy. The capability of FM professionals and stakeholders will be a key enabler in managing the sustainability agenda, as it is central to the improvement of competency and innovation in an organization. Compared to the attempts at developing sustainability guidelines and performance measurement, research efforts relating to people capabilities and skills are still lagging behind. This paper discusses the progress to date of a research project aimed at formulating a people capabilities framework for sustainable FM practices based on expert opinions and industry feedback. Through literature review, the paper explores the challenges of incorporating sustainability principles into general FM practices before focusing specifically on FM personnel capabilities that may impact on the implementation of a holistic sustainability agenda in real life practice. The results of an industry survey are used to propose an action framework to identify, promote and utilise people capabilities in order to promote sustainability integration in FM practices. The paper provides a useful information source for FM personnel and organizations to bridge the gap between extensive tools on sustainable design and construction assessment at the front end and the need to maintain focus throughout the project life-cycle.

Opinions, attitudes and driving experiences of car and racing "enthusiasts" : results of the Ontario Car and Racing Club Survey

In recent years, car club and racing websites and forums have become an increasingly popular way for car enthusiasts to access racing and car club news, chat-rooms and message boards. However, no North American research has been found that has examined opinions and driving experiences of car and racing enthusiasts. The purpose of this study was to examine car club members’ opinions about and experiences with various aspects of driving, road safety and traffic legislation, with a particular focus on street racing. A web-based questionnaire (Survey Monkey) was developed using the expert panel method and was primarily based on validated instruments or questions that were developed from other surveys. The questionnaire included: 1) driver concerns regarding traffic safety issues and legislation; 2) attitudes regarding various driving activities; 3) leisure-time activities, including club activities; 4) driving experiences, including offences and collisions; and 5) socio-demographic questions. The survey was pre- tested and piloted. Electronic information letters were sent out to an identified list of car clubs and forums situated in southern Ontario. Car club participants were invited to fill out the questionnaire. This survey found that members of car clubs share similar concerns regarding various road safety issues with samples of Canadian drivers, although a smaller percentage of car club members are concerned about speeding-related driving. Car club members had varied opinions regarding Ontario’s Street Racers, Stunt and Aggressive Drivers Legislation. The respondents agreed the most with the new offences regarding not sitting in the driver’s seat, having a person in the trunk, or driving as close as possible to another vehicle, pedestrian or object on or near the highway without a reason. The majority disagreed with police powers of impoundment and on-the-spot licence suspensions. About three quarters of respondents reported no collisions or police stops for traffic offences in the past five years. Of those who had been stopped, the most common offence was reported as speeding. This study is the first in Canada to examine car club members’ opinions about and experiences with various aspects of driving, road safety and traffic legislation. Given the ubiquity of car clubs and fora in Canada, insights on members’ opinions and practices provide important information to road safety researchers.

Major issues in business process management : an expert perspective

Study Approach The results presented in this report are part of a larger global study on the major issues in BPM. Only one part of the larger study is reported here, viz. interviews with BPM experts. Interviews of BPM tool vendors together with focus group studies involving user organizations were conducted in parallel and set the groundwork for the identification of BPM issues on a global scale. Through this multi-method approach, we identify four distinct sets of outcomes. First, as is the focus of this report, we identify the BPM issues as perceived by BPM experts. Second, the research design allows us to gain insight into the opinions of organizations deploying BPM solutions. Third, an understanding of organizations’ misconceptions of BPM technologies, as confronted by BPM tool vendors, is obtained. Last, we seek to gain an understanding of BPM issues on a global scale, together with knowledge of matters of concern. This final outcome is aimed to produce an industry-driven research agenda that will inform practitioners and, in particular, the research community worldwide on issues and challenges that are prevalent or emerging in BPM and related areas...

Including or excluding conflicts of interest among expert peer reviewers had little impact on funding success, a case study from Australia

Competition for research funding is intense and the opinions of an expert peer reviewer can mean the difference between success and failure in securing funding. The allocation of expert peer reviewers is therefore vitally important and funding agencies strive to avoid using reviewers who have real or perceived conflicts of interest. This article examines the impact of including or excluding peer reviewers based on their conflicts of interest, and the final ranking of funding proposals. Two 7-person review panels assessed a sample of National Health and Medical Research Council (NHMRC) of Australia proposals in Basic Science or Public Health. Using a pre-post comparison, the proposals were first scored after the exclusion of reviewers with a high or medium conflict, and re-scored after the return of reviewers with medium conflicts. The main outcome measures are the agreements in ranks and funding success before and after excluding the medium conflicts. Including medium conflicts of interest had little impact on the ranks or funding success. The Bland–Altman 95% limits of agreement were ± 3.3 ranks and ± 3.4 ranks in the two panels which both assessed 36 proposals. Overall there were three proposals (4%) that had a reversed funding outcome after including medium conflicts. Relaxing the conflict of interest rules would increase the number of expert reviewers included in the panel discussions which could increase the quality of peer review and make it easier to find reviewers.