Bayesian learning of noisy Markov decision processes

This work addresses the problem of estimating the optimal value function in a Markov Decision Process from observed state-action pairs. We adopt a Bayesian approach to inference, which allows both the model to be estimated and predictions about actions to be made in a unified framework, providing a principled approach to mimicry of a controller on the basis of observed data. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from theposterior distribution over the optimal value function. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Deliberation in the motor system: reflex gains track evolving evidence leading to a decision.

Both decision making and sensorimotor control require real-time processing of noisy information streams. Historically these processes were thought to operate sequentially: cognitive processing leads to a decision, and the outcome is passed to the motor system to be converted into action. Recently, it has been suggested that the decision process may provide a continuous flow of information to the motor system, allowing it to prepare in a graded fashion for the probable outcome. Such continuous flow is supported by electrophysiology in nonhuman primates. Here we provide direct evidence for the continuous flow of an evolving decision variable to the motor system in humans. Subjects viewed a dynamic random dot display and were asked to indicate their decision about direction by moving a handle to one of two targets. We probed the state of the motor system by perturbing the arm at random times during decision formation. Reflex gains were modulated by the strength and duration of motion, reflecting the accumulated evidence in support of the evolving decision. The magnitude and variance of these gains tracked a decision variable that explained the subject's decision accuracy. The findings support a continuous process linking the evolving computations associated with decision making and sensorimotor control.

Partially Observable Markov Decision Processes with continuous observations for dialogue management

This work shows how a dialogue model can be represented as a Partially Observable Markov Decision Process (POMDP) with observations composed of a discrete and continuous component. The continuous component enables the model to directly incorporate a confidence score for automated planning. Using a testbed simulated dialogue management problem, we show how recent optimization techniques are able to find a policy for this continuous POMDP which outperforms a traditional MDP approach. Further, we present a method for automatically improving handcrafted dialogue managers by incorporating POMDP belief state monitoring, including confidence score information. Experiments on the testbed system show significant improvements for several example handcrafted dialogue managers across a range of operating conditions.

Bayesian Learning of Noisy Markov Decision Processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller.

Individual and couple decision behavior under risk: evidence on the dynamics of power balance

This article reports results of an experiment designed to analyze the link between risky decisions made by couples and risky decisions made separately by each spouse. We estimate both the spouses and the couples' degrees of risk aversion, we assess how the risk preferences of the two spouses aggregate when they make risky decisions, and we shed light on the dynamics of the decision process that takes place when couples make risky decisions. We find that, far from being fixed, the balance of power within the household is malleable. In most couples, men have, initially, more decision-making power than women but women who ultimately implement the joint decisions gain more and more power over the course of decision making.

The psychophysics of decision making in a two-direction random dot motion target selection task

La tâche de kinématogramme de points aléatoires est utilisée avec le paradigme de choix forcé entre deux alternatives pour étudier les prises de décisions perceptuelles. Les modèles décisionnels supposent que les indices de mouvement pour les deux alternatives sont encodés dans le cerveau. Ainsi, la différence entre ces deux signaux est accumulée jusqu’à un seuil décisionnel. Cependant, aucune étude à ce jour n’a testé cette hypothèse avec des stimuli contenant des mouvements opposés. Ce mémoire présente les résultats de deux expériences utilisant deux nouveaux stimuli avec des indices de mouvement concurrentiels. Parmi une variété de combinaisons d’indices concurrentiels, la performance des sujets dépend de la différence nette entre les deux signaux opposés. De plus, les sujets obtiennent une performance similaire avec les deux types de stimuli. Ces résultats supportent un modèle décisionnel basé sur l’accumulation des indices de mouvement net et suggèrent que le processus décisionnel peut intégrer les signaux de mouvement à partir d’une grande gamme de directions pour obtenir un percept global de mouvement.

Contact Sensing: A Sequential Decision Approach to Sensing Manipulation Contact

This paper describes a new statistical, model-based approach to building a contact state observer. The observer uses measurements of the contact force and position, and prior information about the task encoded in a graph, to determine the current location of the robot in the task configuration space. Each node represents what the measurements will look like in a small region of configuration space by storing a predictive, statistical, measurement model. This approach assumes that the measurements are statistically block independent conditioned on knowledge of the model, which is a fairly good model of the actual process. Arcs in the graph represent possible transitions between models. Beam Viterbi search is used to match measurement history against possible paths through the model graph in order to estimate the most likely path for the robot. The resulting approach provides a new decision process that can be use as an observer for event driven manipulation programming. The decision procedure is significantly more robust than simple threshold decisions because the measurement history is used to make decisions. The approach can be used to enhance the capabilities of autonomous assembly machines and in quality control applications.

One rapidly growing market of Sweden: organic food : A qualitative study portraying some consumers’ decision to purchase organic food

In recent years there has been a significantly rising trend amongst consumers for health and environmental issues, which has resulted in greater attention for and sales of organic food. Organic food production strives to use natural resources, such as land, energy and water in a sustainable way and the products does not contain artificial fertilizers or chemical pesticides. However, organic food products are also often more expensive and less available in comparison to conventional food products. Despite this, interest for and sales of organic food products have increased around the globe, and in Sweden particularly, the increase in sales has grown largely from an international perspective. This thesis is of qualitative character and is focused on studying some consumers from the Swedish market of organic food. The purpose of this thesis is to contribute with a better understanding on the buying decision process regarding organic food purchase. To achieve this, the authors have studied some consumers that purchase organic food and have searched for patterns that could be identified in the decision process. The consumer buying decision process model has been used for portrayal of the chosen consumers’ decision to purchase organic food products. Interviews with six Swedish consumers were conducted, whereas each respondent continuously purchase organic food products. Results show that the purchase of organic food products begins with discovering an unsatisfied need state, which the consumers of this study desired to change with the purchase of organic food products. This study show how six consumers reason when passing through the stages of the buying decision process, in order to satisfy their desired need state. The authors found that the respondents feel satisfied with purchasing organic food products, which lead them in to continuously buying these products. Altogether, the findings contribute with knowledge that can come to be helpful when wanting to understand more about the consumer decision to purchase organic food.

Using Normative Markov Decision Processes for Evaluating Electronic Contracts: A Case Study in a Simulated Aerospace Aftermarket

Before signing electronic contracts, a rational agent should estimate the expected utilities of these contracts and calculate the violation risks related to them. In order to perform such pre-signing procedures, this agent has to be capable of computing a policy taking into account the norms and sanctions in the contracts. In relation to this, the contribution of this work is threefold. First, we present the Normative Markov Decision Process, an extension of the Markov Decision Process for explicitly representing norms. In order to illustrate the usage of our framework, we model an example in a simulated aerospace aftermarket. Second, we specify an algorithm for identifying the states of the process which characterize the violation of norms. Finally, we show how to compute policies with our framework and how to calculate the risk of violating the norms in the contracts by adopting a particular policy.

Factors influencing consumer decision in crowdfunding

Crowdfunding é um método recente e emergente de captar dinheiro para desenvolvimento de projetos (tanto orientados a lucro ou não) sem a intermediação tradicional de instituições financeiras, liberando empreendedores de custos, regulações e burocracia associada a essa prática. Além disso, também é um método de pré-testar novos produtos com um público selecionado e entusiasmado. O objetivo dessa dissertação é entender que fatores estão influenciando a decisão do consumidor de investir em projetos. A literatura contribui com: (1) fatores intrínsecos, como desejo de patronagem; (2) fatores extrínsecos, como a apresentação do projeto; e (3) pressão social. Há ainda fatores associados com o nível atual de captação e número de investidores, assim como tipo de projeto envolvido, sendo ele de caridade ou não. Além disso, atitudes também possuem um papel em afetar a decisão de compra. Para responder a pergunta de pesquisa, uma metodologia de duas fases foi usada: uma entrevista de profundidade para capturar intenção de investir e motivação, de forma a construir um processo de decisão que englobasse todas as possibilidades descritas pela literatura. Após essa pesquisa qualitativa, uma pesquisa quantitativa foi feita para validar as informações coletadas pela fase anterior e coletar dados adicionais para gerar uma associação entre intenção de investir e comportamento. Dentre as informações geradas pela fase qualitativa, temos o fato que a maioria dos investidores tiveram como principal motivação a compra do produto sendo oferecido como se eles estivessem participando de uma pré-venda. Entretanto, essa não foi a principal razão para o investidor de caridade. Além disso, os respondentes que pré-compraram os produtos o fizeram para única razão que esses produtos satisfizeram desejos que tinham. Esses desejos variavam, sendo desde saudade de jogos antigos como resolver um problema de organização da carteira. Outra característica da pré-compra foi que eles não investiam valores simbólicos, pela razão que se o fizessem não receberiam o produto em troca. Recompensas tiveram um grande papel em atrair os respondentes para investimento em valores maiores que consideravam anteriormente. Também é verdade para o investidor em caridade, que também doou mais. A fase quantitativa confirmou as informações acima e gerou informação extra sobre as categorias de produto. Projetos de caridade e arte concentraram a maioria dos respondentes que disseram que a principal razão para investir foi basicamente ajudar a desenvolver o projeto sem demandar um produto em retorno. Entretanto, outros projetos como Música também apresentaram altos números de comportamento caridoso, possivelmente por causa do envolvimento emocional com o artista. Outras categorias apresentaram um mix de razões para investir ou enviesado a comprar o produto apenas, o que pode ser explicado pelo efeito de recompensas e pelo fato que essas categorias estão simplesmente pré-vendendo produtos. Essa pesquisa também confirmou as principais fontes usadas para conhecer mais sobre os projetos: recomendação pessoal e blogs e fóruns. Outro resultado dessa fase foi o desenvolvimento de fatores a partir de frases atitudinais que puderam explicar intenção de investir. Seis fatores foram criados: Entusiasmo (por crowdfunding), Exclusividade (compra de recompensas), Caridade (doações pequenas para ajudar o desenvolvimento do projeto), Cautela (similar à difusão de responsabilidade, isto é, espera por mais investidores para dar o primeiro passo), Intimidade (projeto foi recomendado ou há ligação emocional com o criador) e Compartilhamento (compartilhar para ajudar a trazer mais investidores para o projeto). Categorias com alto envolvimento emocional apresentaram associação com Intimidade, como música, filme e tecnologia. Dado o fato que a amostra não continha muitos entusiastas por crowdfunding, esse fator não apresentou qualquer associação com as categorias. Categorias que não entregam produtos em troca, como comida e fotografia, apresentaram altos níveis de associação com o fator caridade. Compartilhamento é altamente associado com tecnologia, dado o fato que essa categoria concentra os respondentes que são mais orientados à inovação e entusiastas sobre o produto, então precisam compartilhar e gerar boca-a-boca para ajudar a atingir a meta de investimento.

Giving According to Preferences: Decision-Making in the Group Dictator Game

We study the decision process in a group dictator game in which three subjects can distribute an initial endowment between themselves and a group of recipients. The experiment consists of two stages; first, individuals play a standard dictator game. Second, individuals are randomly matched into groups of three and communicate via instant messaging regarding the decision in the group dictator game. In contrast to former studies our results show that group decisions do not differ from individual decisions in the dictator game. Furthermore, the analysis of the chat history reveals that players make proposals according to their preferences as revealed in the single dictator game and that these proposals in groups drive the final allocation.

Decision strategies for policy decisions under uncertainties: The case of mitigation measures addressing methane emissions from ruminants

Decision strategies aim at enabling reasonable decisions in cases of uncertain policy decision problems which do not meet the conditions for applying standard decision theory. This paper focuses on decision strategies that account for uncertainties by deciding whether a proposed list of policy options should be accepted or revised (scope strategies) and whether to decide now or later (timing strategies). They can be used in participatory approaches to structure the decision process. As a basis, we propose to classify the broad range of uncertainties affecting policy decision problems along two dimensions, source of uncertainty (incomplete information, inherent indeterminacy and unreliable information) and location of uncertainty (information about policy options, outcomes and values). Decision strategies encompass multiple and vague criteria to be deliberated in application. As an example, we discuss which decision strategies may account for the uncertainties related to nutritive technologies that aim at reducing methane (CH4) emissions from ruminants as a means of mitigating climate change, limiting our discussion to published scientific information. These considerations not only speak in favour of revising rather than accepting the discussed list of options, but also in favour of active postponement or semi-closure of decision-making rather than closure or passive postponement.

Analysing the complexity of the model-based decision making processes within the industrial management context

PURPOSE The decision-making process plays a key role in organizations. Every decision-making process produces a final choice that may or may not prompt action. Recurrently, decision makers find themselves in the dichotomous question of following a traditional sequence decision-making process where the output of a decision is used as the input of the next stage of the decision, or following a joint decision-making approach where several decisions are taken simultaneously. The implication of the decision-making process will impact different players of the organization. The choice of the decision- making approach becomes difficult to find, even with the current literature and practitioners’ knowledge. The pursuit of better ways for making decisions has been a common goal for academics and practitioners. Management scientists use different techniques and approaches to improve different types of decisions. The purpose of this decision is to use the available resources as well as possible (data and techniques) to achieve the objectives of the organization. The developing and applying of models and concepts may be helpful to solve managerial problems faced every day in different companies. As a result of this research different decision models are presented to contribute to the body of knowledge of management science. The first models are focused on the manufacturing industry and the second part of the models on the health care industry. Despite these models being case specific, they serve the purpose of exemplifying that different approaches to the problems and could provide interesting results. Unfortunately, there is no universal recipe that could be applied to all the problems. Furthermore, the same model could deliver good results with certain data and bad results for other data. A framework to analyse the data before selecting the model to be used is presented and tested in the models developed to exemplify the ideas. METHODOLOGY As the first step of the research a systematic literature review on the joint decision is presented, as are the different opinions and suggestions of different scholars. For the next stage of the thesis, the decision-making process of more than 50 companies was analysed in companies from different sectors in the production planning area at the Job Shop level. The data was obtained using surveys and face-to-face interviews. The following part of the research into the decision-making process was held in two application fields that are highly relevant for our society; manufacturing and health care. The first step was to study the interactions and develop a mathematical model for the replenishment of the car assembly where the problem of “Vehicle routing problem and Inventory” were combined. The next step was to add the scheduling or car production (car sequencing) decision and use some metaheuristics such as ant colony and genetic algorithms to measure if the behaviour is kept up with different case size problems. A similar approach is presented in a production of semiconductors and aviation parts, where a hoist has to change from one station to another to deal with the work, and a jobs schedule has to be done. However, for this problem simulation was used for experimentation. In parallel, the scheduling of operating rooms was studied. Surgeries were allocated to surgeons and the scheduling of operating rooms was analysed. The first part of the research was done in a Teaching hospital, and for the second part the interaction of uncertainty was added. Once the previous problem had been analysed a general framework to characterize the instance was built. In the final chapter a general conclusion is presented. FINDINGS AND PRACTICAL IMPLICATIONS The first part of the contributions is an update of the decision-making literature review. Also an analysis of the possible savings resulting from a change in the decision process is made. Then, the results of the survey, which present a lack of consistency between what the managers believe and the reality of the integration of their decisions. In the next stage of the thesis, a contribution to the body of knowledge of the operation research, with the joint solution of the replenishment, sequencing and inventory problem in the assembly line is made, together with a parallel work with the operating rooms scheduling where different solutions approaches are presented. In addition to the contribution of the solving methods, with the use of different techniques, the main contribution is the framework that is proposed to pre-evaluate the problem before thinking of the techniques to solve it. However, there is no straightforward answer as to whether it is better to have joint or sequential solutions. Following the proposed framework with the evaluation of factors such as the flexibility of the answer, the number of actors, and the tightness of the data, give us important hints as to the most suitable direction to take to tackle the problem. RESEARCH LIMITATIONS AND AVENUES FOR FUTURE RESEARCH In the first part of the work it was really complicated to calculate the possible savings of different projects, since in many papers these quantities are not reported or the impact is based on non-quantifiable benefits. The other issue is the confidentiality of many projects where the data cannot be presented. For the car assembly line problem more computational power would allow us to solve bigger instances. For the operation research problem there was a lack of historical data to perform a parallel analysis in the teaching hospital. In order to keep testing the decision framework it is necessary to keep applying more case studies in order to generalize the results and make them more evident and less ambiguous. The health care field offers great opportunities since despite the recent awareness of the need to improve the decision-making process there are many opportunities to improve. Another big difference with the automotive industry is that the last improvements are not spread among all the actors. Therefore, in the future this research will focus more on the collaboration between academia and the health care sector.

Intelligent decision support for family financial planning

In today’s financial markets characterized by constantly changing tax laws and increasingly complex transactions, the demand for family financial planning (FFP) services is rising dramatically. However, the current trend to develop advisory systems that focus mainly on the financial or investment side fails to consider the whole picture of FFP. Separating financial or investment advice from legal and accounting advice may result in conflicting advice or important omissions that could lead to users suffering financial loss. In this paper, we propose a conceptual model for FFP decision-making process, followed by a novel architecture to support an aggregated FFP decision process by utilizing intelligentagents and Web-services technology. A prototype system for supporting FFP decision is presented to demonstrate the advances of the proposed Web-service multi-agentsbased system architecture and business value.

Decision support system for risk management:a case study

This study demonstrates a quantitative approach to construction risk management through analytic hierarchy process and decision tree analysis. All the risk factors are identified, their effects are quantified by determining probability and severity, and various alternative responses are generated with cost implication for mitigating the quantified risks. The expected monetary values are then derived for each alternative in a decision tree framework and subsequent probability analysis aids the decision process in managing risks. The entire methodology is explained through a case application of a cross-country petroleum pipeline project in India and its effectiveness in project management is demonstrated.