A unified framework for linear function approximation of value functions in stochastic control

This paper contributes with a unified formulation that merges previ- ous analysis on the prediction of the performance ( value function ) of certain sequence of actions ( policy ) when an agent operates a Markov decision process with large state-space. When the states are represented by features and the value function is linearly approxi- mated, our analysis reveals a new relationship between two common cost functions used to obtain the optimal approximation. In addition, this analysis allows us to propose an efficient adaptive algorithm that provides an unbiased linear estimate. The performance of the pro- posed algorithm is illustrated by simulation, showing competitive results when compared with the state-of-the-art solutions.

A new dominance intensity method to deal with ordinal information about a DM's preferences within MAVT

Dominance measuring methods are a new approach to deal with complex decision-making problems with imprecise information. These methods are based on the computation of pairwise dominance values and exploit the information in the dominance matrix in dirent ways to derive measures of dominance intensity and rank the alternatives under consideration. In this paper we propose a new dominance measuring method to deal with ordinal information about decision-maker preferences in both weights and component utilities. It takes advantage of the centroid of the polytope delimited by ordinal information and builds triangular fuzzy numbers whose distances to the crisp value 0 constitute the basis for the de?nition of a dominance intensity measure. Monte Carlo simulation techniques have been used to compare the performance of this method with other existing approaches.

Dominance intensity methods for ranking of alternatives in mcdm with imprecise information

Los decisores cada vez se enfrentan a problemas más complejos en los que tomar una decisión implica tener que considerar simultáneamente muchos criterios que normalmente son conflictivos entre sí. En la mayoría de los problemas de decisión es necesario considerar criterios económicos, sociales y medioambientales. La Teoría de la Decisión proporciona el marco adecuado para poder ayudar a los decisores a resolver estos problemas de decisión complejos, al permitir considerar conjuntamente la incertidumbre existente sobre las consecuencias de cada alternativa en los diferentes atributos y la imprecisión sobre las preferencias de los decisores. En esta tesis doctoral nos centramos en la imprecisión de las preferencias de los decisores cuando éstas pueden ser representadas mediante una función de utilidad multiatributo aditiva. Por lo tanto, consideramos imprecisión tanto en los pesos como en las funciones de utilidad componentes de cada atributo. Se ha considerado el caso en que la imprecisión puede ser representada por intervalos de valores o bien mediante información ordinal, en lugar de proporcionar valores concretos. En este sentido, hemos propuesto métodos que permiten ordenar las diferentes alternativas basados en los conceptos de intensidad de dominación o intensidad de preferencia, los cuales intentan medir la fuerza con la que cada alternativa es preferida al resto. Para todos los métodos propuestos se ha analizado su comportamiento y se ha comparado con los más relevantes existentes en la literatura científica que pueden ser aplicados para resolver este tipo de problemas. Para ello, se ha realizado un estudio de simulación en el que se han usado dos medidas de eficiencia (hit ratio y coeficiente de correlación de Kendall) para comparar los diferentes métodos. ABSTRACT Decision makers increasingly face complex decision-making problems where they have to simultaneously consider many often conflicting criteria. In most decision-making problems it is necessary to consider economic, social and environmental criteria. Decision making theory provides an adequate framework for helping decision makers to make complex decisions where they can jointly consider the uncertainty about the performance of each alternative for each attribute, and the imprecision of the decision maker's preferences. In this PhD thesis we focus on the imprecision of the decision maker's preferences represented by an additive multiattribute utility function. Therefore, we consider the imprecision of weights, as well as of component utility functions for each attribute. We consider the case in which the imprecision is represented by ranges of values or by ordinal information rather than precise values. In this respect, we propose methods for ranking alternatives based on notions of dominance intensity, also known as preference intensity, which attempt to measure how much more preferred each alternative is to the others. The performance of the propose methods has been analyzed and compared against the leading existing methods that are applicable to this type of problem. For this purpose, we conducted a simulation study using two efficiency measures (hit ratio and Kendall correlation coefficient) to compare the different methods.

Student academic performance stochastic simulator based on the Monte Carlo method

In this paper, a computer-based tool is developed to analyze student performance along a given curriculum. The proposed software makes use of historical data to compute passing/failing probabilities and simulates future student academic performance based on stochastic programming methods (MonteCarlo) according to the specific university regulations. This allows to compute the academic performance rates for the specific subjects of the curriculum for each semester, as well as the overall rates (the set of subjects in the semester), which are the efficiency rate and the success rate. Additionally, we compute the rates for the Bachelors degree, which are the graduation rate measured as the percentage of students who finish as scheduled or taking an extra year and the efficiency rate (measured as the percentage of credits of the curriculum with respect to the credits really taken). In Spain, these metrics have been defined by the National Quality Evaluation and Accreditation Agency (ANECA). Moreover, the sensitivity of the performance metrics to some of the parameters of the simulator is analyzed using statistical tools (Design of Experiments). The simulator has been adapted to the curriculum characteristics of the Bachelor in Engineering Technologies at the Technical University of Madrid(UPM).

Quality-optimization algorithm based on stochastic dynamic programming for MPEG DASH video streaming

In contrast to traditional push-based protocols, adaptive streaming techniques like Dynamic Adaptive Streaming over HTTP (DASH) fix attention on the client, who dynamically requests different-quality portions of the content to cope with a limited and variable bandwidth but aiming at maximizing the quality perceived by the user. Since DASH adaptation logic at the client is not covered by the standard, we propose a solution based on Stochastic Dynamic Programming (SDP) techniques to find the optimal request policies that guarantee the users' Quality of Experience (QoE). Our algorithm is evaluated in a simulated streaming session and is compared with other adaptation approaches. The results show that our proposal outperforms them in terms of QoE, requesting higher qualities on average.

Dominance measuring methods within MAVT/MAUT with imprecise information concerning decision-makers'preferences

Dominance measuring methods are an approach for dealing with complex decision-making problems with imprecise information within multi-attribute value/utility theory. These methods are based on the computation of pairwise dominance values and exploit the information in the dominance matrix in different ways to derive measures of dominance intensity and rank the alternatives under consideration. In this paper we review dominance measuring methods proposed in the literature for dealing with imprecise information (intervals, ordinal information or fuzzy numbers) about decision-makers? preferences and their performance in comparison with other existing approaches, like SMAA and SMAA-II or Sarabando and Dias? method.

Veto values in Group Decision Making within MAUT: aggregating complete rankings derived from dominance intensity measures

We consider a groupdecision-making problem within multi-attribute utility theory, in which the relative importance of decisionmakers (DMs) is known and their preferences are represented by means of an additive function. We allow DMs to provide veto values for the attribute under consideration and build veto and adjust functions that are incorporated into the additive model. Veto functions check whether alternative performances are within the respective veto intervals, making the overall utility of the alternative equal to 0, where as adjust functions reduce the utilty of the alternative performance to match the preferences of other DMs. Dominance measuring methods are used to account for imprecise information in the decision-making scenario and to derive a ranking of alternatives for each DM. Specifically, ordinal information about the relative importance of criteria is provided by each DM. Finally, an extension of Kemeny's method is used to aggregate the alternative rankings from the DMs accounting for the irrelative importance.

Optimization methods for the operating room management under uncertainty: stochastic programming vs. decomposition approach

The operating theatres are the engine of the hospitals; proper management of the operating rooms and its staff represents a great challenge for managers and its results impact directly in the budget of the hospital. This work presents a MILP model for the efficient schedule of multiple surgeries in Operating Rooms (ORs) during a working day. This model considers multiple surgeons and ORs and different types of surgeries. Stochastic strategies are also implemented for taking into account the uncertain in surgery durations (pre-incision, incision, post-incision times). In addition, a heuristic-based methods and a MILP decomposition approach is proposed for solving large-scale ORs scheduling problems in computational efficient way. All these computer-aided strategies has been implemented in AIMMS, as an advanced modeling and optimization software, developing a user friendly solution tool for the operating room management under uncertainty.

A stochastic approximation algorithm with Markov chain Monte-Carlo method for incomplete data estimation problems

We propose a general procedure for solving incomplete data estimation problems. The procedure can be used to find the maximum likelihood estimate or to solve estimating equations in difficult cases such as estimation with the censored or truncated regression model, the nonlinear structural measurement error model, and the random effects model. The procedure is based on the general principle of stochastic approximation and the Markov chain Monte-Carlo method. Applying the theory on adaptive algorithms, we derive conditions under which the proposed procedure converges. Simulation studies also indicate that the proposed procedure consistently converges to the maximum likelihood estimate for the structural measurement error logistic regression model.

Quantitative modeling of stochastic systems in molecular biology by using stochastic Petri nets

An integrated understanding of molecular and developmental biology must consider the large number of molecular species involved and the low concentrations of many species in vivo. Quantitative stochastic models of molecular interaction networks can be expressed as stochastic Petri nets (SPNs), a mathematical formalism developed in computer science. Existing software can be used to define molecular interaction networks as SPNs and solve such models for the probability distributions of molecular species. This approach allows biologists to focus on the content of models and their interpretation, rather than their implementation. The standardized format of SPNs also facilitates the replication, extension, and transfer of models between researchers. A simple chemical system is presented to demonstrate the link between stochastic models of molecular interactions and SPNs. The approach is illustrated with examples of models of genetic and biochemical phenomena where the UltraSAN package is used to present results from numerical analysis and the outcome of simulations.

Human Search for a Target on a Textured Background is Consistent with a Stochastic Model

This research was supported by the James S. McDonnell Foundation (ARH). Early version was supported by EPSRC grants EP/F02553X/1 and EP/D059364/1.

A model of ocular dominance column development by competition for trophic factor

Recent experimental evidence has shown that application of certain neurotrophic factors (NTs) to the developing primary visual cortex prevents the development of ocular dominance (OD) columns. One interpretation of this result is that afferents from the lateral geniculate nucleus compete for postsynaptic trophic factor in an activity-dependent manner. Application of excess trophic factor eliminates this competition, thereby preventing OD column formation. We present a model of OD column development, incorporating Hebbian synaptic modification and activity-driven competition for NT, which accounts for both normal OD column development as well as the prevention of that development when competition is removed. In the “control” situation, when available NT is below a critical amount, OD columns form normally. These columns form without weight normalization procedures and in the presence of positive inter-eye correlations. In the “experimental” case, OD column development is prevented in a local neighborhood in which excess NT has been added. Our model proposes a biologically plausible mechanism for competition between neural populations that is motivated by several pieces of experimental data, thereby accounting for both normal and experimentally perturbed conditions.

Subplate neuron ablation alters neurotrophin expression and ocular dominance column formation

Ocular dominance column formation in visual cortex depends on both the presence of subplate neurons and the endogenous expression of neurotrophins. Here we show that deletion of subplate neurons, which supply glutamatergic inputs to visual cortex, leads to a paradoxical increase in brain-derived neurotrophic factor mRNA in the same region of visual cortex in which ocular dominance columns are absent. Subplate neuron ablation also increases glutamic acid decarboxylase-67 levels, indicating an alteration in cortical inhibition. These observations imply a role for this special class of neurons in modulating activity-dependent competition by regulating levels of neurotrophins and excitability within a developing cortical circuit.

Regular and stochastic dynamics in the real quadratic family

We prove the Regulat or Stochastic Conjecture for the real quadratic family which asserts that almost every real quadratic map Pc, c ∈ [−2, 1/4], has either an attracting cycle or an absolutely continuous invariant measure.

Stochastic processes strongly influence HIV-1 evolution during suboptimal protease-inhibitor therapy

It has long been assumed that HIV-1 evolution is best described by deterministic evolutionary models because of the large population size. Recently, however, it was suggested that the effective population size (Ne) may be rather small, thereby allowing chance to influence evolution, a situation best described by a stochastic evolutionary model. To gain experimental evidence supporting one of the evolutionary models, we investigated whether the development of resistance to the protease inhibitor ritonavir affected the evolution of the env gene. Sequential serum samples from five patients treated with ritonavir were used for analysis of the protease gene and the V3 domain of the env gene. Multiple reverse transcription–PCR products were cloned, sequenced, and used to construct phylogenetic trees and to calculate the genetic variation and Ne. Genotypic resistance to ritonavir developed in all five patients, but each patient displayed a unique combination of mutations, indicating a stochastic element in the development of ritonavir resistance. Furthermore, development of resistance induced clear bottleneck effects in the env gene. The mean intrasample genetic variation, which ranged from 1.2% to 5.7% before treatment, decreased significantly (P < 0.025) during treatment. In agreement with these findings, Ne was estimated to be very small (500–15,000) compared with the total HIV-1 RNA copy number. This study combines three independent observations, strong population bottlenecking, small Ne, and selection of different combinations of protease-resistance mutations, all of which indicate that HIV-1 evolution is best described by a stochastic evolutionary model.