Efficient methods for stochastic optimal control

The Hamilton Jacobi Bellman (HJB) equation is central to stochastic optimal control (SOC) theory, yielding the optimal solution to general problems specified by known dynamics and a specified cost functional. Given the assumption of quadratic cost on the control input, it is well known that the HJB reduces to a particular partial differential equation (PDE). While powerful, this reduction is not commonly used as the PDE is of second order, is nonlinear, and examples exist where the problem may not have a solution in a classical sense. Furthermore, each state of the system appears as another dimension of the PDE, giving rise to the curse of dimensionality. Since the number of degrees of freedom required to solve the optimal control problem grows exponentially with dimension, the problem becomes intractable for systems with all but modest dimension.

In the last decade researchers have found that under certain, fairly non-restrictive structural assumptions, the HJB may be transformed into a linear PDE, with an interesting analogue in the discretized domain of Markov Decision Processes (MDP). The work presented in this thesis uses the linearity of this particular form of the HJB PDE to push the computational boundaries of stochastic optimal control.

This is done by crafting together previously disjoint lines of research in computation. The first of these is the use of Sum of Squares (SOS) techniques for synthesis of control policies. A candidate polynomial with variable coefficients is proposed as the solution to the stochastic optimal control problem. An SOS relaxation is then taken to the partial differential constraints, leading to a hierarchy of semidefinite relaxations with improving sub-optimality gap. The resulting approximate solutions are shown to be guaranteed over- and under-approximations for the optimal value function. It is shown that these results extend to arbitrary parabolic and elliptic PDEs, yielding a novel method for Uncertainty Quantification (UQ) of systems governed by partial differential constraints. Domain decomposition techniques are also made available, allowing for such problems to be solved via parallelization and low-order polynomials.

The optimization-based SOS technique is then contrasted with the Separated Representation (SR) approach from the applied mathematics community. The technique allows for systems of equations to be solved through a low-rank decomposition that results in algorithms that scale linearly with dimensionality. Its application in stochastic optimal control allows for previously uncomputable problems to be solved quickly, scaling to such complex systems as the Quadcopter and VTOL aircraft. This technique may be combined with the SOS approach, yielding not only a numerical technique, but also an analytical one that allows for entirely new classes of systems to be studied and for stability properties to be guaranteed.

The analysis of the linear HJB is completed by the study of its implications in application. It is shown that the HJB and a popular technique in robotics, the use of navigation functions, sit on opposite ends of a spectrum of optimization problems, upon which tradeoffs may be made in problem complexity. Analytical solutions to the HJB in these settings are available in simplified domains, yielding guidance towards optimality for approximation schemes. Finally, the use of HJB equations in temporal multi-task planning problems is investigated. It is demonstrated that such problems are reducible to a sequence of SOC problems linked via boundary conditions. The linearity of the PDE allows us to pre-compute control policy primitives and then compose them, at essentially zero cost, to satisfy a complex temporal logic specification.

Era uma vez uma ilha de Pescadores Artesanais: impactos socioambientais dos grandes complexos industriais, conflitos e resistência (Ilha da Madeira/Itaguaí/RJ)

O presente estudo visa analisar os processos de transformações econômicos, políticos e socioambientais decorrentes da instalação dos grandes empreendimentos em territórios tradicionais da pesca, mais especificamente, as experiências da comunidade pesqueira da Ilha da Madeira/baía de Sepetiba/Itaguaí-RJ, desde a instalação da Cia Ingá Mercantil (1964) até os dias atuais, identificando, nos vários ciclos de industrialização: os fatores endógenos e exógenos que contribuem para a vulnerabilidade ou sustentabilidade da pesca artesanal e do meio ambiente. Sinalizando, nesta experiência, alguns aspectos que possam servir de referência para outras comunidades pesqueiras que vivenciam problemas similares. Introduzimos a problemática a partir da contextualização da pesca artesanal no Brasil, as políticas, a regulamentação da atividade, a organização dos pescadores. Ao evidenciar a pesca artesanal no estado do Rio de Janeiro, destacamos os conflitos socioambientais decorrentes da instalação de complexos industriais em territórios tradicionalmente ocupados por pescadores, com destaque para os conflitos relativos à instalação do Porto de Açu, em São João da Barra/RJ e os gasodutos para a refinaria de petróleo na baía de Guanabara. Aprofundamos a temática, a partir de um estudo de caso na Ilha da Madeira, baía de Sepetiba, Itaguaí/RJ. Esse território, tradicionalmente ocupado por pescadores, mergulhou em uma crise socioambiental a partir da década de 60 e, desde então, vem passando por diversas transformações: alteração radical da paisagem, degradação ambiental além do sufocamento da atividade pesqueira. Os fatos são evidenciados por meio de pesquisas bibliográficas, documentais, registros fotográficos, sobretudo, história de oral. Em entrevistas com informantes-chave resgatamos as memórias pessoais e, nesse percurso, fomos recuperando parte da história do território. Caracterizando a paisagem, a vida e trabalho dos pescadores, a cultura local: tradições, costumes, valores, aspectos materiais e simbólicos, em um período anterior a chegada das indústrias, quando a Ilha da Madeira era de fato, uma Ilha. Em suas narrativas os entrevistados foram pontuando as sucessões dos trágicos acontecimentos que ocorreram após a instalação da Ingá até os dias atuais. Esses fatos são demarcados em ciclos que compõem a crise socioambiental no território. Um estudo que retrata a injustiça ambiental, a vulnerabilidade de uma comunidade pesqueira, cuja experiência serve de alerta para outras comunidades tradicionais. Ressaltamos a importância das articulações entre os movimentos locais com instâncias extras locais, sinalizando para a necessidade de democratização dos processos decisórios e da gestão compartilhada dos recursos de uso comum. Também pontuamos a urgência de superação do paradigma que dissocia desenvolvimento, natureza e sociedade, fortalecendo uma lógica de produção que, ao se impor como hegemônica sufoca todas outras formas de organização do trabalho.

A Novel Scheduling Index Rule Proposal for QoE Maximization in Wireless Networks

This paper deals with the resource allocation problem aimed at maximizing users' perception of quality in wireless channels with time-varying capacity. First of all, we model the subjective quality-aware scheduling problem in the framework of Markovian decision processes. Then, given that the obtaining of the optimal solution of this model is unachievable, we propose a simple scheduling index rule with closed-form expression by using a methodology based on Whittle approach. Finally, we analyze the performance of the achieved scheduling proposal in several relevant scenarios, concluding that it outperforms the most popular existing resource allocation strategies.

Atores e agendas da política externa brasileira para a África e a instrumentalização da cooperação em segurança alimentar (2003-2010)

Durante os dois mandatos presidenciais de Luiz Inácio Lula da Silva (2003-2010), percebeu-se, em virtude de pressões intra e extraburocráticas e de causalidades sistêmicas, maior acentuação do esboroamento da histórica condição insular do Ministério das Relações Exteriores (MRE). A participação de novos entes que não o Itamaraty na configuração da política externa, notadamente em seu vetor de execução, enseja novas agendas cooperativas e processos decisórios. Atores da burocracia federal, como os ministérios, vocalizam preferências que influenciam o jogo interburocrático e têm o condão de estabelecer possíveis pontes com a instituição diplomática, unidade de decisão por excelência. Na perspectiva intraburocrática, a ascensão de corrente de ação e de pensamento dos autonomistas, frente aos institucionalistas pragmáticos, permite escolhas de inserção internacional como o reforço da perspectiva sul-sul, na qual se inserem as parcerias com a África, o que indica a inexistência de monolitismo de opiniões no interior do MRE. Essa dinâmica faz-se presente e é necessária para o entendimento da Cooperação Brasileira para o Desenvolvimento Internacional (CBDI), tipo de Cooperação Sul-Sul (CSS) do Brasil que tem na Cooperação Técnica, Científica e Tecnológica (CTC&T) em segurança alimentar uma de suas modalidades mais atuantes e complexas. Convencionada como instrumento de política externa durante a ascendência dos autonomistas, corrente influenciada por quadros do Partido dos Trabalhadores, a cooperação em segurança alimentar teve o continente africano como locus primordial de manifestação. Embasado na internacionalização de políticas públicas domésticas, o compartilhamento de conhecimentos nas agendas de combate à fome, de combate à pobreza e de desenvolvimento agrário é fenômeno tributário da abertura da caixa preta estatal, o que ratifica o argumento de que há correlação entre níveis de análise. As diversas iniciativas cooperativas para com parceiros da outra margem do Atlântico Sul, eivadas de componente retórico de promoção de ordem internacional menos assimétrica, donde também subjace a busca consecução de interesses diretos e indiretos dos formuladores diplomáticos, guardam relação com as diretrizes mais gerais da política externa articulada no período estudado nesta dissertação.

Assessing the responsiveness of existing production operations

Production responsiveness refers to the ability of a production system to achieve its operational goals in the presence of supplier, internal and customer disturbances, where disturbances are those sources of change which occur independently of the system's intentions. A set of audit tools for assessing the responsiveness of production operations is being prepared as part of an EPSRC funded investigation. These tools are based on the idea that the ability to respond is linked to: the nature of the disturbances or changes requiring a response; their impact on production goals; and the inherent response capabilities of the operation. These response capabilities include information gathering and processing (to detect disturbances and production conditions), decision processes (which initiate system responses to disturbances) and various types of process flexibilities and buffers (which provide the physical means of dealing with disturbances). The paper discusses concepts and issues associated with production responsiveness, describes the audit tools that have been developed and illustrates their use in the context of a steel manufacturing plant.

Scaling POMDPs for dialog management with composite summary point-based value iteration (CSPBVI)

Although partially observable Markov decision processes (POMDPs) have shown great promise as a framework for dialog management in spoken dialog systems, important scalability issues remain. This paper tackles the problem of scaling slot-filling POMDP-based dialog managers to many slots with a novel technique called composite point-based value iteration (CSPBVI). CSPBVI creates a "local" POMDP policy for each slot; at runtime, each slot nominates an action and a heuristic chooses which action to take. Experiments in dialog simulation show that CSPBVI successfully scales POMDP-based dialog managers without compromising performance gains over baseline techniques and preserving robustness to errors in user model estimation. Copyright © 2006, American Association for Artificial Intelligence (www.aaai.org). All rights reserved.

Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better. © 2011 ACM.

Tools for Assessing the Responsiveness of Existing Production Operations

A set of audit tools is being prepared for assessing the response capability of a production operation, as part of an EPSRC1 funded investigation into improving the responsiveness of manufacturing production systems. These tools are based on the idea that the ability to respond is linked to i) the nature of the disturbances or changes requiring a response, ii) their impact on production goals and iii) the decision processes which initiate system responses to disturbances.

Climate resilience of schools: a case study

Social and political concerns are frequently reflected in the design of school buildings, often in turn leading to the development of technical innovations. One example is a recurrent concern about the physical health of the nation, which has at several points over the last century prompted new design approaches to natural light and ventilation. The most critical concern of the current era is the global, rather than the indoor, environment. The resultant political focus on mitigating climate change has resulted in new regulations, and in turn considerable technical changes in building design and construction. The vanguard of this movement has again been in school buildings, set the highest targets for reducing operational carbon by the previous Government. The current austerity measures have moved the focus to the refurbishment and retrofit of existing buildings, in order to bring them up to the exacting new standards. Meanwhile there is little doubt that climate change is happening already, and that the impacts will be considerable. Climate scientists have increasing confidence in their predictions for the future; if today’s buildings are to be resilient to these changes, building designers will need to understand and design for the predicted climates in order to continue to provide comfortable and healthy spaces through the lifetimes of the buildings. This paper describes the decision processes, and the planned design measures, for adapting an existing school for future climates. The project is at St Faith’s School in Cambridge, and focuses on three separate buildings: a large Victorian block built as a substantial domestic dwelling in 1885, a smaller single storey 1970s block with a new extension, and an as-yet unbuilt single storey block designed to passivhaus principles and using environmentally friendly materials. The implications of climate change have been considered for the three particular issues of comfort, construction, and water, as set out in the report on Design for Future Climate: opportunities for adaptation in the built environment (Gething, 2010). The adaptation designs aim to ensure each of the three very different buildings remains fit for purpose throughout the 21st century, continuing to provide a healthy environment for the children. A forth issue, the reduction of carbon and the mitigation of other negative environmental impacts of the construction work, is also a fundamental aim for the school and the project team. Detailed modelling of both the operational and embodied energy and carbon of the design options is therefore being carried out, in order that the whole life carbon costs of the adaptation design options may be minimised. The project has been funded by the Technology Strategy Board as part of the Design for Future Climates programme; the interdisciplinary team includes the designers working on the current school building projects and the school bursar, supported by researchers from the University of Cambridge Centre for Sustainable Development. It is hoped that lessons from the design process, as well as the solutions themselves, will be transferable to other buildings in similar climatic regions.

POMDP-based statistical spoken dialog systems: A review

Statistical dialog systems (SDSs) are motivated by the need for a data-driven framework that reduces the cost of laboriously handcrafting complex dialog managers and that provides robustness against the errors created by speech recognizers operating in noisy environments. By including an explicit Bayesian model of uncertainty and by optimizing the policy via a reward-driven process, partially observable Markov decision processes (POMDPs) provide such a framework. However, exact model representation and optimization is computationally intractable. Hence, the practical application of POMDP-based systems requires efficient algorithms and carefully constructed approximations. This review article provides an overview of the current state of the art in the development of POMDP-based spoken dialog systems. © 1963-2012 IEEE.

The Essential Dynamics Algorithm: Essential Results

This paper presents a novel algorithm for learning in a class of stochastic Markov decision processes (MDPs) with continuous state and action spaces that trades speed for accuracy. A transform of the stochastic MDP into a deterministic one is presented which captures the essence of the original dynamics, in a sense made precise. In this transformed MDP, the calculation of values is greatly simplified. The online algorithm estimates the model of the transformed MDP and simultaneously does policy search against it. Bounds on the error of this approximation are proven, and experimental results in a bicycle riding domain are presented. The algorithm learns near optimal policies in orders of magnitude fewer interactions with the stochastic MDP, using less domain knowledge. All code used in the experiments is available on the project's web site.

Task-Irrelevant Perceptual Learning Specific to the Contrast Polarity of Motion Stimuli

Studies of perceptual learning have focused on aspects of learning that are related to early stages of sensory processing. However, conclusions that perceptual learning results in low-level sensory plasticity are of great controversy, largely because such learning can often be attributed to plasticity in later stages of sensory processing or in the decision processes. To address this controversy, we developed a novel random dot motion (RDM) stimulus to target motion cells selective to contrast polarity, by ensuring the motion direction information arises only from signal dot onsets and not their offsets, and used these stimuli in conjunction with the paradigm of task-irrelevant perceptual learning (TIPL). In TIPL, learning is achieved in response to a stimulus by subliminally pairing that stimulus with the targets of an unrelated training task. In this manner, we are able to probe learning for an aspect of motion processing thought to be a function of directional V1 simple cells with a learning procedure that dissociates the learned stimulus from the decision processes relevant to the training task. Our results show learning for the exposed contrast polarity and that this learning does not transfer to the unexposed contrast polarity. These results suggest that TIPL for motion stimuli may occur at the stage of directional V1 simple cells.

The influence of gender in the direction of Spanish cooperative societies on profitability and indebtedness: an empirical analysis

In this paper the claim for the market for a new business management to ensure the presence of women in decision -making to respond to new social needs addressed. Thus, this paper analyzes the influence of gender diversity of the directors on the profitability and the level of debt for a sample of 5,199 Spanish cooperatives. Unlike capitalist societies, these organizations have a number of peculiarities in their government, and that the partners are themselves major time, agents and customers. The study focuses on the Spanish context, where there is an open debate on the importance of women's business management, as in other countries, driven by the proliferation of legislation on gender equality, being, in addition, Spain, the pioneer in having specific legislation on Social Economy. The results show that cooperatives with greater female representation in theirs Boards have higher profitability. On the other hand, those Boards with a higher percentage of women show a lower level of indebtedness.

¿Barreras invisibles? Actitudes de los estudiantes universitarios ante sus compañeros con discapacidad

El presente trabajo analiza los conocimientos, opiniones y actitudes de los estudiantes universitarios sobre sus compañeros con discapacidad. La integración educativa y social de estos estudiantes es objeto de preocupación y estudio en todo el sistema educativo español, incluida la enseñanza superior en la que 21.942 universitarios con discapacidad se encontraban matriculados en el curso 2013-2014. El estudio se realiza en el marco de dos Proyectos de Innovación Educativa[1] de la Universidad de Málaga en los que se examinaron las actitudes del personal docente e investigador, el alumnado en general y el alumnado con discapacidad de esta institución. En este artículo se recogen las perspectivas de los estudiantes sin discapacidad, necesarias para medir el grado de aceptación dentro de las aulas universitarias. Se empleó una encuesta suministrada mediante correo electrónico a la totalidad de la población de estudiantes de la Universidad de Málaga (N= 32.464). El tamaño de la muestra fue de 620 estudiantes, por lo que trabajamos con un margen de error del 3,9% para la estimación de los porcentajes con un nivel de confianza del 95%. Los datos, analizados mediante análisis estadístico, visibilizan la importancia de un elevado grado de información y conocimiento así como de una experiencia directa con la discapacidad. Ambas circunstancias se traducen en actitudes más positivas y favorables. Tan solo el 20% de los alumnos manifestó haber recibido en alguna ocasión cualquier tipo de información sobre este asunto por parte de la Universidad; no obstante, la mayoría valora positivamente la presencia de alumnos con discapacidad en la Universidad, la posibilidad de contar ellos como compañeros de clase y el enriquecimiento que supone esta convivencia dentro del aula. Se discuten posibles acciones para mejorar las actitudes hacia los estudiantes con discapacidad. Además, se plantea incluir una perspectiva que promueva el empoderamiento de estos estudiantes. [1] PIE 10-135 y PIE 13-103.

Improving appropriate polypharmacy for older people in primary care: selecting components of an evidence-based intervention to target prescribing and dispensing.

Background
The use of multiple medicines (polypharmacy) is increasingly common in older people. Ensuring that patients receive the most appropriate combinations of medications (appropriate polypharmacy) is a significant challenge. The quality of evidence to support the effectiveness of interventions to improve appropriate polypharmacy is low. Systematic identification of mediators of behaviour change, using the Theoretical Domains Framework (TDF), provides a theoretically robust evidence base to inform intervention design. This study aimed to (1) identify key theoretical domains that were perceived to influence the prescribing and dispensing of appropriate polypharmacy to older patients by general practitioners (GPs) and community pharmacists, and (2) map domains to associated behaviour change techniques (BCTs) to include as components of an intervention to improve appropriate polypharmacy in older people in primary care.

Methods
Semi-structured interviews were conducted with members of each healthcare professional (HCP) group using tailored topic guides based on TDF version 1 (12 domains). Questions covering each domain explored HCPs’ perceptions of barriers and facilitators to ensuring the prescribing and dispensing of appropriate polypharmacy to older people. Interviews were audio-recorded and transcribed verbatim. Data analysis involved the framework method and content analysis. Key domains were identified and mapped to BCTs based on established methods and discussion within the research team.

Results
Thirty HCPs were interviewed (15 GPs, 15 pharmacists). Eight key domains were identified, perceived to influence prescribing and dispensing of appropriate polypharmacy: ‘Skills’, ‘Beliefs about capabilities’, ‘Beliefs about consequences’, ‘Environmental context and resources’, ‘Memory, attention and decision processes’, ‘Social/professional role and identity’, ‘Social influences’ and ‘Behavioural regulation’. Following mapping, four BCTs were selected for inclusion in an intervention for GPs or pharmacists: ‘Action planning’, ‘Prompts/cues’, ‘Modelling or demonstrating of behaviour’ and ‘Salience of consequences’. An additional BCT (‘Social support or encouragement’) was selected for inclusion in a community pharmacy-based intervention in order to address barriers relating to interprofessional working that were encountered by pharmacists.

Conclusions
Selected BCTs will be operationalised in a theory-based intervention to improve appropriate polypharmacy for older people, to be delivered in GP practice and community pharmacy settings. Future research will involve development and feasibility testing of this intervention.