Experience generalization for multi-agent reinforcement learning

On-line learning methods have been applied successfully in multi-agent systems to achieve coordination among agents. Learning in multi-agent systems implies in a non-stationary scenario perceived by the agents, since the behavior of other agents may change as they simultaneously learn how to improve their actions. Non-stationary scenarios can be modeled as Markov Games, which can be solved using the Minimax-Q algorithm a combination of Q-learning (a Reinforcement Learning (RL) algorithm which directly learns an optimal control policy) and the Minimax algorithm. However, finding optimal control policies using any RL algorithm (Q-learning and Minimax-Q included) can be very time consuming. Trying to improve the learning time of Q-learning, we considered the QS-algorithm. in which a single experience can update more than a single action value by using a spreading function. In this paper, we contribute a Minimax-QS algorithm which combines the Minimax-Q algorithm and the QS-algorithm. We conduct a series of empirical evaluation of the algorithm in a simplified simulator of the soccer domain. We show that even using a very simple domain-dependent spreading function, the performance of the learning algorithm can be improved.

Optimal attitude corrections for cylindrical spacecraft

A first order analytical model for optimal small amplitude attitude maneuvers of spacecraft with cylindrical symmetry in an elliptical orbits is presented. The optimization problem is formulated as a Mayer problem with the control torques provided by a power limited propulsion system. The state is defined by Seffet-Andoyer's variables and the control by the components of the propulsive torques. The Pontryagin Maximum Principle is applied to the problem and the optimal torques are given explicitly in Serret-Andoyer's variables and their adjoints. For small amplitude attitude maneuvers, the optimal Hamiltonian function is linearized around a reference attitude. A complete first order analytical solution is obtained by simple quadrature and is expressed through a linear algebraic system involving the initial values of the adjoint variables. A numerical solution is obtained by taking the Euler angles formulation of the problem, solving the two-point boundary problem through the shooting method, and, then, determining the Serret-Andoyer variables through Serret-Andoyer transformation. Numerical results show that the first order solution provides a good approximation to the optimal control law and also that is possible to establish an optimal control law for the artificial satellite's attitude. (C) 2003 COSPAR. Published by Elsevier B.V. Ltd. All rights reserved.

On relaxed LMI-based designs for fuzzy regulators and fuzzy observers

Relaxed conditions for stability of nonlinear, continuous and discrete-time systems given by fuzzy models are presented. A theoretical analysis shows that the proposed methods provide better or at least the same results of the methods presented in the literature. Numerical results exemplify this fact. These results are also used for fuzzy regulators and observers designs. The nonlinear systems are represented by fuzzy models proposed by Takagi and Sugeno. The stability analysis and the design of controllers are described by linear matrix inequalities, that can be solved efficiently using convex programming techniques. The specification of the decay rate, constrains on control input and output are also discussed.

Control Design Applied to a Micro Electro Mechanical System: MEMS Comb drive

This paper presents the linear optimal control technique for reducing the chaotic movement of the micro-electro-mechanical Comb Drive system to a small periodic orbit. We analyze the non-linear dynamics in a micro-electro-mechanical Comb Drive and demonstrated that this model has a chaotic behavior. Chaos control problems consist of attempts to stabilize a chaotic system to an equilibrium point, a periodic orbit, or more general, about a given reference trajectory. This technique is applied in analyzes the nonlinear dynamics in an MEMS Comb drive. The simulation results show the identification by linear optimal control is very effective.

State constraints in optimal impulsive controls

Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)

Optimality conditions for infinite horizon control problems with state constraints

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

On an Active Control for a Structurally Nonlinear Mechanical System, Taking Into Account an Energy Pumping

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Mathematical modeling and control of population systems: Applications in biological pest control

The aim of this paper is to apply methods from optimal control theory, and from the theory of dynamic systems to the mathematical modeling of biological pest control. The linear feedback control problem for nonlinear systems has been formulated in order to obtain the optimal pest control strategy only through the introduction of natural enemies. Asymptotic stability of the closed-loop nonlinear Kolmogorov system is guaranteed by means of a Lyapunov function which can clearly be seen to be the solution of the Hamilton-Jacobi-Bellman equation, thus guaranteeing both stability and optimality. Numerical simulations for three possible scenarios of biological pest control based on the Lotka-Volterra models are provided to show the effectiveness of this method. (c) 2007 Elsevier B.V. All rights reserved.

On control and synchronization in chaotic and hyperchaotic systems via linear feedback control

This paper presents the control and synchronization of chaos by designing linear feedback controllers. The linear feedback control problem for nonlinear systems has been formulated under optimal control theory viewpoint. Asymptotic stability of the closed-loop nonlinear system is guaranteed by means of a Lyapunov function which can clearly be seen to be the solution of the Hamilton-Jacobi-Bellman equation thus guaranteeing both stability and optimality. The formulated theorem expresses explicitly the form of minimized functional and gives the sufficient conditions that allow using the linear feedback control for nonlinear system. The numerical simulations were provided in order to show the effectiveness of this method for the control of the chaotic Rossler system and synchronization of the hyperchaotic Rossler system. (C) 2007 Elsevier B.V. All rights reserved.

A class of multiobjective control problems

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Global optimization approach for the H2-norm model reduction problem

A branch and bound algorithm is proposed to solve the H2-norm model reduction problem for continuous-time linear systems, with conditions assuring convergence to the global optimum in finite time. The lower and upper bounds used in the optimization procedure are obtained through Linear Matrix Inequalities formulations. Examples illustrate the results.

Necessary conditions of optimality for vector-valued impulsive control problems

A vector-valued impulsive control problem is considered whose dynamics, defined by a differential inclusion, are such that the vector fields associated with the singular term do not satisfy the so-called Frobenius condition. A concept of robust solution based on a new reparametrization procedure is adopted in order to derive necessary conditions of optimality. These conditions are obtained by taking a limit of those for an appropriate sequence of auxiliary standard optimal control problems approximating the original one. An example to illustrate the nature of the new optimality conditions is provided. © 2000 Elsevier Science B.V. All rights reserved.

Novos resultados sobre a estabilidade e controle de sistemas nao-lineares utilizando modelos fuzzy e LMI

Relaxed conditions for the stability study of nonlinear, continuous and discrete-time systems given by fuzzy models are presented. A theoretical analysis shows that the proposed method provides better or at least the same results of the methods presented in the literature. Digital simulations exemplify this fact. These results are also used for the fuzzy regulators design. The nonlinear systems are represented by the fuzzy models proposed by Takagi and Sugeno. The stability analysis and the design of controllers are described by LMIs (Linear Matrix Inequalities), that can be solved efficiently by convex programming techniques. The specification of the decay rate, constraints on control input and output are also described by LMIs. Finally, the proposed design method is applied in the control of an inverted pendulum.

Analysis of a sub-optimal scheme of drug dosage in the AIDS treatment

Here the results for CD4+T cells count and the viral load obtained from HIV sero-positive patients are compared with results from numerical simulations by computer. Also, the standard scheme of administration of drugs anti HIV (HAART schemes) which uses constant doses is compared with an alternative sub-optimal teatment scheme which uses variable drug dosage according to the evolution of a quantitative measure of the side effects. The quantitative analysis done here shows that it is possible to obtain, using the alternative scheme, the same performance of actual data but using variable dosage and having fewer side effects. Optimal control theory is used to solve and also to provide a prognosis related to the strategies for control of viraemia.

Stochastic Stability for Markovian Jump Linear Systems Subject to a Crucial Failure Event

This paper is concerned with the stability of discrete-time linear systems subject to random jumps in the parameters, described by an underlying finite-state Markov chain. In the model studied, a stopping time τ Δ is associated with the occurrence of a crucial failure after which the system is brought to a halt for maintenance. The usual stochastic stability concepts and associated results are not indicated, since they are tailored to pure infinite horizon problems. Using the concept named stochastic τ-stability, equivalent conditions to ensure the stochastic stability of the system until the occurrence of τ Δ is obtained. In addition, an intermediary and mixed case for which τ represents the minimum between the occurrence of a fix number N of failures and the occurrence of a crucial failure τ Δ is also considered. Necessary and sufficient conditions to ensure the stochastic τ-stability are provided in this setting that are auxiliary to the main result.