Gradient-based reinforcement learning techniques for underwater robotics behavior learning

Darrerament, l'interès pel desenvolupament d'aplicacions amb robots submarins autònoms (AUV) ha crescut de forma considerable. Els AUVs són atractius gràcies al seu tamany i el fet que no necessiten un operador humà per pilotar-los. Tot i això, és impossible comparar, en termes d'eficiència i flexibilitat, l'habilitat d'un pilot humà amb les escasses capacitats operatives que ofereixen els AUVs actuals. L'utilització de AUVs per cobrir grans àrees implica resoldre problemes complexos, especialment si es desitja que el nostre robot reaccioni en temps real a canvis sobtats en les condicions de treball. Per aquestes raons, el desenvolupament de sistemes de control autònom amb l'objectiu de millorar aquestes capacitats ha esdevingut una prioritat. Aquesta tesi tracta sobre el problema de la presa de decisions utilizant AUVs. El treball presentat es centra en l'estudi, disseny i aplicació de comportaments per a AUVs utilitzant tècniques d'aprenentatge per reforç (RL). La contribució principal d'aquesta tesi consisteix en l'aplicació de diverses tècniques de RL per tal de millorar l'autonomia dels robots submarins, amb l'objectiu final de demostrar la viabilitat d'aquests algoritmes per aprendre tasques submarines autònomes en temps real. En RL, el robot intenta maximitzar un reforç escalar obtingut com a conseqüència de la seva interacció amb l'entorn. L'objectiu és trobar una política òptima que relaciona tots els estats possibles amb les accions a executar per a cada estat que maximitzen la suma de reforços totals. Així, aquesta tesi investiga principalment dues tipologies d'algoritmes basats en RL: mètodes basats en funcions de valor (VF) i mètodes basats en el gradient (PG). Els resultats experimentals finals mostren el robot submarí Ictineu en una tasca autònoma real de seguiment de cables submarins. Per portar-la a terme, s'ha dissenyat un algoritme anomenat mètode d'Actor i Crític (AC), fruit de la fusió de mètodes VF amb tècniques de PG.

Hygiene assessment system (HAS) scores: an analysis of the available data from English slaughterhouses

As control systems have developed and the implications of poor hygienic practices have become better known, the evaluation of the hygienic status of premises has become more critical. The assessment of the overall status of premises hygiene call provide useful management data indicating whether the premises are improving or whether, whilst still meeting legal requirements, they might be failing to maintain previously high standards. Since the creation, for the United Kingdom, of the meat hygiene service (MHS), one of the aims of the service was to monitor hygiene on different premises to provide a means of comparing standards and to identify and encourage improvements. This desire led to the implementation of a scoring system known as the hygiene assessment system (HAS). This paper analyses English slaughterhouses HAS scores between 1998 and 2005 outlining the main incidents throughout this period, Although rising initially, the later results displayed a clear decrease in the general hygiene scores. These revealing results coincide with the start of a new meat inspection system where, after several years of discussion, risk based inspection is finally coming to a reality within Europe. The paper considers the implications of these changes in the way hygiene standards will be monitored in the future.

Robust identification for linear-in-the-parameters models

In this paper new robust nonlinear model construction algorithms for a large class of linear-in-the-parameters models are introduced to enhance model robustness, including three algorithms using combined A- or D-optimality or PRESS statistic (Predicted REsidual Sum of Squares) with regularised orthogonal least squares algorithm respectively. A common characteristic of these algorithms is that the inherent computation efficiency associated with the orthogonalisation scheme in orthogonal least squares or regularised orthogonal least squares has been extended such that the new algorithms are computationally efficient. A numerical example is included to demonstrate effectiveness of the algorithms. Copyright (C) 2003 IFAC.

Oriented vehicles travelling in spherical space

This paper tackles the path planning problem for oriented vehicles travelling in the non-Euclidean 3-Dimensional space; spherical space S3. For such problem, the orientation of the vehicle is naturally represented by orthonormal frame bundle; the rotation group SO(4). Orthonormal frame bundles of space forms coincide with their isometry groups and therefore the focus shifts to control systems defined on Lie groups. The oriented vehicles, in this case, are constrained to travel at constant speed in a forward direction and their angular velocities directly controlled. In this paper we identify controls that induce steady motions of these oriented vehicles and yield closed form parametric expressions for these motions. The paths these vehicles trace are defined explicitly in terms of the controls and therefore invariant with respect to the coordinate system used to describe the motion.

A model of intelligent building energy management for the indoor environment

People's interaction with the indoor environment plays a significant role in energy consumption in buildings. Mismatching and delaying occupants' feedback on the indoor environment to the building energy management system is the major barrier to the efficient energy management of buildings. There is an increasing trend towards the application of digital technology to support control systems in order to achieve energy efficiency in buildings. This article introduces a holistic, integrated, building energy management model called `smart sensor, optimum decision and intelligent control' (SMODIC). The model takes into account occupants' responses to the indoor environments in the control system. The model of optimal decision-making based on multiple criteria of indoor environments has been integrated into the whole system. The SMODIC model combines information technology and people centric concepts to achieve energy savings in buildings.

Integrable quadratic Hamiltonians on the Euclidean group of motions

In this paper, we discuss the problem of globally computing sub-Riemannian curves on the Euclidean group of motions SE(3). In particular, we derive a global result for special sub-Riemannian curves whose Hamiltonian satisfies a particular condition. In this paper, sub-Riemannian curves are defined in the context of a constrained optimal control problem. The maximum principle is then applied to this problem to yield an appropriate left-invariant quadratic Hamiltonian. A number of integrable quadratic Hamiltonians are identified. We then proceed to derive convenient expressions for sub-Riemannian curves in SE(3) that correspond to particular extremal curves. These equations are then used to compute sub-Riemannian curves that could potentially be used for motion planning of underwater vehicles.

In vitro examination of the effect of Orlistat on the ability of the faecal microbiota to utilize dietary lipids

Orlistat is an anti-obesity treatment with which several gastrointestinal (GI) side-effects are commonly associated in the initial stages of therapy. There is no physiological explanation as to why two-thirds of those who take the drug experience one or more side-effects. It has been hypothesized that the GI microbiota may protect from or contribute to these GI disturbances. Using in vitro batch culture and human gut model systems, studies were conducted to determine whether increased availability of dietary lipids and/or orlistat affect the composition and/or activity of the faecal microbiota. Results from 24-h batch culture fermentation experiments demonstrated no effect of orlistat in the presence or absence of a dietary lipid (olive oil) on the composition of bacterial communities [as determined by fluorescence in situ hybridization (FISH) and denaturing gradient gel electrophoresis (DGGE) analyses], but did show there was great variability in the lipolytic activities of the microbiotas of individuals, as determined by gas chromatography analysis of long-chain fatty acids in samples. Subsequent studies focused on the effect of orlistat in the presence and absence of lipid in in vitro human gut model systems. Systems were run for 14 days with gut model medium (GMM) only (to steady state, SS), then fed at 12-h intervals with 50 mg orlistat, 2 g olive oil or a mixture of both for 14 days. FISH and DGGE were used to monitor changes in bacterial populations. Bacteria were cultivated from the GMM only (control) systems at SS. All strains isolated were screened for lipolytic activity using tributyrin agar. FISH and DGGE demonstrated that none of the compounds (singly or in combination) added to the systems had any notable effect on microbial population dynamics for any of the donors, although Subdoligranulum populations appeared to be inhibited by orlistat in the presence or absence of lipid. Orlistat had little or no effect on the metabolism of indigenous and added lipids in the fermentation systems, but there was great variability in the way the faecal microbiotas of the donors were able to degrade added lipids. Variability in lipid degradation could be correlated with the number and activity of isolated lipolytic bacteria. The mechanism by which orlistat and the GI microbiota cause side-effects in individuals is unknown, but several hypotheses have been proposed to account for their manifestation. The demonstration of great variability in the lipolytic activity of microbiotas to degrade lipids led to a large-scale cultivation-based study of lipolytic/lipase-positive bacteria present in the human faecal microbiota. Of 4,000 colonies isolated from 15 donors using five different agars, 378 strains were identified that had lipase activity. Molecular identification of strains isolated from five donors demonstrated that lipase activity is more prevalent in the human GI microbiota than previously thought, with members of the phyla Firmicutes, Bacteroidetes and Actinobacteria identified. Molecular identification and characterization of the substrate specificities of the strains will be carried out as part of ongoing work.

The relative order and inverses of recurrent networks

Differential geometry is used to investigate the structure of neural-network-based control systems. The key aspect is relative order—an invariant property of dynamic systems. Finite relative order allows the specification of a minimal architecture for a recurrent network. Any system with finite relative order has a left inverse. It is shown that a recurrent network with finite relative order has a local inverse that is also a recurrent network with the same weights. The results have implications for the use of recurrent networks in the inverse-model-based control of nonlinear systems.

Experiments into biology-technology interaction

In this paper a look is taken at how the use of implant and electrode technology can be employed to create biological brains for robots, to enable human enhancement and to diminish the effects of certain neural illnesses. In all cases the end result is to increase the range of abilities of the recipients. An indication is given of a number of areas in which such technology has already had a profound effect, a key element being the need for a clear interface linking a biological brain directly with computer technology. The emphasis is placed on practical scientific studies that have been and are being undertaken and reported on. The area of focus is the use of electrode technology, where either a connection is made directly with the cerebral cortex and/or nervous system or where implants into the human body are involved. The paper also considers robots that have biological brains in which human neurons can be employed as the sole thinking machine for a real world robot body.

Some necessary and sufficient conditions for eigenstructure assignment

Some necessary and sufficient conditions for closed-loop eigenstructure assignment by output feedback in time-invariant linear multivariable control systems are presented. A simple condition on a square matrix necessary and sufficient for it to be the closed-loop plant matrix of a given system with some output feedback is the basis of the paper. Some known results on entire eigenstructure assignment are deduced from this. The concept of an inner inverse of a matrix is employed to obtain a condition concerning the assignment of an eigenstructure consisting of the eigenvalues and a mixture of left and right eigenvectors.

A comparison between classical techniques and eigenstructure assignment in the design of a longitudinal instrument landing system

Although modern control techniques such as eigenstructure assignment have been given extensive coverage in control literature there is a reluctance to use them in practice as they are often not believed to be as `visible' or as simple as classical methods. A simple aircraft example is used, and it is shown that eigenstructure assignment can be used easily to produce a more viable controller than with simple classical techniques.

Strong structural controllability and the multilink inverted pendulum

This technical note investigates the controllability of the linearized dynamics of the multilink inverted pendulum as the number of links and the number and location of actuators changes. It is demonstrated that, in some instances, there exist sets of parameter values that render the system uncontrollable and so usual methods for assessing controllability are difficult to employ. To assess the controllability, a theorem on strong structural controllability for single-input systems is extended to the multiinput case.