A proposal of a behavior-based control architecture with reinforcement learning for an autonomous underwater robot

Aquesta tesi proposa l'ús d'un seguit de tècniques pel control a alt nivell d'un robot autònom i també per l'aprenentatge automàtic de comportaments. L'objectiu principal de la tesis fou el de dotar d'intel·ligència als robots autònoms que han d'acomplir unes missions determinades en entorns desconeguts i no estructurats. Una de les premisses tingudes en compte en tots els passos d'aquesta tesis va ser la selecció d'aquelles tècniques que poguessin ésser aplicades en temps real, i demostrar-ne el seu funcionament amb experiments reals. El camp d'aplicació de tots els experiments es la robòtica submarina. En una primera part, la tesis es centra en el disseny d'una arquitectura de control que ha de permetre l'assoliment d'una missió prèviament definida. En particular, la tesis proposa l'ús de les arquitectures de control basades en comportaments per a l'assoliment de cada una de les tasques que composen la totalitat de la missió. Una arquitectura d'aquest tipus està formada per un conjunt independent de comportaments, els quals representen diferents intencions del robot (ex.: "anar a una posició", "evitar obstacles",...). Es presenta una recerca bibliogràfica sobre aquest camp i alhora es mostren els resultats d'aplicar quatre de les arquitectures basades en comportaments més representatives a una tasca concreta. De l'anàlisi dels resultats se'n deriva que un dels factors que més influeixen en el rendiment d'aquestes arquitectures, és la metodologia emprada per coordinar les respostes dels comportaments. Per una banda, la coordinació competitiva és aquella en que només un dels comportaments controla el robot. Per altra banda, en la coordinació cooperativa el control del robot és realitza a partir d'una fusió de totes les respostes dels comportaments actius. La tesis, proposa un esquema híbrid d'arquitectura capaç de beneficiar-se dels principals avantatges d'ambdues metodologies. En una segona part, la tesis proposa la utilització de l'aprenentatge per reforç per aprendre l'estructura interna dels comportaments. Aquest tipus d'aprenentatge és adequat per entorns desconeguts i el procés d'aprenentatge es realitza al mateix temps que el robot està explorant l'entorn. La tesis presenta també un estat de l'art d'aquest camp, en el que es detallen els principals problemes que apareixen en utilitzar els algoritmes d'aprenentatge per reforç en aplicacions reals, com la robòtica. El problema de la generalització és un dels que més influeix i consisteix en permetre l'ús de variables continues sense augmentar substancialment el temps de convergència. Després de descriure breument les principals metodologies per generalitzar, la tesis proposa l'ús d'una xarxa neural combinada amb l'algoritme d'aprenentatge per reforç Q_learning. Aquesta combinació proporciona una gran capacitat de generalització i una molt bona disposició per aprendre en tasques de robòtica amb exigències de temps real. No obstant, les xarxes neurals són aproximadors de funcions no-locals, el que significa que en treballar amb un conjunt de dades no homogeni es produeix una interferència: aprendre en un subconjunt de l'espai significa desaprendre en la resta de l'espai. El problema de la interferència afecta de manera directa en robòtica, ja que l'exploració de l'espai es realitza sempre localment. L'algoritme proposat en la tesi té en compte aquest problema i manté una base de dades representativa de totes les zones explorades. Així doncs, totes les mostres de la base de dades s'utilitzen per actualitzar la xarxa neural, i per tant, l'aprenentatge és homogeni. Finalment, la tesi presenta els resultats obtinguts amb la arquitectura de control basada en comportaments i l'algoritme d'aprenentatge per reforç. Els experiments es realitzen amb el robot URIS, desenvolupat a la Universitat de Girona, i el comportament après és el seguiment d'un objecte mitjançant visió per computador. La tesi detalla tots els dispositius desenvolupats pels experiments així com les característiques del propi robot submarí. Els resultats obtinguts demostren la idoneïtat de les propostes en permetre l'aprenentatge del comportament en temps real. En un segon apartat de resultats es demostra la capacitat de generalització de l'algoritme d'aprenentatge mitjançant el "benchmark" del "cotxe i la muntanya". Els resultats obtinguts en aquest problema milloren els resultats d'altres metodologies, demostrant la millor capacitat de generalització de les xarxes neurals.

Modelling, control and supervision for a class of hybrid systems

The aim of this thesis is to narrow the gap between two different control techniques: the continuous control and the discrete event control techniques DES. This gap can be reduced by the study of Hybrid systems, and by interpreting as Hybrid systems the majority of large-scale systems. In particular, when looking deeply into a process, it is often possible to identify interaction between discrete and continuous signals. Hybrid systems are systems that have both continuous, and discrete signals. Continuous signals are generally supposed continuous and differentiable in time, since discrete signals are neither continuous nor differentiable in time due to their abrupt changes in time. Continuous signals often represent the measure of natural physical magnitudes such as temperature, pressure etc. The discrete signals are normally artificial signals, operated by human artefacts as current, voltage, light etc. Typical processes modelled as Hybrid systems are production systems, chemical process, or continuos production when time and continuous measures interacts with the transport, and stock inventory system. Complex systems as manufacturing lines are hybrid in a global sense. They can be decomposed into several subsystems, and their links. Another motivation for the study of Hybrid systems is the tools developed by other research domains. These tools benefit from the use of temporal logic for the analysis of several properties of Hybrid systems model, and use it to design systems and controllers, which satisfies physical or imposed restrictions. This thesis is focused in particular types of systems with discrete and continuous signals in interaction. That can be modelled hard non-linealities, such as hysteresis, jumps in the state, limit cycles, etc. and their possible non-deterministic future behaviour expressed by an interpretable model description. The Hybrid systems treated in this work are systems with several discrete states, always less than thirty states (it can arrive to NP hard problem), and continuous dynamics evolving with expression: with Ki ¡ Rn constant vectors or matrices for X components vector. In several states the continuous evolution can be several of them Ki = 0. In this formulation, the mathematics can express Time invariant linear system. By the use of this expression for a local part, the combination of several local linear models is possible to represent non-linear systems. And with the interaction with discrete events of the system the model can compose non-linear Hybrid systems. Especially multistage processes with high continuous dynamics are well represented by the proposed methodology. Sate vectors with more than two components, as third order models or higher is well approximated by the proposed approximation. Flexible belt transmission, chemical reactions with initial start-up and mobile robots with important friction are several physical systems, which profits from the benefits of proposed methodology (accuracy). The motivation of this thesis is to obtain a solution that can control and drive the Hybrid systems from the origin or starting point to the goal. How to obtain this solution, and which is the best solution in terms of one cost function subject to the physical restrictions and control actions is analysed. Hybrid systems that have several possible states, different ways to drive the system to the goal and different continuous control signals are problems that motivate this research. The requirements of the system on which we work is: a model that can represent the behaviour of the non-linear systems, and that possibilities the prediction of possible future behaviour for the model, in order to apply an supervisor which decides the optimal and secure action to drive the system toward the goal. Specific problems can be determined by the use of this kind of hybrid models are: - The unity of order. - Control the system along a reachable path. - Control the system in a safe path. - Optimise the cost function. - Modularity of control The proposed model solves the specified problems in the switching models problem, the initial condition calculus and the unity of the order models. Continuous and discrete phenomena are represented in Linear hybrid models, defined with defined eighth-tuple parameters to model different types of hybrid phenomena. Applying a transformation over the state vector : for LTI system we obtain from a two-dimensional SS a single parameter, alpha, which still maintains the dynamical information. Combining this parameter with the system output, a complete description of the system is obtained in a form of a graph in polar representation. Using Tagaki-Sugeno type III is a fuzzy model which include linear time invariant LTI models for each local model, the fuzzyfication of different LTI local model gives as a result a non-linear time invariant model. In our case the output and the alpha measure govern the membership function. Hybrid systems control is a huge task, the processes need to be guided from the Starting point to the desired End point, passing a through of different specific states and points in the trajectory. The system can be structured in different levels of abstraction and the control in three layers for the Hybrid systems from planning the process to produce the actions, these are the planning, the process and control layer. In this case the algorithms will be applied to robotics ¡V a domain where improvements are well accepted ¡V it is expected to find a simple repetitive processes for which the extra effort in complexity can be compensated by some cost reductions. It may be also interesting to implement some control optimisation to processes such as fuel injection, DC-DC converters etc. In order to apply the RW theory of discrete event systems on a Hybrid system, we must abstract the continuous signals and to project the events generated for these signals, to obtain new sets of observable and controllable events. Ramadge & Wonham¡¦s theory along with the TCT software give a Controllable Sublanguage of the legal language generated for a Discrete Event System (DES). Continuous abstraction transforms predicates over continuous variables into controllable or uncontrollable events, and modifies the set of uncontrollable, controllable observable and unobservable events. Continuous signals produce into the system virtual events, when this crosses the bound limits. If this event is deterministic, they can be projected. It is necessary to determine the controllability of this event, in order to assign this to the corresponding set, , controllable, uncontrollable, observable and unobservable set of events. Find optimal trajectories in order to minimise some cost function is the goal of the modelling procedure. Mathematical model for the system allows the user to apply mathematical techniques over this expression. These possibilities are, to minimise a specific cost function, to obtain optimal controllers and to approximate a specific trajectory. The combination of the Dynamic Programming with Bellman Principle of optimality, give us the procedure to solve the minimum time trajectory for Hybrid systems. The problem is greater when there exists interaction between adjacent states. In Hybrid systems the problem is to determine the partial set points to be applied at the local models. Optimal controller can be implemented in each local model in order to assure the minimisation of the local costs. The solution of this problem needs to give us the trajectory to follow the system. Trajectory marked by a set of set points to force the system to passing over them. Several ways are possible to drive the system from the Starting point Xi to the End point Xf. Different ways are interesting in: dynamic sense, minimum states, approximation at set points, etc. These ways need to be safe and viable and RchW. And only one of them must to be applied, normally the best, which minimises the proposed cost function. A Reachable Way, this means the controllable way and safe, will be evaluated in order to obtain which one minimises the cost function. Contribution of this work is a complete framework to work with the majority Hybrid systems, the procedures to model, control and supervise are defined and explained and its use is demonstrated. Also explained is the procedure to model the systems to be analysed for automatic verification. Great improvements were obtained by using this methodology in comparison to using other piecewise linear approximations. It is demonstrated in particular cases this methodology can provide best approximation. The most important contribution of this work, is the Alpha approximation for non-linear systems with high dynamics While this kind of process is not typical, but in this case the Alpha approximation is the best linear approximation to use, and give a compact representation.

Architecture for Living Neuronal Network Control of a mobile robot

The intelligent controlling mechanism of a typical mobile robot is usually a computer system. Research is however now ongoing in which biological neural networks are being cultured and trained to act as the brain of an interactive real world robot – thereby either completely replacing or operating in a cooperative fashion with a computer system. Studying such neural systems can give a distinct insight into biological neural structures and therefore such research has immediate medical implications. The principal aims of the present research are to assess the computational and learning capacity of dissociated cultured neuronal networks with a view to advancing network level processing of artificial neural networks. This will be approached by the creation of an artificial hybrid system (animat) involving closed loop control of a mobile robot by a dissociated culture of rat neurons. This paper details the components of the overall animat closed loop system architecture and reports on the evaluation of the results from preliminary real-life and simulated robot experiments.

Perceived controllability modulates the neural response to pain

The response to painful stimulation depends not only on peripheral nociceptive input but also on the cognitive and affective context in which pain occurs. One contextual variable that affects the neural and behavioral response to nociceptive stimulation is the degree to which pain is perceived to be controllable. Previous studies indicate that perceived controllability affects pain tolerance, learning and motivation, and the ability to cope with intractable pain, suggesting that it has profound effects on neural pain processing. To date, however, no neuroimaging studies have assessed these effects. We manipulated the subjects' belief that they had control over a nociceptive stimulus, while the stimulus itself was held constant. Using functional magnetic resonance imaging, we found that pain that was perceived to be controllable resulted in attenuated activation in the three neural areas most consistently linked with pain processing: the anterior cingulate, insular, and secondary somatosensory cortices. This suggests that activation at these sites is modulated by cognitive variables, such as perceived controllability, and that pain imaging studies may therefore overestimate the degree to which these responses are stimulus driven and generalizable across cognitive contexts. [References: 28]

A genetic algorithm for the design of a fuzzy controller for active queue management

Active queue management (AQM) policies are those policies of router queue management that allow for the detection of network congestion, the notification of such occurrences to the hosts on the network borders, and the adoption of a suitable control policy. This paper proposes the adoption of a fuzzy proportional integral (FPI) controller as an active queue manager for Internet routers. The analytical design of the proposed FPI controller is carried out in analogy with a proportional integral (PI) controller, which recently has been proposed for AQM. A genetic algorithm is proposed for tuning of the FPI controller parameters with respect to optimal disturbance rejection. In the paper the FPI controller design metodology is described and the results of the comparison with random early detection (RED), tail drop, and PI controller are presented.

Right ventromedial and dorsolateral prefrontal cortices mediate adaptive decisions under ambiguity by integrating choice utility and outcome evaluation.

The Iowa gambling task (IGT) is one of the most influential behavioral paradigms in reward-related decision making and has been, most notably, associated with ventromedial prefrontal cortex function. However, performance in the IGT relies on a complex set of cognitive subprocesses, in particular integrating information about the outcome of choices into a continuously updated decision strategy under ambiguous conditions. The complexity of the task has made it difficult for neuroimaging studies to disentangle the underlying neurocognitive processes. In this study, we used functional magnetic resonance imaging in combination with a novel adaptation of the task, which allowed us to examine separately activation associated with the moment of decision or the evaluation of decision outcomes. Importantly, using whole-brain regression analyses with individual performance, in combination with the choice/outcome history of individual subjects, we aimed to identify the neural overlap between areas that are involved in the evaluation of outcomes and in the progressive discrimination of the relative value of available choice options, thus mapping the two fundamental cognitive processes that lead to adaptive decision making. We show that activation in right ventromedial and dorsolateral prefrontal cortex was predictive of adaptive performance, in both discriminating disadvantageous from advantageous decisions and confirming negative decision outcomes. We propose that these two prefrontal areas mediate shifting away from disadvantageous choices through their sensitivity to accumulating negative outcomes. These findings provide functional evidence of the underlying processes by which these prefrontal subregions drive adaptive choice in the task, namely through contingency-sensitive outcome evaluation.

Genes related to sex steroids, neural growth, and social-emotional behavior are associated with autistic traits, empathy, and Asperger syndrome

Genetic studies of autism spectrum conditions (ASC) have mostly focused on the "low functioning" severe clinical subgroup, treating it as a rare disorder. However, ASC is now thought to be relatively common ( approximately 1%), and representing one end of a quasi-normal distribution of autistic traits in the general population. Here we report a study of common genetic variation in candidate genes associated with autistic traits and Asperger syndrome (AS). We tested single nucleotide polymorphisms in 68 candidate genes in three functional groups (sex steroid synthesis/transport, neural connectivity, and social-emotional responsivity) in two experiments. These were (a) an association study of relevant behavioral traits (the Empathy Quotient (EQ), the Autism Spectrum Quotient (AQ)) in a population sample (n=349); and (b) a case-control association study on a sample of people with AS, a "high-functioning" subgroup of ASC (n=174). 27 genes showed a nominally significant association with autistic traits and/or ASC diagnosis. Of these, 19 genes showed nominally significant association with AQ/EQ. In the sex steroid group, this included ESR2 and CYP11B1. In the neural connectivity group, this included HOXA1, NTRK1, and NLGN4X. In the socio-responsivity behavior group, this included MAOB, AVPR1B, and WFS1. Fourteen genes showed nominally significant association with AS. In the sex steroid group, this included CYP17A1 and CYP19A1. In the socio-emotional behavior group, this included OXT. Six genes were nominally associated in both experiments, providing a partial replication. Eleven genes survived family wise error rate (FWER) correction using permutations across both experiments, which is greater than would be expected by chance. CYP11B1 and NTRK1 emerged as significantly associated genes in both experiments, after FWER correction (P<0.05). This is the first candidate-gene association study of AS and of autistic traits. The most promising candidate genes require independent replication and fine mapping.

A fuzzy approach for the network congestion problem

In the recent years, the unpredictable growth of the Internet has moreover pointed out the congestion problem, one of the problems that historicallyha ve affected the network. This paper deals with the design and the evaluation of a congestion control algorithm which adopts a FuzzyCon troller. The analogyb etween Proportional Integral (PI) regulators and Fuzzycon trollers is discussed and a method to determine the scaling factors of the Fuzzycon troller is presented. It is shown that the Fuzzycon troller outperforms the PI under traffic conditions which are different from those related to the operating point considered in the design.

Parallel genetic algorithms for the tuning of a fuzzy AQM controller

This paper presents the results of the application of a parallel Genetic Algorithm (GA) in order to design a Fuzzy Proportional Integral (FPI) controller for active queue management on Internet routers. The Active Queue Management (AQM) policies are those policies of router queue management that allow the detection of network congestion, the notification of such occurrences to the hosts on the network borders, and the adoption of a suitable control policy. Two different parallel implementations of the genetic algorithm are adopted to determine an optimal configuration of the FPI controller parameters. Finally, the results of several experiments carried out on a forty nodes cluster of workstations are presented.

An adaptive group sequential design for phase II/III clinical trials that select a single treatment from several

There is increasing interest in combining Phases II and III of clinical development into a single trial in which one of a small number of competing experimental treatments is ultimately selected and where a valid comparison is made between this treatment and the control treatment. Such a trial usually proceeds in stages, with the least promising experimental treatments dropped as soon as possible. In this paper we present a highly flexible design that uses adaptive group sequential methodology to monitor an order statistic. By using this approach, it is possible to design a trial which can have any number of stages, begins with any number of experimental treatments, and permits any number of these to continue at any stage. The test statistic used is based upon efficient scores, so the method can be easily applied to binary, ordinal, failure time, or normally distributed outcomes. The method is illustrated with an example, and simulations are conducted to investigate its type I error rate and power under a range of scenarios.

Adaptive approximate Bayesian computation

Sequential techniques can enhance the efficiency of the approximate Bayesian computation algorithm, as in Sisson et al.'s (2007) partial rejection control version. While this method is based upon the theoretical works of Del Moral et al. (2006), the application to approximate Bayesian computation results in a bias in the approximation to the posterior. An alternative version based on genuine importance sampling arguments bypasses this difficulty, in connection with the population Monte Carlo method of Cappe et al. (2004), and it includes an automatic scaling of the forward kernel. When applied to a population genetics example, it compares favourably with two other versions of the approximate algorithm.

The role of Wnt signalling in the development of somites and neural crest

The Wnt family of secreted signalling molecules controls a wide range of developmental processes in all metazoans. In this investigation we concentrate on the role that members of this family play during the development of (1) the somites and (2) the neural crest. (3) We also isolate a novel component of the Wnt signalling pathway called Naked cuticle and investigate the role that this protein may play in both of the previously mentioned developmental processes. (1) In higher vertebrates the paraxial mesoderm undergoes a mesenchymal-to-epithelial transformation to form segmentally organised structures called somites. Experiments have shown that signals originating from the ectoderm overlying the somites or from midline structures are required for the formation of the somites, but their identity has yet to be determined. Wnt6 is a good candidate as a somite epithelialisation factor from the ectoderm since it is expressed in this tissue. In this study we show that injection of Wnt6-producing cells beneath the ectoderm at the level of the segmental plate or lateral to the segmental plate leads to the formation of numerous small epithelial somites. We show that Wnts are indeed responsible for the epithelialisation of somites by applying Wnt antagonists which result in the segmental plate being unable to form somites. These results show that Wnt6, the only member of this family to be localised to the chick paraxial ectoderm, is able to regulate the development of epithelial somites and that cellular organisation is pivotal in the execution of the differentiation programmes. (2) The neural crest is a population of multipotent progenitor cells that arise from the neural ectoderm in all vertebrate embryos and form a multitude of derivatives including the peripheral sensory neurons, the enteric nervous system, Schwann cells, pigment cells and parts of the craniofacial skeleton. The induction of the neural crest relies on an ectodermally derived signal, but the identity of the molecule performing this role in amniotes is not known. Here we show that Wnt6, a protein expressed in the ectoderm, induces neural crest production. (3) The intracellular response to Wnt signalling depends on the choice of signalling cascade activated in the responding cell. Cells can activate either the canonical pathway that modulates gene expression to control cellular differentiation and proliferation, or the non-canonical pathway that controls cell polarity and movement (Pandur et al. 2002b). Recent work has identified the protein Naked cuticle as an intracellular switch promoting the non-canonical pathway at the expense of the canonical pathway. We have cloned chick Naked cuticle-1 (cNkd1) and demonstrate that it is expressed in a dynamic manner during early embryogenesis. We show that it is expressed in the somites and in particular regions where cells are undergoing movement. Lastly our study shows that the expression of cNkd1 is regulated by Wnt expression originating from the neural tube. This study provides evidence that non-canonical Wnt signalling plays a part in somite development.

Designs for an adaptive tuned vibration absorber with variable shape stiffness element

An adaptive tuned vibration absorber (ATVA) with a smart variable stiffness element is capable of retuning itself in response to a time-varying excitation frequency., enabling effective vibration control over a range of frequencies. This paper discusses novel methods of achieving variable stiffness in an ATVA by changing shape, as inspired by biological paradigms. It is shown that considerable variation in the tuned frequency can be achieved by actuating a shape change, provided that this is within the limits of the actuator. A feasible design for such an ATVA is one in which the device offers low resistance to the required shape change actuation while not being restricted to low values of the effective stiffness of the vibration absorber. Three such original designs are identified: (i) A pinned-pinned arch beam with fixed profile of slight curvature and variable preload through an adjustable natural curvature; (ii) a vibration absorber with a stiffness element formed from parallel curved beams of adjustable curvature vibrating longitudinally; (iii) a vibration absorber with a variable geometry linkage as stiffness element. The experimental results from demonstrators based on two of these designs show good correlation with the theory.

A fuzzy multi-criteria decision approach for enhanced auto-tracking of seismic events

The main activity carried out by the geophysicist when interpreting seismic data, in terms of both importance and time spent is tracking (or picking) seismic events. in practice, this activity turns out to be rather challenging, particularly when the targeted event is interrupted by discontinuities such as geological faults or exhibits lateral changes in seismic character. In recent years, several automated schemes, known as auto-trackers, have been developed to assist the interpreter in this tedious and time-consuming task. The automatic tracking tool available in modem interpretation software packages often employs artificial neural networks (ANN's) to identify seismic picks belonging to target events through a pattern recognition process. The ability of ANNs to track horizons across discontinuities largely depends on how reliably data patterns characterise these horizons. While seismic attributes are commonly used to characterise amplitude peaks forming a seismic horizon, some researchers in the field claim that inherent seismic information is lost in the attribute extraction process and advocate instead the use of raw data (amplitude samples). This paper investigates the performance of ANNs using either characterisation methods, and demonstrates how the complementarity of both seismic attributes and raw data can be exploited in conjunction with other geological information in a fuzzy inference system (FIS) to achieve an enhanced auto-tracking performance.

Preliminary results from a real time control system using optical information

Successful results from training an adaptive controller to use optical information to balance an inverted pendulum are presented in comparison to the training requirements using traditional controller inputs. Results from research into the psychology of the sense of balance in humans are presented as the motivation for the investigation of this new type of controller. The simulated model of the inverted pendulum and the virtual reality environments used to provide the optical input are described The successful introduction of optical information is found to require the preservation of at least two of the traditional input types and entail increased training time for the adaptive controller and reduced performance (measured as the time the pendulum remains upright).