A proposal of a behavior-based control architecture with reinforcement learning for an autonomous underwater robot

Aquesta tesi proposa l'ús d'un seguit de tècniques pel control a alt nivell d'un robot autònom i també per l'aprenentatge automàtic de comportaments. L'objectiu principal de la tesis fou el de dotar d'intel·ligència als robots autònoms que han d'acomplir unes missions determinades en entorns desconeguts i no estructurats. Una de les premisses tingudes en compte en tots els passos d'aquesta tesis va ser la selecció d'aquelles tècniques que poguessin ésser aplicades en temps real, i demostrar-ne el seu funcionament amb experiments reals. El camp d'aplicació de tots els experiments es la robòtica submarina. En una primera part, la tesis es centra en el disseny d'una arquitectura de control que ha de permetre l'assoliment d'una missió prèviament definida. En particular, la tesis proposa l'ús de les arquitectures de control basades en comportaments per a l'assoliment de cada una de les tasques que composen la totalitat de la missió. Una arquitectura d'aquest tipus està formada per un conjunt independent de comportaments, els quals representen diferents intencions del robot (ex.: "anar a una posició", "evitar obstacles",...). Es presenta una recerca bibliogràfica sobre aquest camp i alhora es mostren els resultats d'aplicar quatre de les arquitectures basades en comportaments més representatives a una tasca concreta. De l'anàlisi dels resultats se'n deriva que un dels factors que més influeixen en el rendiment d'aquestes arquitectures, és la metodologia emprada per coordinar les respostes dels comportaments. Per una banda, la coordinació competitiva és aquella en que només un dels comportaments controla el robot. Per altra banda, en la coordinació cooperativa el control del robot és realitza a partir d'una fusió de totes les respostes dels comportaments actius. La tesis, proposa un esquema híbrid d'arquitectura capaç de beneficiar-se dels principals avantatges d'ambdues metodologies. En una segona part, la tesis proposa la utilització de l'aprenentatge per reforç per aprendre l'estructura interna dels comportaments. Aquest tipus d'aprenentatge és adequat per entorns desconeguts i el procés d'aprenentatge es realitza al mateix temps que el robot està explorant l'entorn. La tesis presenta també un estat de l'art d'aquest camp, en el que es detallen els principals problemes que apareixen en utilitzar els algoritmes d'aprenentatge per reforç en aplicacions reals, com la robòtica. El problema de la generalització és un dels que més influeix i consisteix en permetre l'ús de variables continues sense augmentar substancialment el temps de convergència. Després de descriure breument les principals metodologies per generalitzar, la tesis proposa l'ús d'una xarxa neural combinada amb l'algoritme d'aprenentatge per reforç Q_learning. Aquesta combinació proporciona una gran capacitat de generalització i una molt bona disposició per aprendre en tasques de robòtica amb exigències de temps real. No obstant, les xarxes neurals són aproximadors de funcions no-locals, el que significa que en treballar amb un conjunt de dades no homogeni es produeix una interferència: aprendre en un subconjunt de l'espai significa desaprendre en la resta de l'espai. El problema de la interferència afecta de manera directa en robòtica, ja que l'exploració de l'espai es realitza sempre localment. L'algoritme proposat en la tesi té en compte aquest problema i manté una base de dades representativa de totes les zones explorades. Així doncs, totes les mostres de la base de dades s'utilitzen per actualitzar la xarxa neural, i per tant, l'aprenentatge és homogeni. Finalment, la tesi presenta els resultats obtinguts amb la arquitectura de control basada en comportaments i l'algoritme d'aprenentatge per reforç. Els experiments es realitzen amb el robot URIS, desenvolupat a la Universitat de Girona, i el comportament après és el seguiment d'un objecte mitjançant visió per computador. La tesi detalla tots els dispositius desenvolupats pels experiments així com les característiques del propi robot submarí. Els resultats obtinguts demostren la idoneïtat de les propostes en permetre l'aprenentatge del comportament en temps real. En un segon apartat de resultats es demostra la capacitat de generalització de l'algoritme d'aprenentatge mitjançant el "benchmark" del "cotxe i la muntanya". Els resultats obtinguts en aquest problema milloren els resultats d'altres metodologies, demostrant la millor capacitat de generalització de les xarxes neurals.

Memory-based Embodied Cognition: a computational architecture

At its most fundamental, cognition as displayed by biological agents (such as humans) may be said to consist of the manipulation and utilisation of memory. Recent discussions in the field of cognitive robotics have emphasised the role of embodiment and the necessity of a value or motivation for autonomous behaviour. This work proposes a computational architecture – the Memory-Based Cognitive (MBC) architecture – based upon these considerations for the autonomous development of control of a simple mobile robot. This novel architecture will permit the exploration of theoretical issues in cognitive robotics and animal cognition. Furthermore, the biological inspiration of the architecture is anticipated to result in a mobile robot controller which displays adaptive behaviour in unknown environments.

A genetic algorithm for the design of a fuzzy controller for active queue management

Active queue management (AQM) policies are those policies of router queue management that allow for the detection of network congestion, the notification of such occurrences to the hosts on the network borders, and the adoption of a suitable control policy. This paper proposes the adoption of a fuzzy proportional integral (FPI) controller as an active queue manager for Internet routers. The analytical design of the proposed FPI controller is carried out in analogy with a proportional integral (PI) controller, which recently has been proposed for AQM. A genetic algorithm is proposed for tuning of the FPI controller parameters with respect to optimal disturbance rejection. In the paper the FPI controller design metodology is described and the results of the comparison with random early detection (RED), tail drop, and PI controller are presented.

Parallel genetic algorithms for the tuning of a fuzzy AQM controller

This paper presents the results of the application of a parallel Genetic Algorithm (GA) in order to design a Fuzzy Proportional Integral (FPI) controller for active queue management on Internet routers. The Active Queue Management (AQM) policies are those policies of router queue management that allow the detection of network congestion, the notification of such occurrences to the hosts on the network borders, and the adoption of a suitable control policy. Two different parallel implementations of the genetic algorithm are adopted to determine an optimal configuration of the FPI controller parameters. Finally, the results of several experiments carried out on a forty nodes cluster of workstations are presented.

Low molecular weight heparin significantly reduces embolisation after carotid endarterectomy - a randomised controller trial

Objectives The administration of unfractionated heparin (UFH) prior to carotid clamping during carotid endarterectomy (CEA) transiently increases the platelet aggregation response to arachidonic acid (AA) despite the use of aspirin. We hypothesized that this phenomenon might be reduced by using low molecular weight heparin (LMWH) resulting in fewer emboli in the early post-operative period. Methods 183 aspirinated patients undergoing CEA were randomised to 5000 IU UFH (n = 91) or 2500 IU LMWH (dalteparin, n = 92) prior to carotid clamping. End-points were: transcranial Doppler (TCD) measurement of embolisation, effect on bleeding and platelet aggregation to AA and adenosine 5′-diphosphate (ADP). Results Patients randomised to UFH had twice the odds of experiencing a higher number of emboli in the first 3 h after CEA, than those randomised to LMWH (p = 0.04). This was not associated with increased bleeding (mean time from flow restoration to operation end: 23 min (UFH) vs. 24 min (LMWH), p = 0.18). Platelet aggregation to AA increased significantly following heparinisation, but was unaffected by heparin type (p = 0.90). The platelets of patients randomised to LMWH exhibited significantly lower aggregation to ADP compared to UFH (p < 0.0001). Conclusions Intravenous LMWH is associated with a significant reduction in post-operative embolisation without increased bleeding. The higher rate of embolisation seen with UFH may be mediated by increased platelet aggregation to ADP, rather than to AA.

The effects of mass loading the ossicles with a floating mass transducer on middle ear transfer function

Hypothesis: The aim of this study was to measure the mass loading effect of an active middle-ear implant (the Vibrant Soundbridge) in cadaver temporal bones. Background: Implantable middle ear hearing devices such as Vibrant Soundbridge have been used as an alternative to conventional hearing aids for the rehabilitation of sensorineural hearing loss. Other than the obvious disadvantage of requiring implantation middle ear surgery, it also applies a direct weight on the ossicular chain which, in turn, may have an impact on residual hearing. Previous studies have shown that applying a mass directly on the ossicular chain has a damping effect on its response to sound. However, little has been done to investigate the magnitude and the frequency characteristics of the mass loading effect in devices such as the Vibrant Soundbridge. Methods: Five fresh cadaver temporal bones were used. The stapes displacement was measured using laser Doppler vibrometry before and after the placement of a Vibrant Sound-bridge floating mass transducer. The effects of mass and attachment site were compared with the unloaded response. Measurements were obtained at frequencies between 0.1 and 10 kHz and at acoustic input levels of 100 dB sound pressure level. Each temporal bone acted as its own control. Results: Placement of the floating mass transducer caused a reduction of the stapes displacement. There were variations between the bones. The change of the stapes displacement varied from 0 dB to 28 dB. The effect was more prominent at frequencies above 1,000 Hz. Placing the floating mass transducer close to the incudostapedial joint reduced the mass loading effect. Conclusion: The floating mass transducer produces a measurable reduction of the stapes displacement in the temporal bone model. The effect is more prominent at high frequencies.

Investigation of the fracture mode for hard and soft wheat endosperm using the loading-unloading bending test

Investigation of the fracture mode for hard and soft wheat endosperm was aimed at gaining a better understanding of the fragmentation process. Fracture mechanical characterization was based on the three-point bending test which enables stable crack propagation to take place in small rectangular pieces of wheat endosperm. The crack length can be measured in situ by using an optical microscope with light illumination from the side of the specimen or from the back of the specimen. Two new techniques were developed and used to estimate the fracture toughness of wheat endosperm, a geometric approach and a compliance method. The geometric approach gave average fracture toughness values of 53.10 and 27.0 J m(-2) for hard and soft endosperm, respectively. Fracture toughness estimated using the compliance method gave values of 49.9 and 29.7 J m(-2) for hard and soft endosperm, respectively. Compressive properties of the endosperm in three mutually perpendicular axes revealed that the hard and soft endosperms are isotropic composites. Scanning electron microscopy (SEM) observation of the fracture surfaces and the energy-time curves of loading-unloading cycles revealed that there was a plastic flow during crack propagation for both the hard and soft endosperms, and confirmed that the fracture mode is significantly related to the adhesion level between starch granules and the protein matrix.

Autonomous execution of tasks by swarm carrier agents in swarm-array computing

How can a bridge be built between autonomic computing approaches and parallel computing system? The work reported in this paper is motivated towards bridging this gap by proposing swarm-array computing, a novel technique to achieve autonomy for distributed parallel computing systems. Among three proposed approaches, the second approach, namely 'Intelligent Agents' is of focus in this paper. The task to be executed on parallel computing cores is considered as a swarm of autonomous agents. A task is carried to a computing core by carrier. agents and can be seamlessly transferred between cores in the event of a pre-dicted failure, thereby achieving self-ware objectives of autonomic computing. The feasibility of the proposed approach is validated on a multi-agent simulator.

Adaptation of an commercially available stabilised R/C helicopter to a fully autonomous surveillance UAV

This paper presents the development of an autonomous surveillance UAV that competed in the Ministry of Defence Grand Challenge 2008. In order to focus on higher-level mission control, the UAV is built upon an existing commercially available stabilised R/C helicopter platform. The hardware architecture is developed to allow for non-invasion integration with the existing stabilised platform, and to enable to the distributed processing of closed loop control and mission goals. The resulting control system proved highly successful and was capable of flying within 40knott gusts. The software and safety architectures were key to the success of the research and also hold the potential for use in the development of more complex system comprising of multiple UAVs.