Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Unlinked Embedded Graphs

This paper is a self-contained development of an invariant of graphs embedded in three-dimensional Euclidean space using the Jones polynomial and skein theory. Some examples of the invariant are computed. An unlinked embedded graph is one that contains only trivial knots or links. Examples show that the invariant is sufficiently powerful to distinguish some different unlinked embeddings of the same graph.

Decision making models embedded into a web-based tool for assessing pest infestation risk

Current practices in agricultural management involve the application of rules and techniques to ensure high quality and environmentally friendly production. Based on their experience, agricultural technicians and farmers make critical decisions affecting crop growth while considering several interwoven agricultural, technological, environmental, legal and economic factors. In this context, decision support systems and the knowledge models that support them, enable the incorporation of valuable experience into software systems providing support to agricultural technicians to make rapid and effective decisions for efficient crop growth. Pest control is an important issue in agricultural management due to crop yield reductions caused by pests and it involves expert knowledge. This paper presents a formalisation of the pest control problem and the workflow followed by agricultural technicians and farmers in integrated pest management, the crop production strategy that combines different practices for growing healthy crops whilst minimising pesticide use. A generic decision schema for estimating infestation risk of a given pest on a given crop is defined and it acts as a metamodel for the maintenance and extension of the knowledge embedded in a pest management decision support system which is also presented. This software tool has been implemented by integrating a rule-based tool into web-based architecture. Evaluation from validity and usability perspectives concluded that both agricultural technicians and farmers considered it a useful tool in pest control, particularly for training new technicians and inexperienced farmers.

Multirobot Tethering for Localization and Control

Particle filtering has proven to be an effective localization method for wheeled autonomous vehicles. For a given map, a sensor model, and observations, occasions arise where the vehicle could equally likely be in many locations of the map. Because particle filtering algorithms may generate low confidence pose estimates under these conditions, more robust localization strategies are required to produce reliable pose estimates. This becomes more critical if the state estimate is an integral part of system control. We investigate the use of particle filter estimation techniques on a hovercraft vehicle. The marginally stable dynamics of a hovercraft require reliable state estimates for proper stability and control. We use the Monte Carlo localization method, which implements a particle filter in a recursive state estimate algorithm. An H-infinity controller, designed to accommodate the latency inherent in our state estimation, provides stability and controllability to the hovercraft. In order to eliminate the low confidence estimates produced in certain environments, a multirobot system is designed to introduce mobile environment features. By tracking and controlling the secondary robot, we can position the mobile feature throughout the environment to ensure a high confidence estimate, thus maintaining stability in the system. A laser rangefinder is the sensor the hovercraft uses to track the secondary robot, observe the environment, and facilitate successful localization and stability in motion.

PREPARATION OF POLYELECTROLYTE MEMBRANES EMBEDDED WITH ZEOLITE NANOPARTICLES FOR ENHANCED PERFORMANCE IN FORWARD OSMOSIS

Water scarcity is a global issue that has already affected every continent. Membrane technology is considered as one of the most promising candidates for resolving this worsening issue. Among all the membrane processes, the emerging forward osmosis (FO) membrane process is osmotically-driven and has unique advantages compared with other traditional pressure-driven membrane processes. One of the major challenges to advancing the FO membrane process is the lack of a suitable membrane. Polyelectrolyte thin film prepared via layer-by-layer (LbL) technique has demonstrated its excellent performance in many applications including electronics, optics, sensors, etc. Recent studies have revealed the potential of polyelectrolyte thin films in acting as the active separation layer of FO membranes, but significant efforts are still needed to improve the membrane performance and understand the transport mechanisms. This dissertation introduces a novel approach to prepare a zeolite-embedded polyelectrolyte composite membrane for enhanced FO performance. This membrane takes advantages of the versatile LbL process to unprecedentedly incorporate high loading of zeolite nanoparticles, which are anticipated to facilitate water transport due to the uniquely interconnected structure of zeolites. Major topics discussed in this dissertation include: (1) the synthesis and evaluation of the polyelectrolyte-zeolite composite FO membrane, (2) the examination of the fouling resistance to identify its technical limitations, (3) the demonstration of the membrane regenerability as an effective strategy for membrane fouling control, and (4) the investigation of crosslinking effects on the membrane performance to elucidate the transport mechanisms involved in the zeolite-embedded polyelectrolyte membranes. Comparative studies have been made between polyelectrolyte membranes with and without zeolite incorporation. The findings suggest that the zeolite-embedded membrane, although slightly more susceptible to silica scaling, has demonstrated enhanced water flux and separation capability, good resistance to organic fouling, and complete regenerability for fouling control. Additionally, the embedded zeolite nanoparticles are proved to be able to create fast pathways for water transport. Overall, this work provides a novel strategy to create zeolite-polymer composite membranes with enhanced separation performance and unique fouling mitigation properties.

Evaluating alternative gait strategies using evolutionary robotics

Evolutionary robitics is a branch of artificial intelligence concerned with the automatic generation of autonomous robots. Usually the form of the robit is predefined an various computational techniques are used to control the machine's behaviour. One aspect is the spontaneous generation of walking in legged robots and this can be used to investigate the mechanical requiements for efficient walking in bipeds. This paper demonstrates a bipedal simulator that spontaneously generates walking and running gaits. The model can be customized to represent a range of hominoid morphologies and used to predict performance paramets such as preferred speed and metabolic energy cost. Because it does not require any motion capture data it is particularly suitable for investigating locomotion in fossil animals. The predictoins for modern humans are highly accurate in terms of energy cost for a given speend and thus the values predicted for other bipeds are likely to be good estimates. To illustrate this the cost of transport is calculated for Australopithecus afarensis. The model allows the degree of maximum extension at the knee to be varied causing the model to adopt walking gaits varying from chimpanzee-like to human=like. The energy costs associated with these gait choices can thus be calculated and this information used to evaluate possible locomotor strategies in early hominids

Demonstrador para rede tempo-real HaRTES

A utilização de sistemas embutidos distribuídos em diversas áreas como a robótica, automação industrial e aviónica tem vindo a generalizar-se no decorrer dos últimos anos. Este tipo de sistemas são compostos por vários nós, geralmente designados por sistemas embutidos. Estes nós encontram-se interligados através de uma infra-estrutura de comunicação de forma a possibilitar a troca de informação entre eles de maneira a concretizar um objetivo comum. Por norma os sistemas embutidos distribuídos apresentam requisitos temporais bastante exigentes. A tecnologia Ethernet e os protocolos de comunicação, com propriedades de tempo real, desenvolvidos para esta não conseguem associar de uma forma eficaz os requisitos temporais das aplicações de tempo real aos requisitos Quality of Service (QoS) dos diferentes tipos de tráfego. O switch Hard Real-Time Ethernet Switching (HaRTES) foi desenvolvido e implementado com o objetivo de solucionar estes problemas devido às suas capacidades como a sincronização de fluxos diferentes e gestão de diferentes tipos de tráfego. Esta dissertação apresenta a adaptação de um sistemas físico de modo a possibilitar a demonstração do correto funcionamento do sistema de comunicação, que será desenvolvido e implementado, utilizando um switch HaRTES como o elemento responsável pela troca de informação na rede entre os nós. O desempenho da arquitetura de rede desenvolvida será também testada e avaliada.

Phobos: The design and implementation of embedded software for a low cost radar warning receiver

This portfolio thesis describes work undertaken by the author under the Engineering Doctorate program of the Institute for System Level Integration. It was carried out in conjunction with the sponsor company Teledyne Defence Limited. A radar warning receiver is a device used to detect and identify the emissions of radars. They were originally developed during the Second World War and are found today on a variety of military platforms as part of the platform’s defensive systems. Teledyne Defence has designed and built components and electronic subsystems for the defence industry since the 1970s. This thesis documents part of the work carried out to create Phobos, Teledyne Defence’s first complete radar warning receiver. Phobos was designed to be the first low cost radar warning receiver. This was made possible by the reuse of existing Teledyne Defence products, commercial off the shelf hardware and advanced UK government algorithms. The challenges of this integration are described and discussed, with detail given of the software architecture and the development of the embedded application. Performance of the embedded system as a whole is described and qualified within the context of a low cost system.

Predicting the metabolic energy costs of bipedalism using evolutionary robotics

To understand the evolution of bipedalism among the homnoids in an ecological context we need to be able to estimate theenerrgetic cost of locomotion in fossil forms. Ideally such an estimate would be based entirely on morphology since, except for the rare instances where footprints are preserved, this is hte only primary source of evidence available. In this paper we use evolutionary robotics techniques (genetic algoritms, pattern generators and mechanical modeling) to produce a biomimentic simulation of bipedalism based on human body dimensions. The mechnaical simulation is a seven-segment, two-dimensional model with motive force provided by tension generators representing the major muscle groups acting around the lower-limb joints. Metabolic energy costs are calculated from the muscel model, and bipedal gait is generated using a finite-state pattern generator whose parameters are produced using a genetic algorithm with locomotor economy (maximum distance for a fixed energy cost) as the fitness criterion. The model is validated by comparing the values it generates with those for modern humans. The result (maximum efficiency of 200 J m-1) is within 15% of the experimentally derived value, which is very encouraging and suggests that this is a useful analytic technique for investigating the locomotor behaviour of fossil forms. Initial work suggests that in the future this technique could be used to estimate other locomotor parameters such as top speed. In addition, the animations produced by this technique are qualitatively very convincing, which suggests that this may also be a useful technique for visualizing bipedal locomotion.

Microstructure and tensile properties of carbon-epoxy laminates produced by automated fibre placement: Influence of a caul plate on the effects of gap and overlap embedded defects

Automated fibre placement (AFP) enables the trajectory of unidirectional composite tape to be optimized, but laying down complex shapes with this technology can result in the introduction of defects. The aim of this experimental study is to investigate the influence of gaps and overlaps on the microstructure and tensile properties of carbon-epoxy laminates. First, a comparison between a hand-layup and AFP layup, draped and cured under the same conditions, shows equivalent microstructures and tensile properties. This provides the reference values for the study. Then, gap and overlap embedded defects (more or less severe) are introduced during manufacturing, on two cross-ply layups [(0°/(90°)5/0°] and [(90°/0°)2/90°]. Autoclave cure without a caul plate results in local thickness variation and microstructural changes which depend on the defect type. This has a strong influence on mechanical performance. Use of a caul plate avoids these variations and in this case embedded defects hardly affect tensile properties.

QuasiGeostrophic Diagnosis of Mixed-Layer Dynamics Embedded in a Mesoscale Turbulent Field

A quasigeostrophic model is developed to diagnose the three-dimensional circulation, including the vertical velocity, in the upper ocean from high-resolution observations of sea surface height and buoyancy. The formulation for the adiabatic component departs from the classical surface quasigeostrophic framework considered before since it takes into account the stratification within the surface mixed layer that is usually much weaker than that in the ocean interior. To achieve this, the model approximates the ocean with two constant stratification layers: a finite-thickness surface layer (or the mixed layer) and an infinitely deep interior layer. It is shown that the leading-order adiabatic circulation is entirely determined if both the surface streamfunction and buoyancy anomalies are considered. The surface layer further includes a diabatic dynamical contribution. Parameterization of diabatic vertical velocities is based on their restoring impacts of the thermal wind balance that is perturbed by turbulent vertical mixing of momentum and buoyancy. The model skill in reproducing the three-dimensional circulation in the upper ocean from surface data is checked against the output of a high-resolution primitive equation numerical simulation

Leader Based Cyclic Pursuit

In this work a system of autonomous agents engaged in cyclic pursuit (under constant bearing (CB) strategy) is considered, for which one informed agent (the leader) also senses and responds to a stationary beacon. Building on the framework proposed in a previous work on beacon-referenced cyclic pursuit, necessary and suffi- cient conditions for the existence of circling equilibria in a system with one informed agent are derived, with discussion of stability and performance. In a physical testbed, the leader (robot) is equipped with a sound sensing apparatus composed of a real time embedded system, estimating direction of arrival of sound by an Interaural Level and Phase Difference Algorithm, using empirically determined phase and level signatures, and breaking front-back ambiguity with appropriate sensor placement. Furthermore a simple framework for implementing and evaluating the performance of control laws with the Robot Operating System (ROS) is proposed, demonstrated, and discussed.

Types of embedded ties in buyer-supplier relationships and their combined effects on innovation performance

[EN] Purpose of the paper - This research analyzes the impact of three types of embedded ties, namely, specialized complementary resources, idiosyncratic investments, and knowledge sharing, on the innovation capacity of the firms. We also study the particularities of the Machine-Tool industry. Theoretical background – Our evaluation of the embedded buyer-supplier ties is based on the potential sources of relational rents proposed by Dyer and Sing (1998). We also draw on Uzzi and Lancaster (2003), Noordhoff et al. (2011), among others, to discuss the positive and negative aspects of embedded ties. Design/Methodology/Approach ‐ Using data from a survey of 202 European machine-tool firms acting as buyers and sellers, we propose and evaluate a Structural Equation model. Findings ‐ Only knowledge-sharing routines exert a significant positive effect on product innovation performance. Neither an increase in the idiosyncratic investments nor in complementary resources and capabilities enhance innovation performance. Also, knowledge-sharing routines mediate in the effect from idiosyncratic investments on innovation performance. Research Limitations. ‐ The machine tool industry has unique characteristics that make this generalization difficult. Also, there is considerable difficulty associated with testing more deeply the interrelations among these embedded ties in the long run. It is plausible to understand that these interrelations operate within a gradual process. Originality/Value/Contribution of Paper ‐ This research contributes to a better understanding of the role of embedded ties on innovativeness. To the best of our knowledge, there is no previous international empirical research analyzing the mediation effects among specialized complementary resources, idiosyncratic investments and knowledge sharing, and their effects on the innovation capacity of firms.

Cultural Heritage and Mobile Robotics Technologies

This paper reviews current research works at the authors’ Institutions to illustrate how mobile robotics and related technologies can be used to enhance economical fruition, control, protection and social impact of the cultural heritage. Robots allow experiencing on-line, from remote locations, tours at museums, archaeological areas and monuments. These solutions avoid travelling costs, increase beyond actual limits the number of simultaneous visitors, and prevent possible damages that can arise by over-exploitation of fragile environments. The same tools can be used for exploration and monitoring of cultural artifacts located in difficult to reach or dangerous areas. Examples are provided by the use of underwater robots in the exploration of deeply submerged archaeological areas. Besides, technologies commonly employed in robotics can be used to help exploring, monitoring and preserving cultural artifacts. Examples are provided by the development of procedures for data acquisition and mapping and by object recognition and monitoring algorithms.