Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Dynamic set pieces for CAMBADA team

Robotics is an emergent branch of engineering that involves the conception, manufacture, and control of robots. It is a multidisciplinary field that combines electronics, design, computer science, artificial intelligence, mechanics and nanotechnology. Its evolution results in machines that are able to perform tasks with some level of complexity. Multi-agent systems is a researching topic within robotics, thus they allow the solving of higher complexity problems, through the execution of simple routines. Robotic soccer allows the study and development of robotics and multiagent systems, as the agents have to work together as a team, having in consideration most problems found in our quotidian, as for example adaptation to a highly dynamic environment as it is the one of a soccer game. CAMBADA is the robotic soccer team belonging to the group of research IRIS from IEETA, composed by teachers, researchers and students of the University of Aveiro, which annually has as main objective the participation in the RoboCup, in the Middle Size League. The purpose of this work is to improve the coordination in set pieces situations. This thesis introduces a new behavior and the adaptation of the already existing ones in the offensive situation, as well as the proposal of a new positioning method in defensive situations. The developed work was incorporated within the competition software of the robots. Which allows the presentation, in this dissertation, of the experimental results obtained, through simulation software as well as through the physical robots on the laboratory.

Demonstrador para rede tempo-real HaRTES

A utilização de sistemas embutidos distribuídos em diversas áreas como a robótica, automação industrial e aviónica tem vindo a generalizar-se no decorrer dos últimos anos. Este tipo de sistemas são compostos por vários nós, geralmente designados por sistemas embutidos. Estes nós encontram-se interligados através de uma infra-estrutura de comunicação de forma a possibilitar a troca de informação entre eles de maneira a concretizar um objetivo comum. Por norma os sistemas embutidos distribuídos apresentam requisitos temporais bastante exigentes. A tecnologia Ethernet e os protocolos de comunicação, com propriedades de tempo real, desenvolvidos para esta não conseguem associar de uma forma eficaz os requisitos temporais das aplicações de tempo real aos requisitos Quality of Service (QoS) dos diferentes tipos de tráfego. O switch Hard Real-Time Ethernet Switching (HaRTES) foi desenvolvido e implementado com o objetivo de solucionar estes problemas devido às suas capacidades como a sincronização de fluxos diferentes e gestão de diferentes tipos de tráfego. Esta dissertação apresenta a adaptação de um sistemas físico de modo a possibilitar a demonstração do correto funcionamento do sistema de comunicação, que será desenvolvido e implementado, utilizando um switch HaRTES como o elemento responsável pela troca de informação na rede entre os nós. O desempenho da arquitetura de rede desenvolvida será também testada e avaliada.