Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Development of variable displacement oil pump for internal combustion engines

In the last years the need to develop more environmentally friendly and efficient cars as led to the development of several technologies to improve the performance of internal combustion engines, a large part of the innovations are focused in the auxiliary systems of the engine, including, the oil pump, this is an element of great importance in the dynamics of the engine as well a considerable energy consumer. Most solutions for oil pumps to this day are fixed displacement, for medium and high speeds, the pump flow rate is higher than the needs of the engine, this excess flow leads to the need for recirculation of the fluid which represents a waste of energy. Recently, technological advances in this area have led to the creation of variable displacement oil pumps, these have become a 'must have' due to the numerous advantages they bring, although the working principle of vane or piston pumps is relatively well known, the application of this technology for the automotive industry is new and brings new challenges. The focus of this dissertation is to develop a new concept of variable displacement system for automotive oil pumps. The main objective is to obtain a concept that is totally adaptable to existing solutions on the market (engines), both dimensionally as in performance specifications, having at the same time an innovative mechanical system for obtaining variable displacement. The developed design is a vane pump with variable displacement going in line with existing commercial solutions, however, the variation of the eccentricity commonly used to provide an variable displacement delivery is not used, the variable displacement is achieved without varying the eccentricity of the system but with a variation of the length of the pumping chamber. The principle of operation of the pump is different to existing solutions while maintaining the ability to integrate standard parts such as control valves and mechanical safety valves, the pump is compatible with commercial solutions in terms of interfaces for connection between engine systems and pump. A concept prototype of the product was obtained in order to better evaluate the validity of the concept. The developed concept represents an innovation in oil pumps design, being unique in its mechanical system for variable displacement delivery.