Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

Dissecting the role of Profilin-1 in microglial cell function: the impact on ROS production

Microglial cells are the resident immune cells of central nervous system (CNS) and the major players in neuroinflammation. These cells are also responsible for surveilling the neuronal microenvironment, and upon injury to the CNS they change their morphology and molecular profile and become activated. Activated status is associated with microglia proliferation, migration to injury foci, increased phagocytic capacity, production and release of reactive oxygen species (ROS), cytokines (pro- or anti-inflammatory) and reactive nitrogen species. Microglia activation is crucial for tissue repair in the healthy brain. However, their chronic activation or deregulation might contribute for the pathophysiology of neurodegenerative diseases. A better understanding of the mechanisms underlying microglial cell activation is important for defining targets and develop appropriate therapeutic strategies to control the chronic activation of microglia. It has been observed an increase in profilin (Pfn) mRNA in microglial cells in the rat hippocampus after unilateral ablation of its major extrinsic input, the entorhinal cortex. This observation suggested that Pfn might be involved in microglia activation. Pfn1 is an actin binding protein that controls assembly and disassembly of actin filaments and is important for several cellular processes, including, motility, cell proliferation and survival. Here, we studied the role of Pfn1 in microglial cell function. For that, we used primary cortical microglial cell cultures and microglial cell lines in which we knocked down Pfn1 expression and assessed the activation status of microglia, based on classical activation markers, such as: phagocytosis, glutamate release, reactive oxygen species (ROS), pro- and anti-inflammatory cytokines. We demonstrated that Pfn1 (i) is more active in hypoxia-challenged microglia, (ii) modulates microglia pro- and anti-inflammatory signatures and (iii) plays a critical role in ROS generation in microglia. Altogether, we conclude that Pfn1 is a key protein for microglia homeostasis, playing an essential role in their activation, regardless the polarization into a pro or anti-inflammatory signature.