Techniques for batch reinforcement learning in robotics

This thesis addresses the Batch Reinforcement Learning methods in Robotics. This sub-class of Reinforcement Learning has shown promising results and has been the focus of recent research. Three contributions are proposed that aim to extend the state-of-art methods allowing for a faster and more stable learning process, such as required for learning in Robotics. The Q-learning update-rule is widely applied, since it allows to learn without the presence of a model of the environment. However, this update-rule is transition-based and does not take advantage of the underlying episodic structure of collected batch of interactions. The Q-Batch update-rule is proposed in this thesis, to process experiencies along the trajectories collected in the interaction phase. This allows a faster propagation of obtained rewards and penalties, resulting in faster and more robust learning. Non-parametric function approximations are explored, such as Gaussian Processes. This type of approximators allows to encode prior knowledge about the latent function, in the form of kernels, providing a higher level of exibility and accuracy. The application of Gaussian Processes in Batch Reinforcement Learning presented a higher performance in learning tasks than other function approximations used in the literature. Lastly, in order to extract more information from the experiences collected by the agent, model-learning techniques are incorporated to learn the system dynamics. In this way, it is possible to augment the set of collected experiences with experiences generated through planning using the learned models. Experiments were carried out mainly in simulation, with some tests carried out in a physical robotic platform. The obtained results show that the proposed approaches are able to outperform the classical Fitted Q Iteration.

LAP1 interactome and its functional features throughout spermatogenesis

The lamina-associated polypeptide 1 (LAP1) is a type II transmembrane protein of the inner nuclear membrane encoded by the human gene TOR1AIP1. LAP1 is involved in maintaining the nuclear envelope structure and appears be involved in the positioning of lamins and chromatin. In the nuclear envelope, LAP1 is suggested to exist as a complex with A-type and B-type lamins, torsins and emerin. The presence of such complexes suggests that LAP1 may cooperate functionally with these proteins in tissues where they play a critical role. Therefore, the identification of LAP1 binding partners and the signalling pathways where LAP1 participates, is crucial for a better understanding of LAP1 functions. The work described in this thesis addresses novel human LAP1 associated proteins found through bioinformatic tools. Public databases allowed for the discovery of the LAP1 interactome, which was manually curated, identifying several functionally relevant proteins. Subsequently, the integration of multiple bioinformatic tools established novel functions to LAP1 such as DNA damage response and telomere association. In conjunction, bioinformatic results also reinforced the association of LAP1 with mitosis, and the already identified role of LAP1 in nuclear morphology. Interestingly, this association of LAP1 with the regulation of the nuclear envelope structure and mitosis progression, shares functional elements with spermatogenesis. Therefore, this work additionally described the localization of LAP1 and some of its interactors throughout the spermatogenic cycle, in mouse and human testis. The results established that the activity of LAP1 during the mouse spermatogenic cycle is most evident from stage VIII until the end of spermiogenesis, which is characteristic of manchette development. Concomitantly, some LAP1 interactors studied in this work share a similar localization, namely, PP1γ2, Lamin B1 and Lamin A/C. The results obtained from the study of LAP1 throughout different periods of the male reproductive system attributed potential new biological functions to LAP1. Thereby, this work can be the foundation of future studies regarding LAP1 and the regulation of multiple cellular processes and disease conditions.

CD81 interacting proteins

Fertilization is a multistep and complex process culminating in the merge of gamete membranes, cytoplasmic unity and fusion of genome. CD81 is a tetraspanin protein that participates in sperm-oocyte interaction, being present at the oocyte surface. CD81 has also been implicated in other biological processes, however its specific function and molecular mechanisms of action remain to be elucidated. The interaction between CD81 and its binding partner proteins may underlie the CD81 involvement in a variety of cellular processes and modulate CD81/interactors specific functions. Interestingly, in a Yeast two Hybrid system previously performed in our lab, CD81 has emerged as a putative interactor of the Amyloid Precursor Protein (APP). In the work here described, bioinformatics analyses of CD81 interacting proteins were performed and the retrieved information used to construct a protein-protein interaction network, as well as to perform Gene Ontology enrichment analyses. CD81 expression was further evaluated in CHO, GC-1 and SH-SY5Y cell lines, and in human sperm cells. Additionally, its subcellular localization was analyzed in sperm cells and in the neuronal-like SH-SY5Y cell line. Subsequently, coimmunoprecipitation assays were performed in CHO and SH-SY5Y cells to attempt to prove the physical interaction between CD81 and APP. A functional interaction between these two proteins was accessed thought the analyses of the effects of CD81 overexpression on APP levels. A co-localization analysis of CD81 and some interactors proteins retrieved from the bioinformatics analyses, such as APP, AKT1 and cytoskeleton-related proteins, was also performed in sperm cells and in SH-SY5Y cells. The effects of CD81 in cytoskeleton remodeling was evaluated in SH-SY5Y cells through monitoring the effects of CD81 overexpression in actin and tubulin levels, and analyzing the colocalization between overexpressed CD81 and F-actin. Our results showed that CD81 is expressed in all cell lines tested, and also provided the first evidence of the presence of CD81 in human sperm cells. CD81 immunoreactivity was predominantly detected in the sperm head, including the acrosome membrane, and in the midpiece, where it co-localized with APP, as well as in the post-acrosomal region. Furthermore, CD81 co-localizes with APP in the plasma membrane and in cellular projections in SH-SY5Y cells, where CD81 overexpression has an influence on APP levels, also visible in CHO cells. The analysis of CD81 interacting proteins such as AKT1 and cytoskeletonrelated proteins showed that CD81 is involved in a variety of pathways that may underlie cytoskeleton remodeling events, related to processes such as sperm motility, cell migration and neuritogenesis. These results deepen our understanding on the functions of CD81 and some of its interactors in sperm and neuronal cells.