Adaptive and Maladaptive Implications of Reinforcement Learning Processes: Fronto-Striatal Loops and Behavioural Correlates

That humans and animals learn from interaction with the environment is a foundational idea underlying nearly all theories of learning and intelligence. Learning that certain outcomes are associated with specific actions or stimuli (both internal and external), is at the very core of the capacity to adapt behaviour to environmental changes. In the present work, appetitive and aversive reinforcement learning paradigms have been used to investigate the fronto-striatal loops and behavioural correlates of adaptive and maladaptive reinforcement learning processes, aiming to a deeper understanding of how cortical and subcortical substrates interacts between them and with other brain systems to support learning. By combining a large variety of neuroscientific approaches, including behavioral and psychophysiological methods, EEG and neuroimaging techniques, these studies aim at clarifying and advancing the knowledge of the neural bases and computational mechanisms of reinforcement learning, both in normal and neurologically impaired population.

Continual learning for computer vision applications

One of the most visionary goals of Artificial Intelligence is to create a system able to mimic and eventually surpass the intelligence observed in biological systems including, ambitiously, the one observed in humans. The main distinctive strength of humans is their ability to build a deep understanding of the world by learning continuously and drawing from their experiences. This ability, which is found in various degrees in all intelligent biological beings, allows them to adapt and properly react to changes by incrementally expanding and refining their knowledge. Arguably, achieving this ability is one of the main goals of Artificial Intelligence and a cornerstone towards the creation of intelligent artificial agents. Modern Deep Learning approaches allowed researchers and industries to achieve great advancements towards the resolution of many long-standing problems in areas like Computer Vision and Natural Language Processing. However, while this current age of renewed interest in AI allowed for the creation of extremely useful applications, a concerningly limited effort is being directed towards the design of systems able to learn continuously. The biggest problem that hinders an AI system from learning incrementally is the catastrophic forgetting phenomenon. This phenomenon, which was discovered in the 90s, naturally occurs in Deep Learning architectures where classic learning paradigms are applied when learning incrementally from a stream of experiences. This dissertation revolves around the Continual Learning field, a sub-field of Machine Learning research that has recently made a comeback following the renewed interest in Deep Learning approaches. This work will focus on a comprehensive view of continual learning by considering algorithmic, benchmarking, and applicative aspects of this field. This dissertation will also touch on community aspects such as the design and creation of research tools aimed at supporting Continual Learning research, and the theoretical and practical aspects concerning public competitions in this field.

L'influenza del sonno sull'andamento temporale della consolidazione delle abilità procedurali

Background: It is well known, since the pioneristic observation by Jenkins and Dallenbach (Am J Psychol 1924;35:605-12), that a period of sleep provides a specific advantage for the consolidation of newly acquired informations. Recent research about the possible enhancing effect of sleep on memory consolidation has focused on procedural memory (part of non-declarative memory system, according to Squire’s taxonomy), as it appears the memory sub-system for which the available data are more consistent. The acquisition of a procedural skill follows a typical time course, consisting in a substantial practice-dependent learning followed by a slow, off-line improvement. Sleep seems to play a critical role in promoting the process of slow learning, by consolidating memory traces and making them more stable and resistant to interferences. If sleep is critical for the consolidation of a procedural skill, then an alteration of the organization of sleep should result in a less effective consolidation, and therefore in a reduced memory performance. Such alteration can be experimentally induced, as in a deprivation protocol, or it can be naturally observed in some sleep disorders as, for example, in narcolepsy. In this research, a group of narcoleptic patients, and a group of matched healthy controls, were tested in two different procedural abilities, in order to better define the size and time course of sleep contribution to memory consolidation. Experimental Procedure: A Texture Discrimination Task (Karni & Sagi, Nature 1993;365:250-2) and a Finger Tapping Task (Walker et al., Neuron 2002;35:205-11) were administered to two indipendent samples of drug-naive patients with first-diagnosed narcolepsy with cataplexy (International Classification of Sleep Disorder 2nd ed., 2005), and two samples of matched healthy controls. In the Texture Discrimination task, subjects (n=22) had to learn to recognize a complex visual array on the screen of a personal computer, while in the Finger Tapping task (n=14) they had to press a numeric sequence on a standard keyboard, as quickly and accurately as possible. Three subsequent experimental sessions were scheduled for each partecipant, namely a training session, a first retrieval session the next day, and a second retrieval session one week later. To test for possible circadian effects on learning, half of the subjects performed the training session at 11 a.m. and half at 17 p.m. Performance at training session was taken as a measure of the practice-dependent learning, while performance of subsequent sessions were taken as a measure of the consolidation level achieved respectively after one and seven nights of sleep. Between training and first retrieval session, all participants spent a night in a sleep laboratory and underwent a polygraphic recording. Results and Discussion: In both experimental tasks, while healthy controls improved their performance after one night of undisturbed sleep, narcoleptic patients showed a non statistically significant learning. Despite this, at the second retrieval session either healthy controls and narcoleptics improved their skills. Narcoleptics improved relatively more than controls between first and second retrieval session in the texture discrimination ability, while their performance remained largely lower in the motor (FTT) ability. Sleep parameters showed a grater fragmentation in the sleep of the pathological group, and a different distribution of Stage 1 and 2 NREM sleep in the two groups, being thus consistent with the hypothesis of a lower consolidation power of sleep in narcoleptic patients. Moreover, REM density of the first part of the night of healthy subjects showed a significant correlation with the amount of improvement achieved at the first retrieval session in TDT task, supporting the hypothesis that REM sleep plays an important role in the consolidation of visuo-perceptual skills. Taken together, these results speak in favor of a slower, rather than lower consolidation of procedural skills in narcoleptic patients. Finally, an explanation of the results, based on the possible role of sleep in contrasting the interference provided by task repetition is proposed.

Studio longitudinale sulle capacità cognitive del cane: discriminazione di quantità ed apprendimento per imitazione in relazione all'attaccamento

Il presente studio ha indagato e valutato alcune abilità cognitive del cane: la capacità di discriminare quantità e le capacità di apprendimento mediante imitazione; quest’ultima è poi stata messa in relazione con l’attaccamento nei confronti del proprietario. Per l’esecuzione della prima indagine sono stati messi appunto due test: il primo si è basato esclusivamente sulla presentazione di uno stimolo visivo: diversi quantitativi di cibo, differenti tra loro del 50%, sono stati presentati al cane; la scelta effettuata dai soggetti testati è stata premiata con differenti tipi di rinforzo differenziale o non differenziale. Il secondo test è stato diviso in due parti: sono stati presentati al cane diversi quantitativi di cibo sempre differenti tra loro del 50% ma nella prima parte del test l’input sensoriale per il cane è stato esclusivamente uditivo mentre nella seconda parte è stato sia uditivo che visivo. Ove è stato possibile è stato applicato ai cani un cardiofrequenzimetro al fine di eseguire una valutazione delle variazioni della frequenza cardiaca nel corso del test. Lo scopo è stato quello di valutare se i soggetti testati erano in grado di discriminare la quantità maggiore. La seconda indagine ha analizzato le capacità di apprendimento di 36 soggetti che sono stati suddivisi in cani da lavoro e pet. I soggetti protagonisti dello studio hanno eseguito il Mirror Test per la valutazione dell’apprendimento per imitazione. I soggetti presi in considerazione, sono stati sottoposti a scansione termografica all’inizio ed al termine del test ed è stata rilevata la loro frequenza respiratoria nella fase iniziale e finale del test. In 11 soggetti che hanno eseguito il precedente test è stato possibile eseguire anche il Strange Situation Test per la valutazione dell’attaccamento al proprietario; i test in questione sono stati videoregistrati ed analizzati per mezzo di un software preposto (OBSERVER XT 10).

Supporting Scientific Research Through Machine and Deep Learning: Fluorescence Microscopy and Operational Intelligence Use Cases

Although the debate of what data science is has a long history and has not reached a complete consensus yet, Data Science can be summarized as the process of learning from data. Guided by the above vision, this thesis presents two independent data science projects developed in the scope of multidisciplinary applied research. The first part analyzes fluorescence microscopy images typically produced in life science experiments, where the objective is to count how many marked neuronal cells are present in each image. Aiming to automate the task for supporting research in the area, we propose a neural network architecture tuned specifically for this use case, cell ResUnet (c-ResUnet), and discuss the impact of alternative training strategies in overcoming particular challenges of our data. The approach provides good results in terms of both detection and counting, showing performance comparable to the interpretation of human operators. As a meaningful addition, we release the pre-trained model and the Fluorescent Neuronal Cells dataset collecting pixel-level annotations of where neuronal cells are located. In this way, we hope to help future research in the area and foster innovative methodologies for tackling similar problems. The second part deals with the problem of distributed data management in the context of LHC experiments, with a focus on supporting ATLAS operations concerning data transfer failures. In particular, we analyze error messages produced by failed transfers and propose a Machine Learning pipeline that leverages the word2vec language model and K-means clustering. This provides groups of similar errors that are presented to human operators as suggestions of potential issues to investigate. The approach is demonstrated on one full day of data, showing promising ability in understanding the message content and providing meaningful groupings, in line with previously reported incidents by human operators.

Methods for integrating machine learning and constrained optimization

In the framework of industrial problems, the application of Constrained Optimization is known to have overall very good modeling capability and performance and stands as one of the most powerful, explored, and exploited tool to address prescriptive tasks. The number of applications is huge, ranging from logistics to transportation, packing, production, telecommunication, scheduling, and much more. The main reason behind this success is to be found in the remarkable effort put in the last decades by the OR community to develop realistic models and devise exact or approximate methods to solve the largest variety of constrained or combinatorial optimization problems, together with the spread of computational power and easily accessible OR software and resources. On the other hand, the technological advancements lead to a data wealth never seen before and increasingly push towards methods able to extract useful knowledge from them; among the data-driven methods, Machine Learning techniques appear to be one of the most promising, thanks to its successes in domains like Image Recognition, Natural Language Processes and playing games, but also the amount of research involved. The purpose of the present research is to study how Machine Learning and Constrained Optimization can be used together to achieve systems able to leverage the strengths of both methods: this would open the way to exploiting decades of research on resolution techniques for COPs and constructing models able to adapt and learn from available data. In the first part of this work, we survey the existing techniques and classify them according to the type, method, or scope of the integration; subsequently, we introduce a novel and general algorithm devised to inject knowledge into learning models through constraints, Moving Target. In the last part of the thesis, two applications stemming from real-world projects and done in collaboration with Optit will be presented.