Adaptive and Maladaptive Implications of Reinforcement Learning Processes: Fronto-Striatal Loops and Behavioural Correlates

That humans and animals learn from interaction with the environment is a foundational idea underlying nearly all theories of learning and intelligence. Learning that certain outcomes are associated with specific actions or stimuli (both internal and external), is at the very core of the capacity to adapt behaviour to environmental changes. In the present work, appetitive and aversive reinforcement learning paradigms have been used to investigate the fronto-striatal loops and behavioural correlates of adaptive and maladaptive reinforcement learning processes, aiming to a deeper understanding of how cortical and subcortical substrates interacts between them and with other brain systems to support learning. By combining a large variety of neuroscientific approaches, including behavioral and psychophysiological methods, EEG and neuroimaging techniques, these studies aim at clarifying and advancing the knowledge of the neural bases and computational mechanisms of reinforcement learning, both in normal and neurologically impaired population.

Continual learning for computer vision applications

One of the most visionary goals of Artificial Intelligence is to create a system able to mimic and eventually surpass the intelligence observed in biological systems including, ambitiously, the one observed in humans. The main distinctive strength of humans is their ability to build a deep understanding of the world by learning continuously and drawing from their experiences. This ability, which is found in various degrees in all intelligent biological beings, allows them to adapt and properly react to changes by incrementally expanding and refining their knowledge. Arguably, achieving this ability is one of the main goals of Artificial Intelligence and a cornerstone towards the creation of intelligent artificial agents. Modern Deep Learning approaches allowed researchers and industries to achieve great advancements towards the resolution of many long-standing problems in areas like Computer Vision and Natural Language Processing. However, while this current age of renewed interest in AI allowed for the creation of extremely useful applications, a concerningly limited effort is being directed towards the design of systems able to learn continuously. The biggest problem that hinders an AI system from learning incrementally is the catastrophic forgetting phenomenon. This phenomenon, which was discovered in the 90s, naturally occurs in Deep Learning architectures where classic learning paradigms are applied when learning incrementally from a stream of experiences. This dissertation revolves around the Continual Learning field, a sub-field of Machine Learning research that has recently made a comeback following the renewed interest in Deep Learning approaches. This work will focus on a comprehensive view of continual learning by considering algorithmic, benchmarking, and applicative aspects of this field. This dissertation will also touch on community aspects such as the design and creation of research tools aimed at supporting Continual Learning research, and the theoretical and practical aspects concerning public competitions in this field.

L'influenza del sonno sull'andamento temporale della consolidazione delle abilità procedurali

Background: It is well known, since the pioneristic observation by Jenkins and Dallenbach (Am J Psychol 1924;35:605-12), that a period of sleep provides a specific advantage for the consolidation of newly acquired informations. Recent research about the possible enhancing effect of sleep on memory consolidation has focused on procedural memory (part of non-declarative memory system, according to Squire’s taxonomy), as it appears the memory sub-system for which the available data are more consistent. The acquisition of a procedural skill follows a typical time course, consisting in a substantial practice-dependent learning followed by a slow, off-line improvement. Sleep seems to play a critical role in promoting the process of slow learning, by consolidating memory traces and making them more stable and resistant to interferences. If sleep is critical for the consolidation of a procedural skill, then an alteration of the organization of sleep should result in a less effective consolidation, and therefore in a reduced memory performance. Such alteration can be experimentally induced, as in a deprivation protocol, or it can be naturally observed in some sleep disorders as, for example, in narcolepsy. In this research, a group of narcoleptic patients, and a group of matched healthy controls, were tested in two different procedural abilities, in order to better define the size and time course of sleep contribution to memory consolidation. Experimental Procedure: A Texture Discrimination Task (Karni & Sagi, Nature 1993;365:250-2) and a Finger Tapping Task (Walker et al., Neuron 2002;35:205-11) were administered to two indipendent samples of drug-naive patients with first-diagnosed narcolepsy with cataplexy (International Classification of Sleep Disorder 2nd ed., 2005), and two samples of matched healthy controls. In the Texture Discrimination task, subjects (n=22) had to learn to recognize a complex visual array on the screen of a personal computer, while in the Finger Tapping task (n=14) they had to press a numeric sequence on a standard keyboard, as quickly and accurately as possible. Three subsequent experimental sessions were scheduled for each partecipant, namely a training session, a first retrieval session the next day, and a second retrieval session one week later. To test for possible circadian effects on learning, half of the subjects performed the training session at 11 a.m. and half at 17 p.m. Performance at training session was taken as a measure of the practice-dependent learning, while performance of subsequent sessions were taken as a measure of the consolidation level achieved respectively after one and seven nights of sleep. Between training and first retrieval session, all participants spent a night in a sleep laboratory and underwent a polygraphic recording. Results and Discussion: In both experimental tasks, while healthy controls improved their performance after one night of undisturbed sleep, narcoleptic patients showed a non statistically significant learning. Despite this, at the second retrieval session either healthy controls and narcoleptics improved their skills. Narcoleptics improved relatively more than controls between first and second retrieval session in the texture discrimination ability, while their performance remained largely lower in the motor (FTT) ability. Sleep parameters showed a grater fragmentation in the sleep of the pathological group, and a different distribution of Stage 1 and 2 NREM sleep in the two groups, being thus consistent with the hypothesis of a lower consolidation power of sleep in narcoleptic patients. Moreover, REM density of the first part of the night of healthy subjects showed a significant correlation with the amount of improvement achieved at the first retrieval session in TDT task, supporting the hypothesis that REM sleep plays an important role in the consolidation of visuo-perceptual skills. Taken together, these results speak in favor of a slower, rather than lower consolidation of procedural skills in narcoleptic patients. Finally, an explanation of the results, based on the possible role of sleep in contrasting the interference provided by task repetition is proposed.

Studio longitudinale sulle capacità cognitive del cane: discriminazione di quantità ed apprendimento per imitazione in relazione all'attaccamento

Il presente studio ha indagato e valutato alcune abilità cognitive del cane: la capacità di discriminare quantità e le capacità di apprendimento mediante imitazione; quest’ultima è poi stata messa in relazione con l’attaccamento nei confronti del proprietario. Per l’esecuzione della prima indagine sono stati messi appunto due test: il primo si è basato esclusivamente sulla presentazione di uno stimolo visivo: diversi quantitativi di cibo, differenti tra loro del 50%, sono stati presentati al cane; la scelta effettuata dai soggetti testati è stata premiata con differenti tipi di rinforzo differenziale o non differenziale. Il secondo test è stato diviso in due parti: sono stati presentati al cane diversi quantitativi di cibo sempre differenti tra loro del 50% ma nella prima parte del test l’input sensoriale per il cane è stato esclusivamente uditivo mentre nella seconda parte è stato sia uditivo che visivo. Ove è stato possibile è stato applicato ai cani un cardiofrequenzimetro al fine di eseguire una valutazione delle variazioni della frequenza cardiaca nel corso del test. Lo scopo è stato quello di valutare se i soggetti testati erano in grado di discriminare la quantità maggiore. La seconda indagine ha analizzato le capacità di apprendimento di 36 soggetti che sono stati suddivisi in cani da lavoro e pet. I soggetti protagonisti dello studio hanno eseguito il Mirror Test per la valutazione dell’apprendimento per imitazione. I soggetti presi in considerazione, sono stati sottoposti a scansione termografica all’inizio ed al termine del test ed è stata rilevata la loro frequenza respiratoria nella fase iniziale e finale del test. In 11 soggetti che hanno eseguito il precedente test è stato possibile eseguire anche il Strange Situation Test per la valutazione dell’attaccamento al proprietario; i test in questione sono stati videoregistrati ed analizzati per mezzo di un software preposto (OBSERVER XT 10).

Real-time anomaly detection of gamma-ray bursts for the Cherenkov Telescope Array using deep learning

The Cherenkov Telescope Array (CTA) will be the next-generation ground-based observatory to study the universe in the very-high-energy domain. The observatory will rely on a Science Alert Generation (SAG) system to analyze the real-time data from the telescopes and generate science alerts. The SAG system will play a crucial role in the search and follow-up of transients from external alerts, enabling multi-wavelength and multi-messenger collaborations. It will maximize the potential for the detection of the rarest phenomena, such as gamma-ray bursts (GRBs), which are the science case for this study. This study presents an anomaly detection method based on deep learning for detecting gamma-ray burst events in real-time. The performance of the proposed method is evaluated and compared against the Li&Ma standard technique in two use cases of serendipitous discoveries and follow-up observations, using short exposure times. The method shows promising results in detecting GRBs and is flexible enough to allow real-time search for transient events on multiple time scales. The method does not assume background nor source models and doe not require a minimum number of photon counts to perform analysis, making it well-suited for real-time analysis. Future improvements involve further tests, relaxing some of the assumptions made in this study as well as post-trials correction of the detection significance. Moreover, the ability to detect other transient classes in different scenarios must be investigated for completeness. The system can be integrated within the SAG system of CTA and deployed on the onsite computing clusters. This would provide valuable insights into the method's performance in a real-world setting and be another valuable tool for discovering new transient events in real-time. Overall, this study makes a significant contribution to the field of astrophysics by demonstrating the effectiveness of deep learning-based anomaly detection techniques for real-time source detection in gamma-ray astronomy.