Hierarchical Reinforcement Learning in Behavior and the Brain

Dissertation presented to obtain the Ph.D degree in Biology, Neuroscience

Computational modeling of prefrontal cortex circuits

Dissertation presented to obtain the Ph.D degree in Biology

Can genetic algorithms explain experimental anomalies? An application to common property resources

It is common to find in experimental data persistent oscillations in the aggregate outcomes and high levels of heterogeneity in individual behavior. Furthermore, it is not unusual to find significant deviations from aggregate Nash equilibrium predictions. In this paper, we employ an evolutionary model with boundedly rational agents to explain these findings. We use data from common property resource experiments (Casari and Plott, 2003). Instead of positing individual-specific utility functions, we model decision makers as selfish and identical. Agent interaction is simulated using an individual learning genetic algorithm, where agents have constraints in their working memory, a limited ability to maximize, and experiment with new strategies. We show that the model replicates most of the patterns that can be found in common property resource experiments.

Investigació integrada amb EPs i RMf de la interacció entre la memòria de treball i la distracció

Estudi elaborat a partir d’una estada a la University of Wales, Bangor, Regne Unit entre setembre i desembre del 2006. Els sons distractors augmenten el temps de reacció i el nombre de respostes incorrectes en una tasca de classificació visual, demostrant que hi ha distracció conductual durant la realització de la tasca visual. L’enregistrament concomitant de potencials evocats durant la distracció mostra un patró neuroelèctric característic, el potencial de distracció, que es caracteritza per una ona trifàsica. Darrerament, s’ha demostrat que factors “des de dalt” associats al muntatge experimental tindrien una gran influència en els efectes que els estímuls distractors tindrien en la tasca. Estudis recents mostrarien que aquesta resposta d’atenció exògena es pot modular per la càrrega en memòria de treball, reduint-ne la distracció amb la càrrega, fet que contradiu altres dades que mostraven l’efecte oposat. L’objectiu d’aquest estudi ha estat investigar en quines condicions la càrrega en memòria de treball pot exercir un efecte modulador en les respostes conductuals i cerebrals als sons novedosos distractors, i establir la dinàmica espacio-temporal d’aquesta modulació.

Representation of motor habit in a sequence of repetitive reach and grasp movements performed by macaque monkeys: evidence for a contribution of the dorsolateral prefrontal cortex.

In the context of an autologous cell transplantation study, a unilateral biopsy of cortical tissue was surgically performed from the right dorsolateral prefrontal cortex (dlPFC) in two intact adult macaque monkeys (dlPFC lesioned group), together with the implantation of a chronic chamber providing access to the left motor cortex. Three other monkeys were subjected to the same chronic chamber implantation, but without dlPFC biopsy (control group). All monkeys were initially trained to perform sequential manual dexterity tasks, requiring precision grip. The motor performance and the prehension's sequence (temporal order to grasp pellets from different spatial locations) were analysed for each hand. Following the surgery, transient and moderate deficits of manual dexterity per se occurred in both groups, indicating that they were not due to the dlPFC lesion (most likely related to the recording chamber implantation and/or general anaesthesia/medication). In contrast, changes of motor habit were observed for the sequential order of grasping in the two monkeys with dlPFC lesion only. The changes were more prominent in the monkey subjected to the largest lesion, supporting the notion of a specific effect of the dlPFC lesion on the motor habit of the monkeys. These observations are reminiscent of previous studies using conditional tasks with delay that have proposed a specialization of the dlPFC for visuo-spatial working memory, except that this is in a different context of "free-will", non-conditional manual dexterity task, without a component of working memory.

Programme de remédiation cognitive pour patients présentant une schizophrénie ou un trouble associé (Recos) : résultats préliminaires [Cognitive remediation program for individuals living with schizophrenia (Recos): preliminary results].

BACKGROUND: Nowadays, cognitive remediation is widely accepted as an effective treatment for patients with schizophrenia. In French-speaking countries, techniques used in cognitive remediation for patients with schizophrenia have been applied from those used for patients with cerebral injury. As cognitive impairment is a core feature of schizophrenia, the Département de psychiatrie du CHUV in Lausanne (DP-CHUV) intended to develop a cognitive remediation program for patients with a schizophrenia spectrum disease (Recos-Vianin, 2007). Numerous studies show that the specific cognitive deficits greatly differ from one patient to another. Consequently, Recos aims at providing individualized cognitive remediation therapy. In this feasibility trial, we measured the benefits of this individualized therapy for patients with schizophrenia. Before treatment, the patients were evaluated with a large battery of cognitive tests in order to determine which of the five specific training modules - Verbal memory, visuospatial memory and attention, working memory, selective attention, reasoning - could provide the best benefit depending on their deficit. OBJECTIVES: The study was designed to evaluate the benefits of the Recos program by comparing cognitive functioning before and after treatment. METHOD: Twenty-eight patients with schizophrenia spectrum disorders (schizophrenia [n=18], schizoaffective disorder [n=5], schizotypal disorder [n=4], schizophreniform disorder [n=1], DSM-IV-TR) participated in between one and three of the cognitive modules. The choice of the training module was based on the results of the cognitive tests obtained during the first evaluation. The patients participated in 20 training sessions per module (one session per week). At the end of the training period, the cognitive functioning of each patient was reevaluated by using the same neuropsychological battery. RESULTS: The results showed a greater improvement in the cognitive functions, which were specifically trained, compared to the cognitive functions, which were not trained. However, an improvement was also observed in both types of cognitive functions, suggesting an indirect cognitive gain. CONCLUSION: In our view, the great heterogeneity of the observed cognitive deficits in schizophrenia necessitates a detailed neuropsychological investigation as well as an individualized cognitive remediation therapy. These preliminary results need to be confirmed with a more extended sample of patients.

Visual binding abilities in the initial and advanced stages of schizophrenia

OBJECTIVE: The study tests the hypothesis that intramodal visual binding is disturbed in schizophrenia and should be detectable in all illness stages as a stable trait marker. METHOD: Three groups of patients (rehospitalized chronic schizophrenic, first admitted schizophrenic and schizotypal patients believed to be suffering from a pre-schizophrenic prodrome) and a group of normal control subjects were tested on three tasks targeting visual 'binding' abilities (Muller-Lyer's illusion and two figure detection tasks) in addition to control parameters such as reaction time, visual selective attention, Raven's test and two conventional cortical tasks of spatial working memory (SWM) and a global local test. RESULTS: Chronic patients had a decreased performance on the binding tests. Unexpectedly, the prodromal group exhibited an enhanced Gestalt extraction on these tests compared both to schizophrenic patients and to healthy subjects. Furthermore, chronic schizophrenia was associated with a poor performance on cortical tests of SWM, global local and on Raven. This association appears to be mediated by or linked to the chronicity of the illness. CONCLUSION: The study confirms a variety of neurocognitive deficits in schizophrenia which, however, in this sample seem to be linked to chronicity of illness. However, certain aspects of visual processing concerned with Gestalt extraction deserve attention as potential vulnerability- or prodrome- indicators. The initial hypothesis of the study is rejected.

Análisis y sintonización de aplicaciones paralelas/distribuidas de bioinformática: caso de estudio mpiBLAST

En termes de temps d'execució i ús de dades, les aplicacions paral·leles/distribuïdes poden tenir execucions variables, fins i tot quan s'empra el mateix conjunt de dades d'entrada. Existeixen certs aspectes de rendiment relacionats amb l'entorn que poden afectar dinàmicament el comportament de l'aplicació, tals com: la capacitat de la memòria, latència de la xarxa, el nombre de nodes, l'heterogeneïtat dels nodes, entre d'altres. És important considerar que l'aplicació pot executar-se en diferents configuracions de maquinari i el desenvolupador d'aplicacions no port garantir que els ajustaments de rendiment per a un sistema en particular continuïn essent vàlids per a d'altres configuracions. L'anàlisi dinàmica de les aplicacions ha demostrat ser el millor enfocament per a l'anàlisi del rendiment per dues raons principals. En primer lloc, ofereix una solució molt còmoda des del punt de vista dels desenvolupadors mentre que aquests dissenyen i evaluen les seves aplicacions paral·leles. En segon lloc, perquè s'adapta millor a l'aplicació durant l'execució. Aquest enfocament no requereix la intervenció de desenvolupadors o fins i tot l'accés al codi font de l'aplicació. S'analitza l'aplicació en temps real d'execució i es considra i analitza la recerca dels possibles colls d'ampolla i optimitzacions. Per a optimitzar l'execució de l'aplicació bioinformàtica mpiBLAST, vam analitzar el seu comportament per a identificar els paràmetres que intervenen en el rendiment d'ella, com ara: l'ús de la memòria, l'ús de la xarxa, patrons d'E/S, el sistema de fitxers emprat, l'arquitectura del processador, la grandària de la base de dades biològica, la grandària de la seqüència de consulta, la distribució de les seqüències dintre d'elles, el nombre de fragments de la base de dades i/o la granularitat dels treballs assignats a cada procés. El nostre objectiu és determinar quins d'aquests paràmetres tenen major impacte en el rendiment de les aplicacions i com ajustar-los dinàmicament per a millorar el rendiment de l'aplicació. Analitzant el rendiment de l'aplicació mpiBLAST hem trobat un conjunt de dades que identifiquen cert nivell de serial·lització dintre l'execució. Reconeixent l'impacte de la caracterització de les seqüències dintre de les diferents bases de dades i una relació entre la capacitat dels workers i la granularitat de la càrrega de treball actual, aquestes podrien ser sintonitzades dinàmicament. Altres millores també inclouen optimitzacions relacionades amb el sistema de fitxers paral·lel i la possibilitat d'execució en múltiples multinucli. La grandària de gra de treball està influenciat per factors com el tipus de base de dades, la grandària de la base de dades, i la relació entre grandària de la càrrega de treball i la capacitat dels treballadors.

Clinicopathologic correlates in the oldest-old: Commentary on "No disease in the brain of a 115-year-old woman".

den Dunnen et al. [den Dunnen, W.F.A., Brouwer, W.H., Bijlard, E., Kamphuis, J., van Linschoten, K., Eggens-Meijer, E., Holstege, G., 2008. No disease in the brain of a 115-year-old woman. Neurobiol. Aging] had the opportunity to follow up the cognitive functioning of one of the world's oldest woman during the last 3 years of her life. They performed two neuropsychological evaluations at age 112 and 115 that revealed a striking preservation of immediate recall abilities and orientation. In contrast, working memory, retrieval from semantic memory and mental arithmetic performances declined after age 112. Overall, only a one-point decrease of MMSE score occurred (from 27 to 26) reflecting the remarkable preservation of cognitive abilities. The neuropathological assessment showed few neurofibrillary tangles (NFT) in the hippocampal formation compatible with Braak staging II, absence of amyloid deposits and other types of neurodegenerative lesions as well as preservation of neuron numbers in locus coeruleus. This finding was related to a striking paucity of Alzheimer disease (AD)-related lesions in the hippocampal formation. The present report parallels the early descriptions of rare "supernormal" centenarians supporting the dissociation between brain aging and AD processes. In conjunction with recent stereological analyses in cases aged from 90 to 102 years, it also points to the marked resistance of the hippocampal formation to the degenerative process in this age group and possible dissociation between the occurrence of slight cognitive deficits and development of AD-related pathologic changes in neocortical areas. This work is discussed in the context of current efforts to identify the biological and genetic parameters of human longevity.

The contribution of aging to the understanding of the dimensionality of executive functions.

It has been reported in the literature that executive functions may be fractioned into updating, shifting, and inhibition. The present study aimed to explore whether these executive sub-components can be identified in a more age-heterogeneous sample and see if they are prone to an age-related decline. We tested the performances of 81 individuals aged from 18 to 88 years old in each executive sub-component, working memory, fluid intelligence and processing speed. Correlation analysis revealed only a slight positive relationship between the two updating measures. A linear decrement with age was observed only for two complex executive tests. Tasks indexing working memory, processing speed and fluid intelligence showed a stronger linear decline with age than executive tasks. In conclusion, our results did not replicate the executive structure known from the literature, and revealed that decrement in executive function is not an unavoidable concomitant of aging but rather concerns specific executive tasks.

Poor reward sensitivity and apathy after stroke: implication of basal ganglia.

OBJECTIVE: To examine the relationship between reward sensitivity and self-reported apathy in stroke patients and to investigate the neuroanatomical correlates of both reward sensitivity and apathy. METHODS: In this prospective study, 55 chronic stroke patients were administered a questionnaire to assess apathy and a laboratory task to examine reward sensitivity by measuring motivationally driven behavior ("reinforcement-related speeding"). Fifteen participants without brain damage served as controls for the laboratory task. Negative mood, working memory, and global cognitive functioning were also measured to determine whether reward insensitivity and apathy were secondary to cognitive impairments or negative mood. Voxel-based lesion-symptom mapping was used to explore the neuroanatomical substrates of reward sensitivity and apathy. RESULTS: Participants showed reinforcement-related speeding in the highly reinforced condition of the laboratory task. However, this effect was significant for the controls only. For patients, poorer reward sensitivity was associated with greater self-reported apathy (p < 0.05) beyond negative mood and after lesion size was controlled for. Neither apathy nor reward sensitivity was related to working memory or global cognitive functioning. Voxel-based lesion-symptom mapping showed that damage to the ventral putamen and globus pallidus, dorsal thalamus, and left insula and prefrontal cortex was associated with poorer reward sensitivity. The putamen and thalamus were also involved in self-reported apathy. CONCLUSIONS: Poor reward sensitivity in stroke patients with damage to the ventral basal ganglia, dorsal thalamus, insula, or prefrontal cortex constitutes a core feature of apathy. These results provide valuable insight into the neural mechanisms and brain substrate underlying apathy.

Hyperfrontality and hypoconnectivity during refreshing in schizophrenia.

Anomalous activations of the prefrontal cortex (PFC) and posterior cerebral areas have been reported in previous studies of working memory in schizophrenia. Several interpretations have been reported: e.g., neural inefficiency, the use of different strategies and differences in the functional organization of the cerebral cortex. To better understand these abnormal activations, we investigated the cerebral bases of a working memory component process, namely refreshing (i.e., thinking briefly of a just-activated representation). Fifteen patients with schizophrenia and 15 control subjects participated in this functional magnetic resonance imaging (fMRI) study. Participants were told that whenever they saw a word on the screen, they had to read it silently to themselves (read and repeat conditions), and when they saw a dot, they had to think of the just-previous word (refresh condition). The refresh condition (in comparison with the read condition) was associated with significantly increased activation in the left inferior frontal gyrus and significantly decreased connectivity within the prefrontal cortex and between the prefrontal and parietal cortices in patients with schizophrenia in comparison with control subjects. These results suggest that prefrontal dysfunctions in schizophrenia might be related to a defective ability to initiate (rather than to execute) specific cognitive processes.

Neuroanatomical and neuropsychological features of euthymic patients with bipolar disorder.

OBJECTIVE: Previous studies reported that the severity of cognitive deficits in euthymic patients with bipolar disorder (BD) increases with the duration of illness and postulated that progressive neuronal loss or shrinkage and white matter changes may be at the origin of this phenomenon. To explore this issue, the authors performed a case-control study including detailed neuropsychological and magnetic resonance imaging analyses in 17 euthymic elderly patients with BD and 17 healthy individuals. METHODS: Neuropsychological evaluation concerned working memory, episodic memory, processing speed, and executive functions. Volumetric estimates of the amygdala, hippocampus, entorhinal cortex, and anterior cingulate cortex were obtained using both voxel-based and region of interest morphometric methods. Periventricular and deep white matter were assessed semiquantitatively. Differences in cognitive performances and structural data between BD and comparison groups were analyzed using paired t-test or analysis of variance. Wilcoxon test was used in the absence of normal distribution. RESULTS: Compared with healthy individuals, patients with BD obtained significantly lower performances in processing speed, working memory, and episodic memory but not in executive functions. Morphometric analyses did not show significant volumetric or white matter differences between the two groups. CONCLUSIONS: Our results revealed impairment in verbal memory, working memory, and processing speed in euthymic older adults with BD. These cognitive deficits are comparable both in terms of affected functions and size effects to those previously reported in younger cohorts with BD. Both this observation and the absence of structural brain abnormalities in our cohort do not support a progressively evolving neurotoxic effect in BD.

Structural Validity of the Wechsler Intelligence Scale for Children (WISC-IV) in a French-speaking Swiss sample

The Wechsler Intelligence Scale for Children-fourth edition (i.e. WISC-IV) recognizes a four-factor scoring structure in addition to the Full Scale IQ (FSIQ) score: Verbal Comprehension (VCI), Perceptual Reasoning (PRI), Working Memory (WMI), and Processing Speed (PSI) indices. However, several authors suggested that models based on the Cattell-Horn-Carroll (CHC) theory with 5 or 6 factors provided a better fit to the data than does the current four-factor solution. By comparing the current four-factor structure to CHC-based models, this research aimed to investigate the factorial structure and the constructs underlying the WISC-IV subtest scores with French-speaking Swiss children (N = 249). To deal with this goal, confirmatory factor analyses (CFAs) were conducted. Results showed that a CHC-based model with five factors better fitted the French-Swiss data than did the current WISC-IV scoring structure. All together, these results support the hypothesis of the appropriateness of the CHC model with French-speaking children.

Environnement et capacités cognitives chez le rat : compétition entre les facteurs qui influencent les stratégies spatiales

RESUME Ce travail se propose de discuter des résultats comportementaux observés chez des rats obtenus dans trois paradigmes expérimentaux différents : le bassin de Morris (Morris Water Maze, Morris, 1984) ; la table à trous (Homing Board, Schenk, 1989) et le labyrinthe radial (Radial Arm Maze, Olton et Samuelson, 1976). Les deux premières tâches sont spatiales et permettent un apprentissage de place en environnements contrôlés, et la troisième est une tâche comportementale qui différencie deux habiletés particulières, celle d'élimination (basée sur la mémoire de travail) et celle de sélection (basée sur la mémoire de référence). La discussion des résultats porte sur les stratégies de navigation utilisées par les animaux pour résoudre les tâches et plus précisément sur les facteurs qui peuvent influencer le choix de ces stratégies. Le facteur environnemental (environnement contrôlé) et le facteur cognitif (vieillissement) représentent les variables étudiées ici. C'est ainsi que certaines hypothèses communément acceptées ont été malmenées par nos résultats. Or si l'espace est habituellement supposé homogène (toutes les positions spatiales présentent le même degré de difficulté lors d'un apprentissage en champ ouvert), ce travail établit qu'une position associée -sans contiguïté - à l'un des trois indices visuels situés dans la périphérie de l'environnement est plus difficile à apprendre qu'une position située entre deux des trois indices. Deuxièmement, alors qu'il est admis que l'apprentissage d'une place dans un environnement riche requiert le même type d'information. dans la bassin de Morris (tâche nagée) que sur la table à trous (tâche marchée), nous avons montré que la discrimination spatiale en bassin ne peut être assurée par les trois indices visuels périphériques et nécessite la présence d'au moins un élément supplémentaire. Enfin, l'étude du vieillissement a souvent montré que l'âge réduit les capacités cognitives nécessaires à la navigation spatiale, conduisant à un déficit général des performances d'un animal sénescent, alors que dans notre travail, nous avons trouvé les animaux âgés plus performants et plus efficaces que les adultes dans une tâche particulière de collecte de nourriture. Ces expériences s'inscrivent dans une étude générale qui met à l'épreuve le modèle théorique proposé pax Schenk et Jacobs (2003), selon lequel l'encodage de la carte cognitive (Tolman, 1948 ; O'Keefe et Nadel, 1978) se ferait dans l'hippocampe par l'activité de deux modules complémentaires :d'une part le CA3 - Gyrus Denté pour le traitement d'une trame spatiale basée sur des éléments directionnels et Jou distribués en gradient (bearing map) et d'autre part le CAl - Subiculum pour le traitement des représentations locales basées sur les positions relatives des éléments fixes de l'environnement (sketch map). SUMMARY This work proposes to talk about behavioural results observed in three different experimental paradigms with rats: the Morris Water Maze (Morris, 1984); the Homing Board (Schenk, 1989) and the Radial Arm Maze (Olton and Samuelson, 1976). The two first tasks are spatial ones and allow place learning in controlled environments. The third one is a behavioural task which contrasts two particular skills, the elimination (based on working memory) and the selection one (based on reference memory). The topic of the discussion will be the navigation strategies used by animals to solve the different tasks, and more precisely the factors which can bias this strategies' choice. The environmental (controlled) and the cognitive (aging) factors are the variables studied here. Thus, some hypotheses usually accepted were manhandled by our results. Indeed, if space is habitually homogenously considered (all spatial positions present the same degree of difficulty in an open field learning), this work establishes that an associated position -without being adjacent - to one of the three visual cues localised in the environmental periphery is more difficult to learn than a configurationnel position (situated between two of the three cues). Secondly, if it is received that place learning in a rich environment requires the same information in the Morris water maze (swimming task) that on the Homing board (walking task), we showed that spatial discrimination in the water maze can't be provided by the three peripheral cue cards and needs the presence of a supplementary cue. At last, aging studies often showed that oldness decreases cognitive skills in spatial navigation, leading to a general deficit in performances. But, in our work, we found that senescent rats were more efficient than adult ones in a special food collecting task. These experiments come within the scope of a general study which tests the theoretical model proposed by Jacobs and Schenk (2003), according to which the cognitive map's encoding (Tolman, 1948, O'Keefe and Nadel, 1978) should take place in the hippocampus by two complementary modules, first the DG-CA3 should encode directional and/or gradients references (the bearing map), and secondly the Subiculum-CAl should process locale elements (the sketch map).