Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

On the interaction of attentional focus and gaze: The quiet eye inhibits focus-related performance decrements

To date, despite a large body of evidence in favor of the advantage of an effect-related focus of attention compared with a movement-related focus of attention in motor control and learning, the role of vision in this context remains unclear. Therefore, in a golf-putting study, the relation between attentional focus and gaze behavior (in particular, quiet eye, or QE) was investigated. First, the advantage of an effect-related focus, as well as of a long QE duration, could be replicated. Furthermore, in the online-demanding task of golf putting, high performance was associated with later QE offsets. Most decisively, an interaction between attentional focus and gaze behavior was revealed in such a way that the efficiency of the QE selectively manifested under movement-related focus instructions. As these findings suggest neither additive effects nor a causal chain, an alternative hypothesis is introduced explaining positive QE effects by the inhibition of not-to-be parameterized movement variants.

Behavior competence development through e-learning: experience at the undergraduate level in the context of Aula a Distancia Abierta (ADA) Madrid, Spain

This communication presents the results of an innovative approach for competencedevelopment suggesting a new methodology for the integration of these elements in professional development within the ADA initiative (AulaaDistanciaAbierta, Distance and Open Classroom) of the Community of Madrid. The main objective of this initiative is to promote the use of Information and Communication Technologies (ICTs) for educational activities by creating a new learning environment structured on the premises of commitment to self–learning, individual work, communication and virtual interaction, and self and continuous assessment. Results from this experience showed that conceptualization is a positive contribution to learning, as students added names and characteristics to competences and abilities that were previously unknown or underestimated. Also, the diversity of participants’ disciplines indicated multidimensional interest in this idea and supported the theory that this approach to competencedevelopment could be successful in all knowledge areas.

Contributions to Bayesian network learning with applications to neuroscience

Neuronal morphology is a key feature in the study of brain circuits, as it is highly related to information processing and functional identification. Neuronal morphology affects the process of integration of inputs from other neurons and determines the neurons which receive the output of the neurons. Different parts of the neurons can operate semi-independently according to the spatial location of the synaptic connections. As a result, there is considerable interest in the analysis of the microanatomy of nervous cells since it constitutes an excellent tool for better understanding cortical function. However, the morphologies, molecular features and electrophysiological properties of neuronal cells are extremely variable. Except for some special cases, this variability makes it hard to find a set of features that unambiguously define a neuronal type. In addition, there are distinct types of neurons in particular regions of the brain. This morphological variability makes the analysis and modeling of neuronal morphology a challenge. Uncertainty is a key feature in many complex real-world problems. Probability theory provides a framework for modeling and reasoning with uncertainty. Probabilistic graphical models combine statistical theory and graph theory to provide a tool for managing domains with uncertainty. In particular, we focus on Bayesian networks, the most commonly used probabilistic graphical model. In this dissertation, we design new methods for learning Bayesian networks and apply them to the problem of modeling and analyzing morphological data from neurons. The morphology of a neuron can be quantified using a number of measurements, e.g., the length of the dendrites and the axon, the number of bifurcations, the direction of the dendrites and the axon, etc. These measurements can be modeled as discrete or continuous data. The continuous data can be linear (e.g., the length or the width of a dendrite) or directional (e.g., the direction of the axon). These data may follow complex probability distributions and may not fit any known parametric distribution. Modeling this kind of problems using hybrid Bayesian networks with discrete, linear and directional variables poses a number of challenges regarding learning from data, inference, etc. In this dissertation, we propose a method for modeling and simulating basal dendritic trees from pyramidal neurons using Bayesian networks to capture the interactions between the variables in the problem domain. A complete set of variables is measured from the dendrites, and a learning algorithm is applied to find the structure and estimate the parameters of the probability distributions included in the Bayesian networks. Then, a simulation algorithm is used to build the virtual dendrites by sampling values from the Bayesian networks, and a thorough evaluation is performed to show the model’s ability to generate realistic dendrites. In this first approach, the variables are discretized so that discrete Bayesian networks can be learned and simulated. Then, we address the problem of learning hybrid Bayesian networks with different kinds of variables. Mixtures of polynomials have been proposed as a way of representing probability densities in hybrid Bayesian networks. We present a method for learning mixtures of polynomials approximations of one-dimensional, multidimensional and conditional probability densities from data. The method is based on basis spline interpolation, where a density is approximated as a linear combination of basis splines. The proposed algorithms are evaluated using artificial datasets. We also use the proposed methods as a non-parametric density estimation technique in Bayesian network classifiers. Next, we address the problem of including directional data in Bayesian networks. These data have some special properties that rule out the use of classical statistics. Therefore, different distributions and statistics, such as the univariate von Mises and the multivariate von Mises–Fisher distributions, should be used to deal with this kind of information. In particular, we extend the naive Bayes classifier to the case where the conditional probability distributions of the predictive variables given the class follow either of these distributions. We consider the simple scenario, where only directional predictive variables are used, and the hybrid case, where discrete, Gaussian and directional distributions are mixed. The classifier decision functions and their decision surfaces are studied at length. Artificial examples are used to illustrate the behavior of the classifiers. The proposed classifiers are empirically evaluated over real datasets. We also study the problem of interneuron classification. An extensive group of experts is asked to classify a set of neurons according to their most prominent anatomical features. A web application is developed to retrieve the experts’ classifications. We compute agreement measures to analyze the consensus between the experts when classifying the neurons. Using Bayesian networks and clustering algorithms on the resulting data, we investigate the suitability of the anatomical terms and neuron types commonly used in the literature. Additionally, we apply supervised learning approaches to automatically classify interneurons using the values of their morphological measurements. Then, a methodology for building a model which captures the opinions of all the experts is presented. First, one Bayesian network is learned for each expert, and we propose an algorithm for clustering Bayesian networks corresponding to experts with similar behaviors. Then, a Bayesian network which represents the opinions of each group of experts is induced. Finally, a consensus Bayesian multinet which models the opinions of the whole group of experts is built. A thorough analysis of the consensus model identifies different behaviors between the experts when classifying the interneurons in the experiment. A set of characterizing morphological traits for the neuronal types can be defined by performing inference in the Bayesian multinet. These findings are used to validate the model and to gain some insights into neuron morphology. Finally, we study a classification problem where the true class label of the training instances is not known. Instead, a set of class labels is available for each instance. This is inspired by the neuron classification problem, where a group of experts is asked to individually provide a class label for each instance. We propose a novel approach for learning Bayesian networks using count vectors which represent the number of experts who selected each class label for each instance. These Bayesian networks are evaluated using artificial datasets from supervised learning problems. Resumen La morfología neuronal es una característica clave en el estudio de los circuitos cerebrales, ya que está altamente relacionada con el procesado de información y con los roles funcionales. La morfología neuronal afecta al proceso de integración de las señales de entrada y determina las neuronas que reciben las salidas de otras neuronas. Las diferentes partes de la neurona pueden operar de forma semi-independiente de acuerdo a la localización espacial de las conexiones sinápticas. Por tanto, existe un interés considerable en el análisis de la microanatomía de las células nerviosas, ya que constituye una excelente herramienta para comprender mejor el funcionamiento de la corteza cerebral. Sin embargo, las propiedades morfológicas, moleculares y electrofisiológicas de las células neuronales son extremadamente variables. Excepto en algunos casos especiales, esta variabilidad morfológica dificulta la definición de un conjunto de características que distingan claramente un tipo neuronal. Además, existen diferentes tipos de neuronas en regiones particulares del cerebro. La variabilidad neuronal hace que el análisis y el modelado de la morfología neuronal sean un importante reto científico. La incertidumbre es una propiedad clave en muchos problemas reales. La teoría de la probabilidad proporciona un marco para modelar y razonar bajo incertidumbre. Los modelos gráficos probabilísticos combinan la teoría estadística y la teoría de grafos con el objetivo de proporcionar una herramienta con la que trabajar bajo incertidumbre. En particular, nos centraremos en las redes bayesianas, el modelo más utilizado dentro de los modelos gráficos probabilísticos. En esta tesis hemos diseñado nuevos métodos para aprender redes bayesianas, inspirados por y aplicados al problema del modelado y análisis de datos morfológicos de neuronas. La morfología de una neurona puede ser cuantificada usando una serie de medidas, por ejemplo, la longitud de las dendritas y el axón, el número de bifurcaciones, la dirección de las dendritas y el axón, etc. Estas medidas pueden ser modeladas como datos continuos o discretos. A su vez, los datos continuos pueden ser lineales (por ejemplo, la longitud o la anchura de una dendrita) o direccionales (por ejemplo, la dirección del axón). Estos datos pueden llegar a seguir distribuciones de probabilidad muy complejas y pueden no ajustarse a ninguna distribución paramétrica conocida. El modelado de este tipo de problemas con redes bayesianas híbridas incluyendo variables discretas, lineales y direccionales presenta una serie de retos en relación al aprendizaje a partir de datos, la inferencia, etc. En esta tesis se propone un método para modelar y simular árboles dendríticos basales de neuronas piramidales usando redes bayesianas para capturar las interacciones entre las variables del problema. Para ello, se mide un amplio conjunto de variables de las dendritas y se aplica un algoritmo de aprendizaje con el que se aprende la estructura y se estiman los parámetros de las distribuciones de probabilidad que constituyen las redes bayesianas. Después, se usa un algoritmo de simulación para construir dendritas virtuales mediante el muestreo de valores de las redes bayesianas. Finalmente, se lleva a cabo una profunda evaluaci ón para verificar la capacidad del modelo a la hora de generar dendritas realistas. En esta primera aproximación, las variables fueron discretizadas para poder aprender y muestrear las redes bayesianas. A continuación, se aborda el problema del aprendizaje de redes bayesianas con diferentes tipos de variables. Las mixturas de polinomios constituyen un método para representar densidades de probabilidad en redes bayesianas híbridas. Presentamos un método para aprender aproximaciones de densidades unidimensionales, multidimensionales y condicionales a partir de datos utilizando mixturas de polinomios. El método se basa en interpolación con splines, que aproxima una densidad como una combinación lineal de splines. Los algoritmos propuestos se evalúan utilizando bases de datos artificiales. Además, las mixturas de polinomios son utilizadas como un método no paramétrico de estimación de densidades para clasificadores basados en redes bayesianas. Después, se estudia el problema de incluir información direccional en redes bayesianas. Este tipo de datos presenta una serie de características especiales que impiden el uso de las técnicas estadísticas clásicas. Por ello, para manejar este tipo de información se deben usar estadísticos y distribuciones de probabilidad específicos, como la distribución univariante von Mises y la distribución multivariante von Mises–Fisher. En concreto, en esta tesis extendemos el clasificador naive Bayes al caso en el que las distribuciones de probabilidad condicionada de las variables predictoras dada la clase siguen alguna de estas distribuciones. Se estudia el caso base, en el que sólo se utilizan variables direccionales, y el caso híbrido, en el que variables discretas, lineales y direccionales aparecen mezcladas. También se estudian los clasificadores desde un punto de vista teórico, derivando sus funciones de decisión y las superficies de decisión asociadas. El comportamiento de los clasificadores se ilustra utilizando bases de datos artificiales. Además, los clasificadores son evaluados empíricamente utilizando bases de datos reales. También se estudia el problema de la clasificación de interneuronas. Desarrollamos una aplicación web que permite a un grupo de expertos clasificar un conjunto de neuronas de acuerdo a sus características morfológicas más destacadas. Se utilizan medidas de concordancia para analizar el consenso entre los expertos a la hora de clasificar las neuronas. Se investiga la idoneidad de los términos anatómicos y de los tipos neuronales utilizados frecuentemente en la literatura a través del análisis de redes bayesianas y la aplicación de algoritmos de clustering. Además, se aplican técnicas de aprendizaje supervisado con el objetivo de clasificar de forma automática las interneuronas a partir de sus valores morfológicos. A continuación, se presenta una metodología para construir un modelo que captura las opiniones de todos los expertos. Primero, se genera una red bayesiana para cada experto y se propone un algoritmo para agrupar las redes bayesianas que se corresponden con expertos con comportamientos similares. Después, se induce una red bayesiana que modela la opinión de cada grupo de expertos. Por último, se construye una multired bayesiana que modela las opiniones del conjunto completo de expertos. El análisis del modelo consensuado permite identificar diferentes comportamientos entre los expertos a la hora de clasificar las neuronas. Además, permite extraer un conjunto de características morfológicas relevantes para cada uno de los tipos neuronales mediante inferencia con la multired bayesiana. Estos descubrimientos se utilizan para validar el modelo y constituyen información relevante acerca de la morfología neuronal. Por último, se estudia un problema de clasificación en el que la etiqueta de clase de los datos de entrenamiento es incierta. En cambio, disponemos de un conjunto de etiquetas para cada instancia. Este problema está inspirado en el problema de la clasificación de neuronas, en el que un grupo de expertos proporciona una etiqueta de clase para cada instancia de manera individual. Se propone un método para aprender redes bayesianas utilizando vectores de cuentas, que representan el número de expertos que seleccionan cada etiqueta de clase para cada instancia. Estas redes bayesianas se evalúan utilizando bases de datos artificiales de problemas de aprendizaje supervisado.

PUE attack detection in CWSN using collaboration and learning behavior

Cognitive Wireless Sensor Network (CWSN) is a new paradigm which integrates cognitive features in traditional Wireless Sensor Networks (WSNs) to mitigate important problems such as spectrum occupancy. Security in Cognitive Wireless Sensor Networks is an important problem because these kinds of networks manage critical applications and data. Moreover, the specific constraints of WSN make the problem even more critical. However, effective solutions have not been implemented yet. Among the specific attacks derived from new cognitive features, the one most studied is the Primary User Emulation (PUE) attack. This paper discusses a new approach, based on anomaly behavior detection and collaboration, to detect the PUE attack in CWSN scenarios. A nonparametric CUSUM algorithm, suitable for low resource networks like CWSN, has been used in this work. The algorithm has been tested using a cognitive simulator that brings important results in this area. For example, the result shows that the number of collaborative nodes is the most important parameter in order to improve the PUE attack detection rates. If the 20% of the nodes collaborates, the PUE detection reaches the 98% with less than 1% of false positives.

Low-status monkeys “play dumb” when learning in mixed social groups

Many primates, including humans, live in complex hierarchical societies where social context and status affect daily life. Nevertheless, primate learning studies typically test single animals in limited laboratory settings where the important effects of social interactions and relationships cannot be studied. To investigate the impact of sociality on associative learning, we compared the individual performances of group-tested rhesus monkeys (Macaca mulatta) across various social contexts. We used a traditional discrimination paradigm that measures an animal’s ability to form associations between cues and the obtaining of food in choice situations; but we adapted the task for group testing. After training a 55-member colony to separate on command into two subgroups, composed of either high- or low-status families, we exposed animals to two color discrimination problems, one with all monkeys present (combined condition), the other in their “dominant” and “subordinate” cohorts (split condition). Next, we manipulated learning history by testing animals on the same problems, but with the social contexts reversed. Monkeys from dominant families excelled in all conditions, but subordinates performed well in the split condition only, regardless of learning history. Subordinate animals had learned the associations, but expressed their knowledge only when segregated from higher-ranking animals. Because aggressive behavior was rare, performance deficits probably reflected voluntary inhibition. This experimental evidence of rank-related, social modulation of performance calls for greater consideration of social factors when assessing learning and may also have relevance for the evaluation of human scholastic achievement.

Prenatal stress produces learning deficits associated with an inhibition of neurogenesis in the hippocampus

Early experiences such as prenatal stress significantly influence the development of the brain and the organization of behavior. In particular, prenatal stress impairs memory processes but the mechanism for this effect is not known. Hippocampal granule neurons are generated throughout life and are involved in hippocampal-dependent learning. Here, we report that prenatal stress in rats induced lifespan reduction of neurogenesis in the dentate gyrus and produced impairment in hippocampal-related spatial tasks. Prenatal stress blocked the increase of learning-induced neurogenesis. These data strengthen pathophysiological hypotheses that propose an early neurodevelopmental origin for psychopathological vulnerabilities in aging.

Spatial memory is related to hippocampal subcellular concentrations of calcium-dependent protein kinase C isoforms in young and aged rats

Relationships were examined between spatial learning and hippocampal concentrations of the α, β2, and γ isoforms of protein kinase C (PKC), an enzyme implicated in neuronal plasticity and memory formation. Concentrations of PKC were determined for individual 6-month-old (n = 13) and 24-month-old (n = 27) male Long–Evans rats trained in the water maze on a standard place-learning task and a transfer task designed for rapid acquisition. The results showed significant relationships between spatial learning and the amount of PKC among individual subjects, and those relationships differed according to age, isoform, and subcellular fraction. Among 6-month-old rats, those with the best spatial memory were those with the highest concentrations of PKCγ in the particulate fraction and of PKCβ2 in the soluble fraction. Aged rats had increased hippocampal PKCγ concentrations in both subcellular fractions in comparison with young rats, and memory impairment was correlated with higher PKCγ concentrations in the soluble fraction. No age difference or correlations with behavior were found for concentrations of PKCγ in a comparison structure, the neostriatum, or for PKCα in the hippocampus. Relationships between spatial learning and hippocampal concentrations of calcium-dependent PKC are isoform-specific. Moreover, age-related spatial memory impairment is associated with altered subcellular concentrations of PKCγ and may be indicative of deficient signal transduction and neuronal plasticity in the hippocampal formation.

Lack of tissue glucocorticoid reactivation in 11β-hydroxysteroid dehydrogenase type 1 knockout mice ameliorates age-related learning impairments

11β-hydroxysteroid dehydrogenase type 1 (11β-HSD-1) intracellularly regenerates active corticosterone from circulating inert 11-dehydrocorticosterone (11-DHC) in specific tissues. The hippocampus is a brain structure particularly vulnerable to glucocorticoid neurotoxicity with aging. In intact hippocampal cells in culture, 11β-HSD-1 acts as a functional 11β-reductase reactivating inert 11-DHC to corticosterone, thereby potentiating kainate neurotoxicity. We examined the functional significance of 11β-HSD-1 in the central nervous system by using knockout mice. Aged wild-type mice developed elevated plasma corticosterone levels that correlated with learning deficits in the watermaze. In contrast, despite elevated plasma corticosterone levels throughout life, this glucocorticoid-associated learning deficit was ameliorated in aged 11β-HSD-1 knockout mice, implicating lower intraneuronal corticosterone levels through lack of 11-DHC reactivation. Indeed, aged knockout mice showed significantly lower hippocampal tissue corticosterone levels than wild-type controls. These findings demonstrate that tissue corticosterone levels do not merely reflect plasma levels and appear to play a more important role in hippocampal functions than circulating blood levels. The data emphasize the crucial importance of local enzymes in determining intracellular glucocorticoid activity. Selective 11β-HSD-1 inhibitors may protect against hippocampal function decline with age.

Song selectivity and sensorimotor signals in vocal learning and production

Bird song, like human speech, is a learned vocal behavior that requires auditory feedback. Both as juveniles, while they learn to sing, and as adults, songbirds use auditory feedback to compare their own vocalizations with an internal model of a target song. Here we describe experiments that explore a role for the songbird anterior forebrain pathway (AFP), a basal ganglia-forebrain circuit, in evaluating song feedback and modifying vocal output. First, neural recordings in anesthetized, juvenile birds show that single AFP neurons are specialized to process the song stimuli that are compared during sensorimotor learning. AFP neurons are tuned to both the bird's own song and the tutor song, even when these stimuli are manipulated to be very different from each other. Second, behavioral experiments in adult birds demonstrate that lesions to the AFP block the deterioration of song that normally follows deafening. This observation suggests that deafening results in an instructive signal, indicating a mismatch between feedback and the internal song model, and that the AFP is involved in generating or transmitting this instructive signal. Finally, neural recordings from behaving birds reveal robust singing-related activity in the AFP. This activity is likely to originate from premotor areas and could be modulated by auditory feedback of the bird's own voice. One possibility is that this activity represents an efference copy, predicting the sensory consequences of motor commands. Overall, these studies illustrate that sensory and motor processes are highly interrelated in this circuit devoted to vocal learning, as is true for brain areas involved in speech.

Age-related learning deficits in transgenic mice expressing the 751-amino acid isoform of human beta-amyloid precursor protein.

The beta-amyloid precursor protein (beta-APP), from which the beta-A4 peptide is derived, is considered to be central to the pathogenesis of Alzheimer disease (AD). Transgenic mice expressing the 751-amino acid isoform of human beta-APP (beta-APP751) have been shown to develop early AD-like histopathology with diffuse deposits of beta-A4 and aberrant tau protein expression in the brain, particularly in the hippocampus, cortex, and amygdala. We now report that beta-APP751 transgenic mice exhibit age-dependent deficits in spatial learning in a water-maze task and in spontaneous alternation in a Y maze. These deficits were mild or absent in 6-month-old transgenic mice but were severe in 12-month-old transgenic mice compared to age-matched wild-type control mice. No other behavioral abnormalities were observed. These mice therefore model the progressive learning and memory impairment that is a cardinal feature of AD. These results provide evidence for a relationship between abnormal expression of beta-APP and cognitive impairments.