Stability criteria for unsupervised temporal association networks

A biologically realizable, unsupervised learning rule is described for the online extraction of object features, suitable for solving a range of object recognition tasks. Alterations to the basic learning rule are proposed which allow the rule to better suit the parameters of a given input space. One negative consequence of such modifications is the potential for learning instability. The criteria for such instability are modeled using digital filtering techniques and predicted regions of stability and instability tested. The result is a family of learning rules which can be tailored to the specific environment, improving both convergence times and accuracy over the standard learning rule, while simultaneously insuring learning stability.

Information learning and the stability of fiat money

We analyze the stability of monetary regimes in a decentralized economy where fiat money is endogenously created, information about its value is imperfect, and agents only learn from their personal trading experiences. We show that in poorly informed economies, monetary stability depends heavily on the government's commitment to the long run value of money, whereas in economies where agents gather information more easily, monetary stability can be an endogenous outcome. We generate a dynamics on the acceptability of fiat money that resembles historical accounts of the rise and eventual colIapse of overissued paper money. Moreover, our results provide an explanation of the fact that, despite its obvious advantages, the widespread use of fiat money is a very recent development.

Stochastic stability analysis for the constant-modulus algorithm

We derive an easy-to-compute approximate bound for the range of step-sizes for which the constant-modulus algorithm (CMA) will remain stable if initialized close to a minimum of the CM cost function. Our model highlights the influence, of the signal constellation used in the transmission system: for smaller variation in the modulus of the transmitted symbols, the algorithm will be more robust, and the steady-state misadjustment will be smaller. The theoretical results are validated through several simulations, for long and short filters and channels.

The acquisition of movement skills: Practice enhances the dynamic stability of bimanual coordination

During bimanual movements, two relatively stable inherent patterns of coordination (in-phase and anti-phase) are displayed (e.g., Kelso, Am. J. Physiol. 246 (1984) R1000). Recent research has shown that new patterns of coordination can be learned. For example, following practice a 90 degrees out-of-phase pattern can emerge as an additional, relatively stable, state (e.g., Zanone & Kelso, J. Exp. Psychol.: Human Performance and Perception 18 (1992) 403). On this basis, it has been concluded that practice leads to the evolution and stabilisation of the newly learned pattern and that this process of learning changes the entire attractor layout of the dynamic system. A general feature of such research has been to observe the changes of the targeted pattern's stability characteristics during training at a single movement frequency. The present study was designed to examine how practice affects the maintenance of a coordinated pattern as the movement frequency is scaled. Eleven volunteers were asked to perform a bimanual forearm pronation-supination task. Time to transition onset was used as an index of the subjects' ability to maintain two symmetrically opposite coordinated patterns (target task - 90 degrees out-of-phase - transfer task - 270 degrees out-of-phase). Their ability to maintain the target task and the transfer task were examined again after five practice sessions each consisting of 15 trials of only the 90 degrees out-of-phase pattern. Concurrent performance feedback (a Lissajous figure) was available to the participants during each practice trial. A comparison of the time to transition onset showed that the target task was more stable after practice (p = 0.025). These changes were still observed one week (p = 0.05) and two months (p = 0.075) after the practice period. Changes in the stability of the transfer task were not observed until two months after practice (p = 0.025). Notably, following practice, transitions from the 90 degrees pattern were generally to the anti-phase (180 degrees) pattern, whereas, transitions from the 270 degrees pattern were to the 90 degrees pattern. These results suggest that practice does improve the stability of a 90 degrees pattern, and that such improvements are transferable to the performance of the unpractised 270 degrees pattern. In addition, the anti-phase pattern remained more stable than the practised 90 degrees pattern throughout. (C) 2001 Elsevier Science B.V. All rights reserved.

Dynamic equilibrium through reinforcement learning

Reinforcement Learning is an area of Machine Learning that deals with how an agent should take actions in an environment such as to maximize the notion of accumulated reward. This type of learning is inspired by the way humans learn and has led to the creation of various algorithms for reinforcement learning. These algorithms focus on the way in which an agent’s behaviour can be improved, assuming independence as to their surroundings. The current work studies the application of reinforcement learning methods to solve the inverted pendulum problem. The importance of the variability of the environment (factors that are external to the agent) on the execution of reinforcement learning agents is studied by using a model that seeks to obtain equilibrium (stability) through dynamism – a Cart-Pole system or inverted pendulum. We sought to improve the behaviour of the autonomous agents by changing the information passed to them, while maintaining the agent’s internal parameters constant (learning rate, discount factors, decay rate, etc.), instead of the classical approach of tuning the agent’s internal parameters. The influence of changes on the state set and the action set on an agent’s capability to solve the Cart-pole problem was studied. We have studied typical behaviour of reinforcement learning agents applied to the classic BOXES model and a new form of characterizing the environment was proposed using the notion of convergence towards a reference value. We demonstrate the gain in performance of this new method applied to a Q-Learning agent.

Automatic learning for the classification of chemical reactions and in statistical thermodynamics

This Thesis describes the application of automatic learning methods for a) the classification of organic and metabolic reactions, and b) the mapping of Potential Energy Surfaces(PES). The classification of reactions was approached with two distinct methodologies: a representation of chemical reactions based on NMR data, and a representation of chemical reactions from the reaction equation based on the physico-chemical and topological features of chemical bonds. NMR-based classification of photochemical and enzymatic reactions. Photochemical and metabolic reactions were classified by Kohonen Self-Organizing Maps (Kohonen SOMs) and Random Forests (RFs) taking as input the difference between the 1H NMR spectra of the products and the reactants. The development of such a representation can be applied in automatic analysis of changes in the 1H NMR spectrum of a mixture and their interpretation in terms of the chemical reactions taking place. Examples of possible applications are the monitoring of reaction processes, evaluation of the stability of chemicals, or even the interpretation of metabonomic data. A Kohonen SOM trained with a data set of metabolic reactions catalysed by transferases was able to correctly classify 75% of an independent test set in terms of the EC number subclass. Random Forests improved the correct predictions to 79%. With photochemical reactions classified into 7 groups, an independent test set was classified with 86-93% accuracy. The data set of photochemical reactions was also used to simulate mixtures with two reactions occurring simultaneously. Kohonen SOMs and Feed-Forward Neural Networks (FFNNs) were trained to classify the reactions occurring in a mixture based on the 1H NMR spectra of the products and reactants. Kohonen SOMs allowed the correct assignment of 53-63% of the mixtures (in a test set). Counter-Propagation Neural Networks (CPNNs) gave origin to similar results. The use of supervised learning techniques allowed an improvement in the results. They were improved to 77% of correct assignments when an ensemble of ten FFNNs were used and to 80% when Random Forests were used. This study was performed with NMR data simulated from the molecular structure by the SPINUS program. In the design of one test set, simulated data was combined with experimental data. The results support the proposal of linking databases of chemical reactions to experimental or simulated NMR data for automatic classification of reactions and mixtures of reactions. Genome-scale classification of enzymatic reactions from their reaction equation. The MOLMAP descriptor relies on a Kohonen SOM that defines types of bonds on the basis of their physico-chemical and topological properties. The MOLMAP descriptor of a molecule represents the types of bonds available in that molecule. The MOLMAP descriptor of a reaction is defined as the difference between the MOLMAPs of the products and the reactants, and numerically encodes the pattern of bonds that are broken, changed, and made during a chemical reaction. The automatic perception of chemical similarities between metabolic reactions is required for a variety of applications ranging from the computer validation of classification systems, genome-scale reconstruction (or comparison) of metabolic pathways, to the classification of enzymatic mechanisms. Catalytic functions of proteins are generally described by the EC numbers that are simultaneously employed as identifiers of reactions, enzymes, and enzyme genes, thus linking metabolic and genomic information. Different methods should be available to automatically compare metabolic reactions and for the automatic assignment of EC numbers to reactions still not officially classified. In this study, the genome-scale data set of enzymatic reactions available in the KEGG database was encoded by the MOLMAP descriptors, and was submitted to Kohonen SOMs to compare the resulting map with the official EC number classification, to explore the possibility of predicting EC numbers from the reaction equation, and to assess the internal consistency of the EC classification at the class level. A general agreement with the EC classification was observed, i.e. a relationship between the similarity of MOLMAPs and the similarity of EC numbers. At the same time, MOLMAPs were able to discriminate between EC sub-subclasses. EC numbers could be assigned at the class, subclass, and sub-subclass levels with accuracies up to 92%, 80%, and 70% for independent test sets. The correspondence between chemical similarity of metabolic reactions and their MOLMAP descriptors was applied to the identification of a number of reactions mapped into the same neuron but belonging to different EC classes, which demonstrated the ability of the MOLMAP/SOM approach to verify the internal consistency of classifications in databases of metabolic reactions. RFs were also used to assign the four levels of the EC hierarchy from the reaction equation. EC numbers were correctly assigned in 95%, 90%, 85% and 86% of the cases (for independent test sets) at the class, subclass, sub-subclass and full EC number level,respectively. Experiments for the classification of reactions from the main reactants and products were performed with RFs - EC numbers were assigned at the class, subclass and sub-subclass level with accuracies of 78%, 74% and 63%, respectively. In the course of the experiments with metabolic reactions we suggested that the MOLMAP / SOM concept could be extended to the representation of other levels of metabolic information such as metabolic pathways. Following the MOLMAP idea, the pattern of neurons activated by the reactions of a metabolic pathway is a representation of the reactions involved in that pathway - a descriptor of the metabolic pathway. This reasoning enabled the comparison of different pathways, the automatic classification of pathways, and a classification of organisms based on their biochemical machinery. The three levels of classification (from bonds to metabolic pathways) allowed to map and perceive chemical similarities between metabolic pathways even for pathways of different types of metabolism and pathways that do not share similarities in terms of EC numbers. Mapping of PES by neural networks (NNs). In a first series of experiments, ensembles of Feed-Forward NNs (EnsFFNNs) and Associative Neural Networks (ASNNs) were trained to reproduce PES represented by the Lennard-Jones (LJ) analytical potential function. The accuracy of the method was assessed by comparing the results of molecular dynamics simulations (thermal, structural, and dynamic properties) obtained from the NNs-PES and from the LJ function. The results indicated that for LJ-type potentials, NNs can be trained to generate accurate PES to be used in molecular simulations. EnsFFNNs and ASNNs gave better results than single FFNNs. A remarkable ability of the NNs models to interpolate between distant curves and accurately reproduce potentials to be used in molecular simulations is shown. The purpose of the first study was to systematically analyse the accuracy of different NNs. Our main motivation, however, is reflected in the next study: the mapping of multidimensional PES by NNs to simulate, by Molecular Dynamics or Monte Carlo, the adsorption and self-assembly of solvated organic molecules on noble-metal electrodes. Indeed, for such complex and heterogeneous systems the development of suitable analytical functions that fit quantum mechanical interaction energies is a non-trivial or even impossible task. The data consisted of energy values, from Density Functional Theory (DFT) calculations, at different distances, for several molecular orientations and three electrode adsorption sites. The results indicate that NNs require a data set large enough to cover well the diversity of possible interaction sites, distances, and orientations. NNs trained with such data sets can perform equally well or even better than analytical functions. Therefore, they can be used in molecular simulations, particularly for the ethanol/Au (111) interface which is the case studied in the present Thesis. Once properly trained, the networks are able to produce, as output, any required number of energy points for accurate interpolations.

Expectations, Learning and Monetary Policy: An Overview of Recent Research

Expectations about the future are central for determination of current macroeconomic outcomes and the formulation of monetary policy. Recent literature has explored ways for supplementing the benchmark of rational expectations with explicit models of expectations formation that rely on econometric learning. Some apparently natural policy rules turn out to imply expectational instability of private agents’ learning. We use the standard New Keynesian model to illustrate this problem and survey the key results about interest-rate rules that deliver both uniqueness and stability of equilibrium under econometric learning. We then consider some practical concerns such as measurement errors in private expectations, observability of variables and learning of structural parameters required for policy. We also discuss some recent applications including policy design under perpetual learning, estimated models with learning, recurrent hyperinflations, and macroeconomic policy to combat liquidity traps and deflation.

Accurate performance of a rat model of schizophrenia in the water maze depends on visual cue availability and stability: a distortion in cognitive mapping abilities?

Rats were treated postnatally (PND 5-16) with BSO (l-buthionine-(S,R)-sulfoximine) in an animal model of schizophrenia based on transient glutathione deficit. The BSO treated rats were impaired in patrolling a maze or a homing table when adult, yet demonstrated preserved escape learning, place discrimination and reversal in a water maze task [37]. In the present work, BSO rats' performance in the water maze was assessed in conditions controlling for the available visual cues. First, in a completely curtained environment with two salient controlled cues, BSO rats showed little accuracy compared to control rats. Secondly, pre-trained BSO rats were impaired in reaching the familiar spatial position when curtains partially occluded different portions of the room environment in successive sessions. The apparently preserved place learning in a classical water maze task thus appears to require the stability and the richness of visual landmarks from the surrounding environment. In other words, the accuracy of BSO rats in place and reversal learning is impaired in a minimal cue condition or when the visual panorama changes between trials. However, if the panorama remains rich and stable between trials, BSO rats are equally efficient in reaching a familiar position or in learning a new one. This suggests that the BSO accurate performance in the water maze does not satisfy all the criteria for a cognitive map based navigation on the integration of polymodal cues. It supports the general hypothesis of a binding deficit in BSO rats.

Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed

Adaptive learning in models with lagged variables

In this work I study the stability of the dynamics generated by adaptivelearning processes in intertemporal economies with lagged variables. Iprove that determinacy of the steady state is a necessary condition for the convergence of the learning dynamics and I show that the reciprocal is not true characterizing the economies where convergence holds. In the case of existence of cycles I show that there is not, in general, a relationship between determinacy and convergence of the learning process to the cycle. I also analyze the expectational stability of these equilibria.

Reversal of activity-mediated spine dynamics and learning impairment in a mouse model of Fragile X syndrome.

Fragile X syndrome (FXS) is characterized by intellectual disability and autistic traits, and results from the silencing of the FMR1 gene coding for a protein implicated in the regulation of protein synthesis at synapses. The lack of functional Fragile X mental retardation protein has been proposed to result in an excessive signaling of synaptic metabotropic glutamate receptors, leading to alterations of synapse maturation and plasticity. It remains, however, unclear how mechanisms of activity-dependent spine dynamics are affected in Fmr knockout (Fmr1-KO) mice and whether they can be reversed. Here we used a repetitive imaging approach in hippocampal slice cultures to investigate properties of structural plasticity and their modulation by signaling pathways. We found that basal spine turnover was significantly reduced in Fmr1-KO mice, but markedly enhanced by activity. Additionally, activity-mediated spine stabilization was lost in Fmr1-KO mice. Application of the metabotropic glutamate receptor antagonist α-Methyl-4-carboxyphenylglycine (MCPG) enhanced basal turnover, improved spine stability, but failed to reinstate activity-mediated spine stabilization. In contrast, enhancing phosphoinositide-3 kinase (PI3K) signaling, a pathway implicated in various aspects of synaptic plasticity, reversed both basal turnover and activity-mediated spine stabilization. It also restored defective long-term potentiation mechanisms in slices and improved reversal learning in Fmr1-KO mice. These results suggest that modulation of PI3K signaling could contribute to improve the cognitive deficits associated with FXS.

Learning wind fields with multiple kernels

This paper presents multiple kernel learning (MKL) regression as an exploratory spatial data analysis and modelling tool. The MKL approach is introduced as an extension of support vector regression, where MKL uses dedicated kernels to divide a given task into sub-problems and to treat them separately in an effective way. It provides better interpretability to non-linear robust kernel regression at the cost of a more complex numerical optimization. In particular, we investigate the use of MKL as a tool that allows us to avoid using ad-hoc topographic indices as covariables in statistical models in complex terrains. Instead, MKL learns these relationships from the data in a non-parametric fashion. A study on data simulated from real terrain features confirms the ability of MKL to enhance the interpretability of data-driven models and to aid feature selection without degrading predictive performances. Here we examine the stability of the MKL algorithm with respect to the number of training data samples and to the presence of noise. The results of a real case study are also presented, where MKL is able to exploit a large set of terrain features computed at multiple spatial scales, when predicting mean wind speed in an Alpine region.

JNK regulates dendrite and spine architecture in the central nervous system influencing spatial learning and motor tasks

JNK1 is a MAP-kinase that has proven a significant player in the central nervous system. It regulates brain development and the maintenance of dendrites and axons. Several novel phosphorylation targets of JNK1 were identified in a screen performed in the Coffey lab. These proteins were mainly involved in the regulation of neuronal cytoskeleton, influencing the dynamics and stability of microtubules and actin. These structural proteins form the dynamic backbone for the elaborate architecture of the dendritic tree of a neuron. The initiation and branching of the dendrites requires a dynamic interplay between the cytoskeletal building blocks. Both microtubules and actin are decorated by associated proteins which regulate their dynamics. The dendrite-specific, high molecular weight microtubule associated protein 2 (MAP2) is an abundant protein in the brain, the binding of which stabilizes microtubules and influences their bundling. Its expression in non-neuronal cells induces the formation of neurite-like processes from the cell body, and its function is highly regulated by phosphorylation. JNK1 was shown to phosphorylate the proline-rich domain of MAP2 in vivo in a previous study performed in the group. Here we verify three threonine residues (T1619, T1622 and T1625) as JNK1 targets, the phosphorylation of which increases the binding of MAP2 to microtubules. This binding stabilizes the microtubules and increases process formation in non-neuronal cells. Phosphorylation-site mutants were engineered in the lab. The non-phosphorylatable mutant of MAP2 (MAP2- T1619A, T1622A, T1625A) in these residues fails to bind microtubules, while the pseudo-phosphorylated form, MAP2- T1619D, T1622D, Thr1625D, efficiently binds and induces process formation even without the presence of active JNK1. Ectopic expression of the MAP2- T1619D, T1622D, Thr1625D in vivo in mouse brain led to a striking increase in the branching of cortical layer 2/3 (L2/3) pyramidal neurons, compared to MAP2-WT. The dendritic complexity defines the receptive field of a neuron and dictates the output to the postsynaptic cells. Previous studies in the group indicated altered dendrite architecture of the pyramidal neurons in the Jnk1-/- mouse motor cortex. Here, we used Lucifer Yellow loading and Sholl analysis of neurons in order to study the dendritic branching in more detail. We report a striking, opposing effect in the absence of Jnk1 in the cortical layers 2/3 and 5 of the primary motor cortex. The basal dendrites of pyramidal neurons close to the pial surface at L2/3 show a reduced complexity. In contrast, the L5 neurons, which receive massive input from the L2/3 neurons, show greatly increased branching. Another novel substrate identified for JNK1 was MARCKSL1, a protein that regulates actin dynamics. It is highly expressed in neurons, but also in various cancer tissues. Three phosphorylation target residues for JNK1 were identified, and it was demonstrated that their phosphorylation reduces actin turnover and retards migration of these cells. Actin is the main cytoskeletal component in dendritic spines, the site of most excitatory synapses in pyramidal neurons. The density and gross morphology of the Lucifer Yellow filled dendrites were characterized and we show reduced density and altered morphology of spines in the motor cortex and in the hippocampal area CA3. The dynamic dendritic spines are widely considered to function as the cellular correlate during learning. We used a Morris water maze to test spatial memory. Here, the wild-type mice outperformed the knock-out mice during the acquisition phase of the experiment indicating impaired special memory. The L5 pyramidal neurons of the motor cortex project to the spinal cord and regulate the movement of distinct muscle groups. Thus the altered dendrite morphology in the motor cortex was expected to have an effect on the input-output balance in the signaling from the cortex to the lower motor circuits. A battery of behavioral tests were conducted for the wild-type and Jnk1-/- mice, and the knock-outs performed poorly compared to wild-type mice in tests assessing balance and fine motor movements. This study expands our knowledge of JNK1 as an important regulator of the dendritic fields of neurons and their manifestations in behavior.

Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed