Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed

A behavior-based scheme using reinforcement learning for autonomous underwater vehicles

This paper presents a hybrid behavior-based scheme using reinforcement learning for high-level control of autonomous underwater vehicles (AUVs). Two main features of the presented approach are hybrid behavior coordination and semi on-line neural-Q_learning (SONQL). Hybrid behavior coordination takes advantages of robustness and modularity in the competitive approach as well as efficient trajectories in the cooperative approach. SONQL, a new continuous approach of the Q_learning algorithm with a multilayer neural network is used to learn behavior state/action mapping online. Experimental results show the feasibility of the presented approach for AUVs

Scene classifier for real time road detection

L’objectiu principal del projecte és el de classificar escenes de carretera en funció del contingut de les imatges per així poder fer un desglossament sobre quin tipus de situació tenim en el moment. És important que fixem els paràmetres necessaris en funció de l’escenari en què ens trobem per tal de treure el màxim rendiment possible a cada un dels algoritmes. La seva funcionalitat doncs, ha de ser la d’avís i suport davant els diferents escenaris de conducció. És a dir, el resultat final ha de contenir un algoritme o aplicació capaç de classificar les imatges d’entrada en diferents tipus amb la màxima eficiència espacial i temporal possible. L’algoritme haurà de classificar les imatges en diferents escenaris. Els algoritmes hauran de ser parametritzables i fàcilment manejables per l’usuari. L’eina utilitzada per aconseguir aquests objectius serà el MATLAB amb les toolboxs de visió i xarxes neuronals instal·lades.

Gestió de xarxes Internet basada en SNMP

El treball es divideix en dos grans parts, una primera eminentment teòrica en la que s'estudia SNMP, i una segona íntegrament pràctica en la que s'implementa el protocol. Per a cada part es defineix una metodologia ben diferenciada.

Avenços en l'estudi de la litiasi renal: mètodes de chemical imaging i intel·ligencia artificial (ANN) per la caracterització de càlculs renals

La litiasi urinària és un trastorn que implica la formació de precipitats en qualsevol part del tracte urinari. Aquest és un desordre comú que afecta aproximadament a una desena part de la població de la Unió Europea al llarg de la seva vida. A més, durant els cinc anys posteriors a un episodi litiàsic el percentatge de recurrència dels pacients és del 45 al 75%. Aquest trastorn urinari està influït per una gran quantitat de variables, d’origen fisiològic, psicològic i ambiental. Els episodis litiàsics, es poden solucionar espontàniament, amb l’expulsió del càlcul renal, o bé a través de diverses intervencions mèdiques. Els tractaments mèdics derivats de la litiasi urinària; és a dir, la fragmentació del càlcul, intervencions quirúrgiques i tractaments posteriors generen unes grans despeses als sistemes mèdics. Pels motius exposats, la identificació del desordre que ha originat l’episodi litiàsic és de radical importància, per tal de minimitzar el risc de reincidència. Els mètodes més usuals per determinar les causes que desencadenen la formació del càlcul renal són les anàlisis d’orina i l’estudi del càlcul generat. La correcta descripció de la composició i, especialment, l’estructura del càlcul renal pot aportar informació clau sobre les possibles causes de la seva formació, tant de l’inici de nucleació del càlcul com de les successives etapes de creixement cristal·lí. Tenint en compte aquest darrer aspecte, el present estudi s’ha dirigit a la caracterització de càlculs urinaris mitjançant l’aplicació de metodologies d’imatge química (Hyperspectral Imaging), el que va contribuir a determinar les principals característiques espectrals de cada compost majoritari als càlculs renals. D’altra banda, la utilització de mostres de composició coneguda ha possibilitat la creació d’un model amb Xarxes Neuronals Artificials, que permet la classificació de noves mostres de composició desconeguda, de manera més ràpida que el procediment actual. Aquest treball constitueix una nova contribució a la comprensió de l’estructura de les pedres de ronyó, així com les condicions de la seva formació. Els resultats obtinguts destaquen les possibilitats que presenten les tècniques emprades al camp de la litiasi renal, que permeten complementar els coneixements existents enfocats a millorar la qualitat de vida dels pacients.

The role of fixed delays in neuronal rate models

Fixed delays in neuronal interactions arise through synaptic and dendritic processing. Previous work has shown that such delays, which play an important role in shaping the dynamics of networks of large numbers of spiking neurons with continuous synaptic kinetics, can be taken into account with a rate model through the addition of an explicit, fixed delay. Here we extend this work to account for arbitrary symmetric patterns of synaptic connectivity and generic nonlinear transfer functions. Specifically, we conduct a weakly nonlinear analysis of the dynamical states arising via primary instabilities of the stationary uniform state. In this way we determine analytically how the nature and stability of these states depend on the choice of transfer function and connectivity. While this dependence is, in general, nontrivial, we make use of the smallness of the ratio in the delay in neuronal interactions to the effective time constant of integration to arrive at two general observations of physiological relevance. These are: 1 - fast oscillations are always supercritical for realistic transfer functions. 2 - Traveling waves are preferred over standing waves given plausible patterns of local connectivity.

Analysis of nonlinear noisy integrate & fire neuron models: blow-up and steady states

Nonlinear Noisy Leaky Integrate and Fire (NNLIF) models for neurons networks can be written as Fokker-Planck-Kolmogorov equations on the probability density of neurons, the main parameters in the model being the connectivity of the network and the noise. We analyse several aspects of the NNLIF model: the number of steady states, a priori estimates, blow-up issues and convergence toward equilibrium in the linear case. In particular, for excitatory networks, blow-up always occurs for initial data concentrated close to the firing potential. These results show how critical is the balance between noise and excitatory/inhibitory interactions to the connectivity parameter.

Extensió de les capacitats de monitoratge de JFFNM en entorns Wlan

Aquest TFC pretén recollir un conjunt de paràmetres de radi d'un dispositiu Wireless i poder representar-los en l'aplicació de monitoratge Just For Fun Network Management System (JFFNMS).

Redes neuronales multimodelo aplicadas al control de sistemas

En este proyecto se plantea un nuevo modelo de red neuronal llamado multimodelo. Si en un modelo convencional de red neuronal obtenemos la salida a partir de una serie de parámetros, en un sistema multimodelo recalculamos la salida a partir de los mismos valores de entrada, pero variando los parámetros. Con esto se consigue una mayor muestra de salidas, que permiten decidir cuales son los parámetros más acertados (los que devuelven una salida más cercana a la salida deseada del sistema de control). Para llevar a cabo este proyecto se ha hecho uso de Matlab 7.3 (R2006b).

Impairment of mossy fiber long-term potentiation and associative learning in pituitary adenylate cyclase activating polypeptide type I receptor-deficient Mice

The pituitary adenylate cyclase activating polypeptide (PACAP) type I receptor (PAC1) is a G-protein-coupled receptor binding the strongly conserved neuropeptide PACAP with 1000-fold higher affinity than the related peptide vasoactive intestinal peptide. PAC1-mediated signaling has been implicated in neuronal differentiation and synaptic plasticity. To gain further insight into the biological significance of PAC1-mediated signaling in vivo, we generated two different mutant mouse strains, harboring either a complete or a forebrain-specific inactivation of PAC1. Mutants from both strains show a deficit in contextual fear conditioning, a hippocampus-dependent associative learning paradigm. In sharp contrast, amygdala-dependent cued fear conditioning remains intact. Interestingly, no deficits in other hippocampus-dependent tasks modeling declarative learning such as the Morris water maze or the social transmission of food preference are observed. At the cellular level, the deficit in hippocampus-dependent associative learning is accompanied by an impairment of mossy fiber long-term potentiation (LTP). Because the hippocampal expression of PAC1 is restricted to mossy fiber terminals, we conclude that presynaptic PAC1-mediated signaling at the mossy fiber synapse is involved in both LTP and hippocampus-dependent associative learning.

Bradykinin, but not muscarinic, inhibition of M-current in rat sympathetic ganglion neurons involves phospholipase C-β4

Rat superior cervical ganglion (SCG) neurons express low-threshold noninactivating M-type potassium channels (I-K(M)), which can be inhibited by activation of M-1 muscarinic receptors (M-1 mAChR) and bradykinin (BK) B-2 receptors. Inhibition by the M1 mAChR agonist oxotremorine methiodide (Oxo-M) is mediated, at least in part, by the pertussis toxin-insensitive G-protein G alpha (q) (Caulfield et al., 1994; Haley et al., 1998a), whereas BK inhibition involves G alpha (q) and/or G alpha (11) (Jones et al., 1995). G alpha (q) and G alpha (11) can stimulate phospholipase C-beta (PLC-beta), raising the possibility that PLC is involved in I-K(M) inhibition by Oxo-M and BK. RT-PCR and antibody staining confirmed the presence of PLC-beta1, - beta2, - beta3, and - beta4 in rat SCG. We have tested the role of two PLC isoforms (PLC-beta1 and PLC-beta4) using antisense-expression constructs. Antisense constructs, consisting of the cytomegalovirus promoter driving antisense cRNA corresponding to the 3'-untranslated regions of PLC-beta1 and PLC-beta4, were injected into the nucleus of dissociated SCG neurons. Injected cells showed reduced antibody staining for the relevant PLC-beta isoform when compared to uninjected cells 48 hr later. BK inhibition of I-K(M) was significantly reduced 48 hr after injection of the PLC-beta4, but not the PLC-beta1, antisense-encoding plasmid. Neither PLC-beta antisense altered M-1 mAChR inhibition by Oxo-M. These data support the conclusion of Cruzblanca et al. (1998) that BK, but not M-1 mAChR, inhibition of I-K(M) involves PLC and extends this finding by indicating that PLC-beta4 is involved.

Impaired fast-spiking, suppressed cortical inhibition and increased susceptibility to seizures in mice lacking Kv3.2 K+ channel proteins

Voltage-gated K+ channels of the Kv3 subfamily have unusual electrophysiological properties, including activation at very depolarized voltages (positive to −10 mV) and very fast deactivation rates, suggesting special roles in neuronal excitability. In the brain, Kv3 channels are prominently expressed in select neuronal populations, which include fast-spiking (FS) GABAergic interneurons of the neocortex, hippocampus, and caudate, as well as other high-frequency firing neurons. Although evidence points to a key role in high-frequency firing, a definitive understanding of the function of these channels has been hampered by a lack of selective pharmacological tools. We therefore generated mouse lines in which one of the Kv3 genes, Kv3.2, was disrupted by gene-targeting methods. Whole-cell electrophysiological recording showed that the ability to fire spikes at high frequencies was impaired in immunocytochemically identified FS interneurons of deep cortical layers (5-6) in which Kv3.2 proteins are normally prominent. No such impairment was found for FS neurons of superficial layers (2-4) in which Kv3.2 proteins are normally only weakly expressed. These data directly support the hypothesis that Kv3 channels are necessary for high-frequency firing. Moreover, we found that Kv3.2 −/− mice showed specific alterations in their cortical EEG patterns and an increased susceptibility to epileptic seizures consistent with an impairment of cortical inhibitory mechanisms. This implies that, rather than producing hyperexcitability of the inhibitory interneurons, Kv3.2 channel elimination suppresses their activity. These data suggest that normal cortical operations depend on the ability of inhibitory interneurons to generate high-frequency firing.

K(+) channel expression distinguishes subpopulations of parvalbumin- and somatostatin-containing neocortical interneurons

Kv3.1 and Kv3.2 K+ channel proteins form similar voltage-gated K+ channels with unusual properties, including fast activation at voltages positive to −10 mV and very fast deactivation rates. These properties are thought to facilitate sustained high-frequency firing. Kv3.1 subunits are specifically found in fast-spiking, parvalbumin (PV)-containing cortical interneurons, and recent studies have provided support for a crucial role in the generation of the fast-spiking phenotype. Kv3.2 mRNAs are also found in a small subset of neocortical neurons, although the distribution of these neurons is different. We raised antibodies directed against Kv3.2 proteins and used dual-labeling methods to identify the neocortical neurons expressing Kv3.2 proteins and to determine their subcellular localization. Kv3.2 proteins are prominently expressed in patches in somatic and proximal dendritic membrane as well as in axons and presynaptic terminals of GABAergic interneurons. Kv3.2 subunits are found in all PV-containing neurons in deep cortical layers where they probably form heteromultimeric channels with Kv3.1 subunits. In contrast, in superficial layer PV-positive neurons Kv3.2 immunoreactivity is low, but Kv3.1 is still prominently expressed. Because Kv3.1 and Kv3.2 channels are differentially modulated by protein kinases, these results raise the possibility that the fast-spiking properties of superficial- and deep-layer PV neurons are differentially regulated by neuromodulators. Interestingly, Kv3.2 but not Kv3.1 proteins are also prominent in a subset of seemingly non-fast-spiking, somatostatin- and calbindin-containing interneurons, suggesting that the Kv3.1–Kv3.2 current type can have functions other than facilitating high-frequency firing.

From SNPs to pathways: integration of functional effect of sequence variations on models of cell signalling pathways

Background: Single nucleotide polymorphisms (SNPs) are the most frequent type of sequence variation between individuals, and represent a promising tool for finding genetic determinants of complex diseases and understanding the differences in drug response. In this regard, it is of particular interest to study the effect of non-synonymous SNPs in the context of biological networks such as cell signalling pathways. UniProt provides curated information about the functional and phenotypic effects of sequence variation, including SNPs, as well as on mutations of protein sequences. However, no strategy has been developed to integrate this information with biological networks, with the ultimate goal of studying the impact of the functional effect of SNPs in the structure and dynamics of biological networks. Results: First, we identified the different challenges posed by the integration of the phenotypic effect of sequence variants and mutations with biological networks. Second, we developed a strategy for the combination of data extracted from public resources, such as UniProt, NCBI dbSNP, Reactome and BioModels. We generated attribute files containing phenotypic and genotypic annotations to the nodes of biological networks, which can be imported into network visualization tools such as Cytoscape. These resources allow the mapping and visualization of mutations and natural variations of human proteins and their phenotypic effect on biological networks (e.g. signalling pathways, protein-protein interaction networks, dynamic models). Finally, an example on the use of the sequence variation data in the dynamics of a network model is presented. Conclusion: In this paper we present a general strategy for the integration of pathway and sequence variation data for visualization, analysis and modelling purposes, including the study of the functional impact of protein sequence variations on the dynamics of signalling pathways. This is of particular interest when the SNP or mutation is known to be associated to disease. We expect that this approach will help in the study of the functional impact of disease-associated SNPs on the behaviour of cell signalling pathways, which ultimately will lead to a better understanding of the mechanisms underlying complex diseases.

The neuronal basis of attention: rate versus synchronization modulation

Extensive theoretical and experimental work on the neuronal correlates of visual attention raises two hypotheses about the underlying mechanisms. The first hypothesis, named biased competition, originates from experimental single-cell recordings that have shown that attention upmodulates the firing rates of the neurons encoding the attended features and downregulates the firing rates of the neurons encoding the unattended features. Furthermore, attentional modulation of firing rates increases along the visual pathway. The other, newer hypothesis assigns synchronization a crucial role in the attentional process. It stems from experiments that have shown that attention modulates gamma-frequency synchronization. In this paper, we study the coexistence of the two phenomena using a theoretical framework. We find that the two effects can vary independently of each other and across layers. Therefore, the two phenomena are not concomitant. However, we show that there is an advantage in the processing of information if rate modulation is accompanied by gamma modulation, namely that reaction times are shorter, implying behavioral relevance for gamma synchronization.