On Recurrent and Deep Neural Networks

L'apprentissage profond est un domaine de recherche en forte croissance en apprentissage automatique qui est parvenu à des résultats impressionnants dans différentes tâches allant de la classification d'images à la parole, en passant par la modélisation du langage. Les réseaux de neurones récurrents, une sous-classe d'architecture profonde, s'avèrent particulièrement prometteurs. Les réseaux récurrents peuvent capter la structure temporelle dans les données. Ils ont potentiellement la capacité d'apprendre des corrélations entre des événements éloignés dans le temps et d'emmagasiner indéfiniment des informations dans leur mémoire interne. Dans ce travail, nous tentons d'abord de comprendre pourquoi la profondeur est utile. Similairement à d'autres travaux de la littérature, nos résultats démontrent que les modèles profonds peuvent être plus efficaces pour représenter certaines familles de fonctions comparativement aux modèles peu profonds. Contrairement à ces travaux, nous effectuons notre analyse théorique sur des réseaux profonds acycliques munis de fonctions d'activation linéaires par parties, puisque ce type de modèle est actuellement l'état de l'art dans différentes tâches de classification. La deuxième partie de cette thèse porte sur le processus d'apprentissage. Nous analysons quelques techniques d'optimisation proposées récemment, telles l'optimisation Hessian free, la descente de gradient naturel et la descente des sous-espaces de Krylov. Nous proposons le cadre théorique des méthodes à région de confiance généralisées et nous montrons que plusieurs de ces algorithmes développés récemment peuvent être vus dans cette perspective. Nous argumentons que certains membres de cette famille d'approches peuvent être mieux adaptés que d'autres à l'optimisation non convexe. La dernière partie de ce document se concentre sur les réseaux de neurones récurrents. Nous étudions d'abord le concept de mémoire et tentons de répondre aux questions suivantes: Les réseaux récurrents peuvent-ils démontrer une mémoire sans limite? Ce comportement peut-il être appris? Nous montrons que cela est possible si des indices sont fournis durant l'apprentissage. Ensuite, nous explorons deux problèmes spécifiques à l'entraînement des réseaux récurrents, à savoir la dissipation et l'explosion du gradient. Notre analyse se termine par une solution au problème d'explosion du gradient qui implique de borner la norme du gradient. Nous proposons également un terme de régularisation conçu spécifiquement pour réduire le problème de dissipation du gradient. Sur un ensemble de données synthétique, nous montrons empiriquement que ces mécanismes peuvent permettre aux réseaux récurrents d'apprendre de façon autonome à mémoriser des informations pour une période de temps indéfinie. Finalement, nous explorons la notion de profondeur dans les réseaux de neurones récurrents. Comparativement aux réseaux acycliques, la définition de profondeur dans les réseaux récurrents est souvent ambiguë. Nous proposons différentes façons d'ajouter de la profondeur dans les réseaux récurrents et nous évaluons empiriquement ces propositions.

Altered inhibitory synaptic transmission in superficial dorsal horn neurones in spastic and oscillator mice

The spastic (spa) and oscillator (ot) mouse have naturally occurring mutations in the inhibitory glycine receptor (GlyR) and exhibit severe motor disturbances when exposed to unexpected sensory stimuli. We examined the effects of the spa and ot mutations on GlyR- and GABA(A)R-mediated synaptic transmission in the superficial dorsal horn (SFDH), a spinal cord region where inhibition is important for nociceptive processing. Spontaneous mIPSCs were recorded from visually identified neurones in parasagittal spinal cord slices. Neurones received exclusively GABA(A)R-mediated mIPSCs, exclusively GlyR-mediated mIPSCs or both types of mIPSCs. In control mice (wild-type and spa/+) over 40 % of neurones received both types of mIPSCs, over 30 % received solely GABA(A)R-mediated mIPSCs and the remainder received solely GlyR-mediated mIPSCs. In spa/spa animals, 97 % of the neurones received exclusively GABA(A)ergic or both types of mIPSCs. In ot/ot animals, over 80 % of the neurones received exclusively GABA(A)R-mediated mIPSCs. GlyR-mediated mIPSC amplitude and charge were reduced in spa/spa and ot/ot animals. GABA,Rmediated mIPSC amplitude and charge were elevated in spa/spa but unaltered in ot/ot animals. GlyR- and GABA(A)R-mediated mIPSC decay times were similar for all genotypes, consistent with the mutations altering receptor numbers but not kinetics. These findings suggest the spastic and oscillator mutations, traditionally considered motor disturbances, also disrupt inhibition in a sensory region associated with nociceptive transmission. Furthermore, the spastic mutation results in a compensatory increase in GABA(A)ergic transmission in SFDH neurones, a form of inhibitory synaptic plasticity absent in the oscillator mouse.

Alpha-oestrogen and progestin receptor expression in the hypothalamus and preoptic area dopaminergic neurones during oestrous in cycling rats

A secretory surge of prolactin occurs on the afternoon of oestrous in cycling rats. Although prolactin is regulated by ovarian steroids, plasma oestradiol and progesterone levels do not vary during oestrous. Because prolactin release is tonically inhibited by hypothalamic dopamine and modulated by dopamine transmission in the preoptic area (POA), the present study aimed to evaluate whether oestrogen receptor (ER)-alpha and progestin receptor (PR) expression in the dopaminergic neurones of arcuate (ARC), periventricular, anteroventral periventricular (AVPe) and ventromedial preoptic (VMPO) nuclei changes during the day of oestrous. Cycling rats were perfused every 2 h from 10-20 h on oestrous. Brain sections were double-labelled to ER alpha or PR and tyrosine hydroxylase (TH). The number of TH-immunoreactive (ir) neurones did not vary significantly in any area evaluated. ER alpha expression in TH-ir neurones increased at 14 and 16 h in the rostral-ARC and dorsomedial-ARC, 14 h in the caudal-ARC and 16 h in the VMPO, whereas it was unaltered in the ventrolateral-ARC, periventricular and AVPe. PR expression in TH-ir neurones of the periventricular and rostral, dorsomedial, ventrolateral and caudal-ARC decreased transitorily during the afternoon, showing the lowest levels between 14 and 16 h; but it did not vary in the AVPe and VMPO. Plasma oestradiol and progesterone concentrations were low and unaltered during oestrous, indicating that the changes in receptors expression were probably not due to variation in ligand levels. Thus, our data suggest that variations in ER alpha and PR expression may promote changes in the activity of medial basal hypothalamus and POA dopaminergic neurones, even under unaltered secretion of ovarian steroids, which could facilitate the occurrence and modulate the magnitude of the prolactin surge on oestrous.

Activity of Hypothalamic Dopaminergic Neurones During the Day of Oestrus: Involvement in Prolactin Secretion

A secretory surge of prolactin occurs on the afternoon of oestrus in cycling rats. Pituitary prolactin is inhibited by dopamine. We evaluated the activity of the neuroendocrine dopaminergic neurones during oestrus and dioestrus, as determined by dopaminergic activity in the median eminence and neurointermediate lobe of the pituitary, as well as Fos-related antigen expression in tyrosine hydroxylase (TH)-immunoreactive (ir) neurones of the arcuate nucleus (ARC) and periventricular nucleus (Pe). During oestrus, the 4-dihydroxyphenylacetic acid/dopamine ratio in the median eminence decreased at 16.00 h, coinciding with the increase in plasma prolactin levels. Similarly, the expression of Fos-related antigen in TH-ir neurones of Pe and rostral-, dorsomedial- and caudal-ARC also decreased at 16.00 h. On dioestrus, 4-dihydroxyphenylacetic acid/dopamine ratio in the median eminence and Fos-related antigen expression in TH-ir neurones of Pe and rostral-ARC decreased at 18.00 h, whereas prolactin levels were unaltered. No variation in dopaminergic activity was found in the neurointermediate lobe of the pituitary on either oestrus or dioestrus. The number of TH-ir neurones in the ARC and parameters of dopaminergic activity were found to be generally lower on oestrus compared to dioestrus. The transitory decrease in the activity of neuroendocrine dopaminergic neurones temporally associated with the prolactin surge on the afternoon of oestrus suggests a role for dopamine in the generation of the oestrous prolactin surge.

Medullary neurones regulate hypothalamic corticotropin-releasing factor cell responses to an emotional stressor

Hypothalamic-pituitary-adrenal axis activation is a hallmark of the stress response. In the case of physical stressors, there is considerable evidence that medullary catecholamine neurones are critical to the activation of the paraventricular nucleus corticotropin-releasing factor cells that constitute the apex of the hypothalamic-pituitary-adrenal axis. In contrast, it has been thought that hypothalamic-pituitary-adrenal axis responses to emotional stressors do not involve brainstem neurones. To investigate this issue we have mapped patterns of restraint-induced neuronal c fos expression in intact animals and in animals prepared with either paraventricular nucleus-directed injections of a retrograde tracer, lesions of paraventricular nucleus catecholamine terminals, or lesions of the medulla corresponding to the A1 or A2 noradrenergic cell groups. Restraint-induced patterns of neuronal activation within the medulla of intact animals were very similar to those previously reported in response to physical stressors, including the fact that most stressor-responsive, paraventricular nucleus-projecting cells were certainly catecholaminergic and probably noradrenergic. Despite this, the destruction of paraventricular nucleus catecholamine terminals with 6-hydroxydopamine did not alter corticotropin-releasing factor cell responses to restraint. However, animals with ibotenic acid lesions encompassing either the A1 or A2 noradrenergic cell groups displayed significantly suppressed corticotropin-releasing factor cell responses to restraint. Notably, these medullary lesions also suppressed neuronal responses in the medial amygdala, an area that is now considered critical to hypothalamic-pituitary-adrenal axis responses to emotional stressors and that is also known to display a significant increase in noradrenaline turnover during restraint. We conclude that medullary neurones influence corticotropin-releasing factor cell responses to emotional stressors via a multisynaptic pathway that may involve a noradrenergic input to the medial amygdala. These results overturn the idea that hypothalamic-pituitary-adrenal axis response to emotional stressors can occur independently of the brainstem. (C) 2001 IBRO. Published by Elsevier Science Ltd. All rights reserved.

Pyramidal neurones in macaque visual cortex: Interareal phenotypic variation of dendritic branching patterns

The basal dendritic arbors of over 500-layer III pyramidal neurones of the macaque cortex were compared by fractal analyses, which provides a measure of the space filling (or branching pattern) of dendritic arbors. Fractal values (D) of individual cells were compared between the cytochrome oxidase (CO)-rich blobs and CO-poor interblobs, of middle and upper layer III, and between sublaminae, in the primary visual area (Vi). These data were compared with those in the CO compartments in the second visual area (V2), and seven other extrastriate cortical areas. (V4, MT, LIP, 7a, TEO, TE and STP). There were significant differences in the fractal dimensions, and therefore the dendritic branching patterns, of cells in striate and extrastriate areas. Of the 55 possible pairwise comparisons of fractal dimension of neurones in different cortical areas (or CO compartments), 39 proved to be significantly different. The markedly different morphologies of pyramidal cells in the different cortical areas may be one of the features that determine the functional signatures of these cells by influencing the number of inputs received by, and propagation of potentials through, their dendritic arbors.

An ATP-sensitive K+ conductance in dissociated neurones from adult rat intracardiac ganglia

1. An ATP-sensitive K+ (K-ATP) conductance has been identified using the perforated patch recording configuration in a population (52%) of dissociated neurones from adult rat intracardiac ganglia. The presence of the sulphonylurea receptor in approximately half of the intracardiac neurones was confirmed by labelling with fluorescent glibenclamide-BODIPY. 2. Under current clamp conditions in physiological solutions, leveromakalim (10 muM) evoked a hyperpolarization, which was inhibited by the sulphonylurea drugs glibenclamide and tolbutamide. 3. Under voltage clamp conditions in symmetrical (140 mM) K+ solutions, hath application of levcromakalim evoked an inward current with a density of similar to8 pA pF(-1) at -50 mV and a slope conductance of similar to9 nS, which reversed close to the potassium equilibrium potential (E-K). Cell dialysis with an ATP-free intracellular solution also evoked an inward current, which was inhibited by tolbutamide. 4. Bath application of either glibenclamide (10 muM) or tolbutamide (100 muM) depolarized adult intracardiac neurones by 3-5 mV, suggesting that a K-ATP conductance is activated under resting conditions and contributes to the resting membrane potential. 5. Activation of a membrane current by levcromakalim leas concentration dependent, with an EC50 of 1.6 muM. Inhibition of the levcromakalim-activated current by glibenclamide leas also concentration dependent, with an IC50 of 55 nM. 6. Metabolic inhibition with 2,4-dinitrophenol and iodoacetic acid or superfusion with hypoxic solution (P-O2 similar to 16 mmHg) also activated a membrane current. These currents exhibited similar I-P characteristics to the levcroinakalim-induced current and were inhibited by glibenclamide. 7. Activation of K-ATP channels in mammalian intracardiac neurones may contribute to changes in neural regulation of the mature heart and. cardiac function during ischaemia-reperfusion.

Ionic selectivity of native ATP-activated (P2X) receptor channels in dissociated neurones from rat parasympathetic ganglia

1. The relative permeability of the native P2X receptor channel to monovalent and divalent inorganic and organic cations was determined from reversal potential measurements of ATP-evoked currents in parasympathetic neurones dissociated from rat submandibular ganglia using the dialysed whole-cell patch clamp technique. 2. The P2X receptor-channel exhibited weak selectivity among the alkali metals with a selectivity sequence of Na+ > Li+ > Cs+ > Rb+ > K+, and permeability ratios relative to Cs+ (P-X/P-Cs) ranging from 1. 11 to 0.86. 3. The selectivity for the divalent alkaline earth cations was also weak with the sequence Ca2+ > Sr2+ > Ba2+ > Mn2+ > Mg2+. ATP-evoked currents were strongly inhibited when the extracellular divalent cation concentration was increased. 4, The calculated permeability ratios of different ammonium cations are higher than those of the alkali metal cations. The permeability sequence obtained for the saturated organic cations is inversely correlated with the size of the cation. The unsaturated organic cations have a higher permeability than that predicted by molecular size. 5. Acidification to pH 6.2 increased the ATP-induced current amplitude twofold, whereas alkalization to 8.2 and 9.2 markedly reduced current amplitude. Cell dialysis with either anti-P2X(2) and/or anti-P2X(4) but not anti-P2X(1) antibodies attenuated the ATP-evoked current amplitude. Taken together, these data are consistent with homomeric and/or heteromeric P2X(2) and P2X(4) receptor subtypes expressed in rat submandibular neurones. 6. The permeability ratios for the series of monovalent organic cations, with the exception of unsaturated cations, were approximately related to the ionic size. The relative permeabilities of the monovalent inoganic and organic cations tested are similar to those reported previously for cloned rat P2X2 receptors expressed in mammalian cells.

Excited to death: different ways to lose your neurones

The selective loss of neurones in a range of neurodegenerative diseases is widely thought to involve the process of excitotoxicity, in which glutamate-mediated neuronal killing is elaborated through the excessive stimulation of cell-surface receptors. Every such disease exhibits a distinct regional and subregional pattern of neuronal loss. so processes must be locally triggered to different extents to account for this. We have studied several mechanisms which could lead to excitotoxic glutamate pathophysiology and compared them in different diseases. Our data suggest that glutamate can reach toxic extracellular levels in Alzheimer disease by malfunctions in cellular transporters, and that the toxicity may be exacerbated by continued glutamate release from presynaptic neurones acting on hypersensitive postsynaptic receptors. Thus the excitotoxicity is direct. In contrast, alcoholic brain damage arises in regions where GABA-mediated inhibition is deficient, and fails properly to dampen trans-synaptic excitation, Thus the excitotoxicity is indirect. A variety of such mechanisms is possible, which may combine in different ways.

C/EBPbeta couples dopamine signalling to substance P precursor gene expression in striatal neurones.

Dopamine-induced changes in striatal gene expression are thought to play an important role in drug addiction and compulsive behaviour. In this study we report that dopamine induces the expression of the transcription factor CCAAT/Enhancer Binding Protein beta (C/EBP)-beta in primary cultures of striatal neurones. We identified the preprotachykinin-A (PPT-A) gene coding for substance P and neurokinin-A as a potential target gene of C/EBPbeta. We demonstrated that C/EBPbeta physically interacts with an element of the PPT-A promoter, thereby facilitating substance P precursor gene transcription. The regulation of PPT-A gene by C/EBPbeta could subserve many important physiological processes involving substance P, such as nociception, neurogenic inflammation and addiction. Given that substance P is known to increase dopamine signalling in the striatum and, in turn, dopamine increases substance P expression in medium spiny neurones, our results implicate C/EBPbeta in a positive feedback loop, changes of which might contribute to the development of drug addiction.

Increased vulnerability of neurones and glial cells to low concentrations of methylmercury in a prooxidant situation.

Using reaggregating rat brain cell cultures at two different stages of differentiation, we examined the biochemical effects of a 10-day treatment with nanomolar concentrations of methylmercuric chloride (monomethylmercury), in the presence or absence of promoters of hydroxyl radical formation (10 microM copper sulphate plus 100 microM ascorbate). A decrease in total protein content accounted for the general cytotoxicity of these compounds, whereas selective effects were assessed by determining the activities of cell type-specific enzymes. Methylmercury, up to 100 nM, as well as the copper ascorbate mixture, when applied separately, induced no general cytotoxicity, and only slight effects on neuronal parameters. However, when applying 100 nM methylmercury and the copper-ascorbate mixture together, a drastic decrease in neuronal and glial parameters was found. Under these conditions, the content of reactive oxygen species, assessed by 2',7'-dichlorofluorescin oxidation, increased greatly, while the activities of antioxidant enzymes decreased. In the presence of copper and ascorbate, differentiated cultures appeared more resistant than immature ones to low methylmercury concentrations (1-10 mM), but did undergo similar changes in both cell type-specific and antioxidant enzyme activities at 100 nM methylmercury. These results suggest that in prooxidant conditions low doses of mercury can become much more deleterious for the central nervous system.

Inhibition of voltage-gated Na(+) currents in sensory neurones by the sea anemone toxin APETx2.

BACKGROUND AND PURPOSE: APETx2, a toxin from the sea anemone Anthropleura elegantissima, inhibits acid-sensing ion channel 3 (ASIC3)-containing homo- and heterotrimeric channels with IC(50) values < 100 nM and 0.1-2 µM respectively. ASIC3 channels mediate acute acid-induced and inflammatory pain response and APETx2 has been used as a selective pharmacological tool in animal studies. Toxins from sea anemones also modulate voltage-gated Na(+) channel (Na(v) ) function. Here we tested the effects of APETx2 on Na(v) function in sensory neurones.¦EXPERIMENTAL APPROACH: Effects of APETx2 on Na(v) function were studied in rat dorsal root ganglion (DRG) neurones by whole-cell patch clamp.¦KEY RESULTS: APETx2 inhibited the tetrodotoxin (TTX)-resistant Na(v) 1.8 currents of DRG neurones (IC(50) , 2.6 µM). TTX-sensitive currents were less inhibited. The inhibition of Na(v) 1.8 currents was due to a rightward shift in the voltage dependence of activation and a reduction of the maximal macroscopic conductance. The inhibition of Na(v) 1.8 currents by APETx2 was confirmed with cloned channels expressed in Xenopus oocytes. In current-clamp experiments in DRG neurones, the number of action potentials induced by injection of a current ramp was reduced by APETx2.¦CONCLUSIONS AND IMPLICATIONS: APETx2 inhibited Na(v) 1.8 channels, in addition to ASIC3 channels, at concentrations used in in vivo studies. The limited specificity of this toxin should be taken into account when using APETx2 as a pharmacological tool. Its dual action will be an advantage for the use of APETx2 or its derivatives as analgesic drugs.