Modeling High-Dimensional Audio Sequences with Recurrent Neural Networks

Cette thèse étudie des modèles de séquences de haute dimension basés sur des réseaux de neurones récurrents (RNN) et leur application à la musique et à la parole. Bien qu'en principe les RNN puissent représenter les dépendances à long terme et la dynamique temporelle complexe propres aux séquences d'intérêt comme la vidéo, l'audio et la langue naturelle, ceux-ci n'ont pas été utilisés à leur plein potentiel depuis leur introduction par Rumelhart et al. (1986a) en raison de la difficulté de les entraîner efficacement par descente de gradient. Récemment, l'application fructueuse de l'optimisation Hessian-free et d'autres techniques d'entraînement avancées ont entraîné la recrudescence de leur utilisation dans plusieurs systèmes de l'état de l'art. Le travail de cette thèse prend part à ce développement. L'idée centrale consiste à exploiter la flexibilité des RNN pour apprendre une description probabiliste de séquences de symboles, c'est-à-dire une information de haut niveau associée aux signaux observés, qui en retour pourra servir d'à priori pour améliorer la précision de la recherche d'information. Par exemple, en modélisant l'évolution de groupes de notes dans la musique polyphonique, d'accords dans une progression harmonique, de phonèmes dans un énoncé oral ou encore de sources individuelles dans un mélange audio, nous pouvons améliorer significativement les méthodes de transcription polyphonique, de reconnaissance d'accords, de reconnaissance de la parole et de séparation de sources audio respectivement. L'application pratique de nos modèles à ces tâches est détaillée dans les quatre derniers articles présentés dans cette thèse. Dans le premier article, nous remplaçons la couche de sortie d'un RNN par des machines de Boltzmann restreintes conditionnelles pour décrire des distributions de sortie multimodales beaucoup plus riches. Dans le deuxième article, nous évaluons et proposons des méthodes avancées pour entraîner les RNN. Dans les quatre derniers articles, nous examinons différentes façons de combiner nos modèles symboliques à des réseaux profonds et à la factorisation matricielle non-négative, notamment par des produits d'experts, des architectures entrée/sortie et des cadres génératifs généralisant les modèles de Markov cachés. Nous proposons et analysons également des méthodes d'inférence efficaces pour ces modèles, telles la recherche vorace chronologique, la recherche en faisceau à haute dimension, la recherche en faisceau élagué et la descente de gradient. Finalement, nous abordons les questions de l'étiquette biaisée, du maître imposant, du lissage temporel, de la régularisation et du pré-entraînement.

Prediction of Parkinson’s Disease tremor onset using a radial basis function neural network based on particle swarm optimization

Deep Brain Stimulation (DBS) has been successfully used throughout the world for the treatment of Parkinson's disease symptoms. To control abnormal spontaneous electrical activity in target brain areas DBS utilizes a continuous stimulation signal. This continuous power draw means that its implanted battery power source needs to be replaced every 18–24 months. To prolong the life span of the battery, a technique to accurately recognize and predict the onset of the Parkinson's disease tremors in human subjects and thus implement an on-demand stimulator is discussed here. The approach is to use a radial basis function neural network (RBFNN) based on particle swarm optimization (PSO) and principal component analysis (PCA) with Local Field Potential (LFP) data recorded via the stimulation electrodes to predict activity related to tremor onset. To test this approach, LFPs from the subthalamic nucleus (STN) obtained through deep brain electrodes implanted in a Parkinson patient are used to train the network. To validate the network's performance, electromyographic (EMG) signals from the patient's forearm are recorded in parallel with the LFPs to accurately determine occurrences of tremor, and these are compared to the performance of the network. It has been found that detection accuracies of up to 89% are possible. Performance comparisons have also been made between a conventional RBFNN and an RBFNN based on PSO which show a marginal decrease in performance but with notable reduction in computational overhead.

The occipital lateral plate mesoderm - a novel source for vertebrate neck musculature

In vertebrates, body musculature originates from somites, whereas head muscles originate from the cranial mesoderm. Neck muscles are located in the transition between these regions. We show that the chick occipital lateral plate mesoderm has myogenic capacity and gives rise to large muscles located in the neck and thorax. We present molecular and genetic evidence to show that these muscles not only have a unique origin, but additionally display a distinct temporal development, forming later than any other muscle group described to date. We further report that these muscles, found in the body of the animal, develop like head musculature rather than deploying the programme used by the trunk muscles. Using mouse genetics we reveal that these muscles are formed in trunk muscle mutants but are absent in head muscle mutants. In concordance with this conclusion, their connective tissue is neural crest in origin. Finally, we provide evidence that the mechanism by which these neck muscles develop is conserved in vertebrates.

The occipital lateral plate mesoderm is a novel source for vertebrate neck musculature

In vertebrates, body musculature originates from somites, whereas head muscles originate from the cranial mesoderm. Neck muscles are located in the transition between these regions. We show that the chick occipital lateral plate mesoderm has myogenic capacity and gives rise to large muscles located in the neck and thorax. We present molecular and genetic evidence to show that these muscles not only have a unique origin, but additionally display a distinct temporal development, forming later than any other muscle group described to date. We further report that these muscles, found in the body of the animal, develop like head musculature rather than deploying the programme used by the trunk muscles. Using mouse genetics we reveal that these muscles are formed in trunk muscle mutants but are absent in head muscle mutants. In concordance with this conclusion, their connective tissue is neural crest in origin. Finally, we provide evidence that the mechanism by which these neck muscles develop is conserved in vertebrates.

A dynamic causal model of the coupling between pulse stimulation and neural activity

We present a dynamic causal model that can explain context-dependent changes in neural responses, in the rat barrel cortex, to an electrical whisker stimulation at different frequencies. Neural responses were measured in terms of local field potentials. These were converted into current source density (CSD) data, and the time series of the CSD sink was extracted to provide a time series response train. The model structure consists of three layers (approximating the responses from the brain stem to the thalamus and then the barrel cortex), and the latter two layers contain nonlinearly coupled modules of linear second-order dynamic systems. The interaction of these modules forms a nonlinear regulatory system that determines the temporal structure of the neural response amplitude for the thalamic and cortical layers. The model is based on the measured population dynamics of neurons rather than the dynamics of a single neuron and was evaluated against CSD data from experiments with varying stimulation frequency (1–40 Hz), random pulse trains, and awake and anesthetized animals. The model parameters obtained by optimization for different physiological conditions (anesthetized or awake) were significantly different. Following Friston, Mechelli, Turner, and Price (2000), this work is part of a formal mathematical system currently being developed (Zheng et al., 2005) that links stimulation to the blood oxygen level dependent (BOLD) functional magnetic resonance imaging (fMRI) signal through neural activity and hemodynamic variables. The importance of the model described here is that it can be used to invert the hemodynamic measurements of changes in blood flow to estimate the underlying neural activity.

Theory and generalization of Monte Carlo models of the BOLD signal source

The dependency of the blood oxygenation level dependent (BOLD) signal on underlying hemodynamics is not well understood. Building a forward biophysical model of this relationship is important for the quantitative estimation of the hemodynamic changes and neural activity underlying functional magnetic resonance imaging (fMRI) signals. We have developed a general model of the BOLD signal which can model both intra- and extravascular signals for an arbitrary tissue model across a wide range of imaging parameters. The model of the BOLD signal was instantiated as a look-up-table (LuT), and was verified against concurrent fMRI and optical imaging measurements of activation induced hemodynamics. Magn Reson Med, 2008. © 2008 Wiley-Liss, Inc.

The hemodynamic impulse response to a single neural event

This article investigates the relation between stimulus-evoked neural activity and cerebral hemodynamics. Specifically, the hypothesis is tested that hemodynamic responses can be modeled as a linear convolution of experimentally obtained measures of neural activity with a suitable hemodynamic impulse response function. To obtain a range of neural and hemodynamic responses, rat whisker pad was stimulated using brief (less than or equal to2 seconds) electrical stimuli consisting of single pulses (0.3 millisecond, 1.2 mA) combined both at different frequencies and in a paired-pulse design. Hemodynamic responses were measured using concurrent optical imaging spectroscopy and laser Doppler flowmetry, whereas neural responses were assessed through current source density analysis of multielectrode recordings from a single barrel. General linear modeling was used to deconvolve the hemodynamic impulse response to a single "neural event" from the hemodynamic and neural responses to stimulation. The model provided an excellent fit to the empirical data. The implications of these results for modeling schemes and for physiologic systems coupling neural and hemodynamic activity are discussed.

Highly efficient neural differentiation of human somatic stem cells, isolated by minimally invasive periodontal surgery

Neural stem cells (NSCs) are potential sources for cell therapy of neurodegenerative diseases and for drug screening. Despite their potential benefits, ethical and practical considerations limit the application of NSCs derived from human embryonic stem cells (ES) or adult brain tissue. Thus, alternative sources are required to satisfy the criteria of ready accessibility, rapid expansion in chemically defined media and reliable induction to a neuronal fate. We isolated somatic stem cells from the human periodontium that were collected during minimally invasive periodontal access flap surgery as part of guided tissue regeneration therapy. These cells could be propagated as neurospheres in serum-free medium, which underscores their cranial neural crest cell origin. Culture in the presence of epidermal growth factor (EGF) and fibroblast growth factor-2 (FGF-2) under serum-free conditions resulted in large numbers of nestin-positive/Sox-2-positive NSCs. These periodontium-derived (pd) NSCs are highly proliferative and migrate in response to chemokines that have been described as inducing NSC migration. We used immunocytochemical techniques and RT-PCR analysis to assess neural differentiation after treatment of the expanded cells with a novel induction medium. Adherence to substrate, growth factor deprivation, and retinoic acid treatment led to the acquisition of neuronal morphology and stable expression of markers of neuronal differentiation by more than 90% of the cells. Thus, our novel method might provide nearly limitless numbers of neuronal precursors from a readily accessible autologous adult human source, which could be used as a platform for further experimental studies and has potential therapeutic implications.

Isolation of novel multipotent neural crest-derived stem cells from adult human inferior turbinate

Adult human neural crest-derived stem cells (NCSCs) are of extraordinary high plasticity and promising candidates for the use in regenerative medicine. Here we describe for the first time a novel neural crest-derived stem cell population within the respiratory epithelium of human adult inferior turbinate. In contrast to superior and middle turbinates, high amounts of source material could be isolated from human inferior turbinates. Using minimally-invasive surgery methods isolation is efficient even in older patients. Within their endogenous niche, inferior turbinate stem cells (ITSCs) expressed high levels of nestin, p75(NTR), and S100. Immunoelectron microscopy using anti-p75 antibodies displayed that ITSCs are of glial origin and closely related to nonmyelinating Schwann cells. Cultivated ITSCs were positive for nestin and S100 and the neural crest markers Slug and SOX10. Whole genome microarray analysis showed pronounced differences to human ES cells in respect to pluripotency markers OCT4, SOX2, LIN28, and NANOG, whereas expression of WDR5, KLF4, and c-MYC was nearly similar. ITSCs were able to differentiate into cells with neuro-ectodermal and mesodermal phenotype. Additionally ITSCs are able to survive and perform neural crest typical chain migration in vivo when transplanted into chicken embryos. However ITSCs do not form teratomas in severe combined immunodeficient mice. Finally, we developed a separation strategy based on magnetic cell sorting of p75(NTR) positive ITSCs that formed larger neurospheres and proliferated faster than p75(NTR) negative ITSCs. Taken together our study describes a novel, readily accessible source of multipotent human NCSCs for potential cell-replacement therapy.

Alternative generation of CNS neural stem cells and PNS derivatives from neural crest-derived peripheral stem cells

Neural crest-derived stem cells (NCSCs) from the embryonic peripheral nervous system (PNS) can be reprogrammed in neurosphere (NS) culture to rNCSCs that produce central nervous system (CNS) progeny, including myelinating oligodendrocytes. Using global gene expression analysis we now demonstrate that rNCSCs completely lose their previous PNS characteristics and acquire the identity of neural stem cells derived from embryonic spinal cord. Reprogramming proceeds rapidly and results in a homogenous population of Olig2-, Sox3-, and Lex-positive CNS stem cells. Low-level expression of pluripotency inducing genes Oct4, Nanog, and Klf4 argues against a transient pluripotent state during reprogramming. The acquisition of CNS properties is prevented in the presence of BMP4 (BMP NCSCs) as shown by marker gene expression and the potential to produce PNS neurons and glia. In addition, genes characteristic for mesenchymal and perivascular progenitors are expressed, which suggests that BMP NCSCs are directed toward a pericyte progenitor/mesenchymal stem cell (MSC) fate. Adult NCSCs from mouse palate, an easily accessible source of adult NCSCs, display strikingly similar properties. They do not generate cells with CNS characteristics but lose the neural crest markers Sox10 and p75 and produce MSC-like cells. These findings show that embryonic NCSCs acquire a full CNS identity in NS culture. In contrast, MSC-like cells are generated from BMP NCSCs and pNCSCs, which reveals that postmigratory NCSCs are a source for MSC-like cells up to the adult stage.

Maintenance and differentiation of neural stem cells

The adult mammalian brain contains self-renewable, multipotent neural stem cells (NSCs) that are responsible for neurogenesis and plasticity in specific regions of the adult brain. Extracellular matrix, vasculature, glial cells, and other neurons are components of the niche where NSCs are located. This surrounding environment is the source of extrinsic signals that instruct NSCs to either self-renew or differentiate. Additionally, factors such as the intracellular epigenetics state and retrotransposition events can influence the decision of NSC`s fate into neurons or glia. Extrinsic and intrinsic factors form an intricate signaling network, which is not completely understood. These factors altogether reflect a few of the key players characterized so far in the new field of NSC research and are covered in this review. (C) 2010 John Wiley & Sons, Inc. WIREs Syst Biol Med 2011 3 107-114 DOI:10.1002/wsbm:100

NDVI With Artificial Neural Networks For SRTM Elevation Model Improvement – Hydrological Model Application

Digital elevation model (DEM) plays a substantial role in hydrological study, from understanding the catchment characteristics, setting up a hydrological model to mapping the flood risk in the region. Depending on the nature of study and its objectives, high resolution and reliable DEM is often desired to set up a sound hydrological model. However, such source of good DEM is not always available and it is generally high-priced. Obtained through radar based remote sensing, Shuttle Radar Topography Mission (SRTM) is a publicly available DEM with resolution of 92m outside US. It is a great source of DEM where no surveyed DEM is available. However, apart from the coarse resolution, SRTM suffers from inaccuracy especially on area with dense vegetation coverage due to the limitation of radar signals not penetrating through canopy. This will lead to the improper setup of the model as well as the erroneous mapping of flood risk. This paper attempts on improving SRTM dataset, using Normalised Difference Vegetation Index (NDVI), derived from Visible Red and Near Infra-Red band obtained from Landsat with resolution of 30m, and Artificial Neural Networks (ANN). The assessment of the improvement and the applicability of this method in hydrology would be highlighted and discussed.

An analog implementation of radial basis neural networks (RBNN) using BiCMOS technology

This paper describes a analog implementation of radial basis neural networks (RBNN) in BiCMOS technology. The RBNN uses a gaussian function obtained through the characteristic of the bipolar differential pair. The gaussian parameters (gain, center and width) is changed with programmable current source. Results obtained with PSPICE software is showed.

Harmonic content identification based on neural method for single phase power systems

An alternative method is presented in this paper to identify the harmonic components of non-linear loads in single phase power systems based on artificial neural networks. The components are identified by analyzing the single phase current waveform in time domain in half-cycle of the ac voltage source. The proposed method is compared to the fast Fourier transform. Simulation and experimental results are presented to validate the proposed approach.

Deconvolução de perfis de poço através de rede neural recorrente

Para a indústria do petróleo, a interpretação dos perfis de poço é a principal fonte de informação sobre a presença e quantificação de hidrocarbonetos em subsuperfície. Entretanto, em duas situações as novas tecnologias, tanto em termos do processo construtivo das ferramentas, quanto da transmissão dos dados não têm justificativa econômica, ensejando a utilização de um conjunto de perfis convencionais: reavaliações de campos maduros e avaliações de campos marginais. Os procedimentos de aquisição dos perfis convencionais podem alterar o valor da propriedade física bem como a localização dos limites verticais de uma camada rochosa. Este é um antigo problema na geofísica de poço – o paradoxo entre a resolução vertical e a profundidade de investigação de uma ferramenta de perfilagem. Hoje em dia, isto é contornado através da alta tecnologia na construção das novas ferramentas, entretanto, este problema ainda persiste no caso das ferramentas convencionais como, a ferramenta de raio gama natural (GR). Apresenta-se, neste trabalho, um novo método para atenuar as alterações induzidas no perfil pela ferramenta, através da integração do clássico modelo convolucional do perfil com as redes neurais recorrentes. Assume-se que um perfil de poço pode ser representado através da operação de convolução em profundidade entre a variação da propriedade física da rocha (perfil ideal) e uma função que representa a alteração produzida sobre a propriedade física, chamada como resposta vertical da ferramenta. Assim, desenvolve-se um processamento iterativo dos perfis, o qual atua na forma da operação de deconvolução, composto por três redes neurais recorrentes. A primeira visa estimar a resposta vertical da ferramenta; a segunda procura definir os limites verticais de cada camada rochosa e a última é construída para estimar o valor real da propriedade física. Este processamento é iniciado com uma estimativa externa tanto para o perfil ideal, quanto para a resposta vertical da ferramenta. Finalmente, mostram-se as melhorias na resolução vertical e na avaliação da propriedade física produzida por esta metodologia em perfis sintéticos e em perfis reais da formação Lagunillas, bacia do Lago Maracaibo, Venezuela.