On Recurrent and Deep Neural Networks

L'apprentissage profond est un domaine de recherche en forte croissance en apprentissage automatique qui est parvenu à des résultats impressionnants dans différentes tâches allant de la classification d'images à la parole, en passant par la modélisation du langage. Les réseaux de neurones récurrents, une sous-classe d'architecture profonde, s'avèrent particulièrement prometteurs. Les réseaux récurrents peuvent capter la structure temporelle dans les données. Ils ont potentiellement la capacité d'apprendre des corrélations entre des événements éloignés dans le temps et d'emmagasiner indéfiniment des informations dans leur mémoire interne. Dans ce travail, nous tentons d'abord de comprendre pourquoi la profondeur est utile. Similairement à d'autres travaux de la littérature, nos résultats démontrent que les modèles profonds peuvent être plus efficaces pour représenter certaines familles de fonctions comparativement aux modèles peu profonds. Contrairement à ces travaux, nous effectuons notre analyse théorique sur des réseaux profonds acycliques munis de fonctions d'activation linéaires par parties, puisque ce type de modèle est actuellement l'état de l'art dans différentes tâches de classification. La deuxième partie de cette thèse porte sur le processus d'apprentissage. Nous analysons quelques techniques d'optimisation proposées récemment, telles l'optimisation Hessian free, la descente de gradient naturel et la descente des sous-espaces de Krylov. Nous proposons le cadre théorique des méthodes à région de confiance généralisées et nous montrons que plusieurs de ces algorithmes développés récemment peuvent être vus dans cette perspective. Nous argumentons que certains membres de cette famille d'approches peuvent être mieux adaptés que d'autres à l'optimisation non convexe. La dernière partie de ce document se concentre sur les réseaux de neurones récurrents. Nous étudions d'abord le concept de mémoire et tentons de répondre aux questions suivantes: Les réseaux récurrents peuvent-ils démontrer une mémoire sans limite? Ce comportement peut-il être appris? Nous montrons que cela est possible si des indices sont fournis durant l'apprentissage. Ensuite, nous explorons deux problèmes spécifiques à l'entraînement des réseaux récurrents, à savoir la dissipation et l'explosion du gradient. Notre analyse se termine par une solution au problème d'explosion du gradient qui implique de borner la norme du gradient. Nous proposons également un terme de régularisation conçu spécifiquement pour réduire le problème de dissipation du gradient. Sur un ensemble de données synthétique, nous montrons empiriquement que ces mécanismes peuvent permettre aux réseaux récurrents d'apprendre de façon autonome à mémoriser des informations pour une période de temps indéfinie. Finalement, nous explorons la notion de profondeur dans les réseaux de neurones récurrents. Comparativement aux réseaux acycliques, la définition de profondeur dans les réseaux récurrents est souvent ambiguë. Nous proposons différentes façons d'ajouter de la profondeur dans les réseaux récurrents et nous évaluons empiriquement ces propositions.

Immune potential and differentiation of equine induced pluripotent stem cells (eiPSC)

Induced pluripotent stem cells (iPSC) have the capacity to self renew and differentiate into a myriad of cell types making them potential candidates for cell therapy and regenerative medicine. The goal of this thesis was to determine the characteristics of equine iPSC (eiPSC) that can be harnessed for potential use in veterinary regenerative medicine. Trauma to a horse’s limb often leads to the development of a chronic non-healing wound that lacks a keratinocyte cover, vital to healing. Thus, the overall hypothesis of this thesis was that eiPSC might offer a solution for providing wound coverage for such problematic wounds. Prior to considering eiPSC for clinical applications, their immunogenicity must be studied to ensure that the transplanted cells will be accepted and integrate into host tissues. The first objective of this thesis was to determine the immune response to eiPSC. To investigate the immunogenicity of eiPSC, the expression of major histocompatibility complex (MHC) molecules by the selected lines was determined, then the cells were used in an intradermal transplantation model developed for this study. While transplantation of allogeneic, undifferentiated eiPSC elicited a moderate cellular response in experimental horses, it did not cause acute rejection. This strategy enabled the selection of weakly immunogenic eiPSC lines for subsequent differentiation into lineages of therapeutic importance. Equine iPSC offer a potential solution to deficient epithelial coverage by providing a keratinocyte graft with the ability to differentiate into other accessory structures of the epidermis. The second objective of this thesis was to develop a protocol for the differentiation of eiPSC into a keratinocyte lineage. The protocol was shown to be highly efficient at inducing the anticipated phenotype within 30 days. Indeed, the eiPSC derived vi keratinocytes (eiPSC-KC) showed both morphologic and functional characteristics of primary equine keratinocytes (PEK). Moreover, the proliferative capacity of eiPSC-KC was superior while the migratory capacity, measured as the ability to epithelialize in vitro wounds, was comparable to that of PEK, suggesting exciting potential for grafting onto in vivo wound models. In conclusion, equine iPSC-derived keratinocytes exhibit features that are promising to the development of a stem cell-based skin construct with the potential to fully regenerate lost or damaged skin in horses. However, since eiPSC do not fully escape immune surveillance despite low MHC expression, strategies to improve engraftment of iPSC derivatives must be pursued.

Modelling and Analysis of Recurrent Event Data with Multiple Causes

This thesis Entitled “modelling and analysis of recurrent event data with multiple causes.Survival data is a term used for describing data that measures the time to occurrence of an event.In survival studies, the time to occurrence of an event is generally referred to as lifetime.Recurrent event data are commonly encountered in longitudinal studies when individuals are followed to observe the repeated occurrences of certain events. In many practical situations, individuals under study are exposed to the failure due to more than one causes and the eventual failure can be attributed to exactly one of these causes.The proposed model was useful in real life situations to study the effect of covariates on recurrences of certain events due to different causes.In Chapter 3, an additive hazards model for gap time distributions of recurrent event data with multiple causes was introduced. The parameter estimation and asymptotic properties were discussed .In Chapter 4, a shared frailty model for the analysis of bivariate competing risks data was presented and the estimation procedures for shared gamma frailty model, without covariates and with covariates, using EM algorithm were discussed. In Chapter 6, two nonparametric estimators for bivariate survivor function of paired recurrent event data were developed. The asymptotic properties of the estimators were studied. The proposed estimators were applied to a real life data set. Simulation studies were carried out to find the efficiency of the proposed estimators.

MicroRNA–mRNA interaction network using TSK-type recurrent neural fuzzy network

MicroRNAs are short non-coding RNAs that can regulate gene expression during various crucial cell processes such as differentiation, proliferation and apoptosis. Changes in expression profiles of miRNA play an important role in the development of many cancers, including CRC. Therefore, the identification of cancer related miRNAs and their target genes are important for cancer biology research. In this paper, we applied TSK-type recurrent neural fuzzy network (TRNFN) to infer miRNA–mRNA association network from paired miRNA, mRNA expression profiles of CRC patients. We demonstrated that the method we proposed achieved good performance in recovering known experimentally verified miRNA–mRNA associations. Moreover, our approach proved successful in identifying 17 validated cancer miRNAs which are directly involved in the CRC related pathways. Targeting such miRNAs may help not only to prevent the recurrence of disease but also to control the growth of advanced metastatic tumors. Our regulatory modules provide valuable insights into the pathogenesis of cancer

Construction of recurrent fractal interpolation surfaces(RFISs) on rectangular grids

A recurrent iterated function system (RIFS) is a genaralization of an IFS and provides nonself-affine fractal sets which are closer to natural objects. In general, it's attractor is not a continuous surface in R3. A recurrent fractal interpolation surface (RFIS) is an attractor of RIFS which is a graph of bivariate continuous interpolation function. We introduce a general method of generating recurrent interpolation surface which are at- tractors of RIFSs about any data set on a grid.

Image compression predicated on recurrent iterated function systems

Recurrent iterated function systems (RIFSs) are improvements of iterated function systems (IFSs) using elements of the theory of Marcovian stochastic processes which can produce more natural looking images. We construct new RIFSs consisting substantially of a vertical contraction factor function and nonlinear transformations. These RIFSs are applied to image compression.

Using Recurrent Networks for Dimensionality Reduction

This report explores how recurrent neural networks can be exploited for learning high-dimensional mappings. Since recurrent networks are as powerful as Turing machines, an interesting question is how recurrent networks can be used to simplify the problem of learning from examples. The main problem with learning high-dimensional functions is the curse of dimensionality which roughly states that the number of examples needed to learn a function increases exponentially with input dimension. This thesis proposes a way of avoiding this problem by using a recurrent network to decompose a high-dimensional function into many lower dimensional functions connected in a feedback loop.

Exact Solution of the Nonlinear Dynamics of Recurrent Neural Mechanisms for Direction Selectivity

Different theoretical models have tried to investigate the feasibility of recurrent neural mechanisms for achieving direction selectivity in the visual cortex. The mathematical analysis of such models has been restricted so far to the case of purely linear networks. We present an exact analytical solution of the nonlinear dynamics of a class of direction selective recurrent neural models with threshold nonlinearity. Our mathematical analysis shows that such networks have form-stable stimulus-locked traveling pulse solutions that are appropriate for modeling the responses of direction selective cortical neurons. Our analysis shows also that the stability of such solutions can break down giving raise to a different class of solutions ("lurching activity waves") that are characterized by a specific spatio-temporal periodicity. These solutions cannot arise in models for direction selectivity with purely linear spatio-temporal filtering.

Frequency of Low-level Mosaicism in X-Cromosome in Couples with Antecedent of Recurrent Miscarriages

Recurrent miscarriage occurs in around 1 to 7 percent of couples. The etiology involves genetic, immunologic, anatomic, hormonal, metabolic, thrombophilic and infectious factors. With the aim of establishing the frequency of low-level mosaicism in the X-chromosome, in a population of couples with prior recurrent miscarriages, a prospective case-control cytogenetic study took place on 20 couples, at the biogenetic laboratory in CECOLFES (Colombian Center of Fertility and Sterility). Clinical pathologic evaluation, anatomic, hormonal, infectious, andrologic and genetic studies were performed. As a conventional method in cytogenetic techniques, banding GTG was used for the study of structural and numeric chromosomal abnormalities whereas the molecular method of Fluorescence In Situ Hybridization (FISH) was used to confirm the mosaicism in sexual chromosomes. According to paraclinic results from the participating couples, diagnosis showed immunologic (75%), anatomic (30%), hormonal (25%), male (25%), infectious (25%), genetic (15%) and idiophatic factors (10%). Results from the cytogenetic analysis, were 10% of low-level mosaicism in the X-chromosome in two women whose final diagnosis included genetic and infectious factors for one and genetic and immunologic factors for the other. Only 10 % of the total miscarriages from the couples were evaluated. Conclusions include aspects such as multifactorial evidence of pathogenesis in recurrent miscarriage, the sub-diagnosis of genetic factors and the need to focus future investigations on cytogenetic interpretation and the clinicalpathological association between low-level mosaicism in the X-cromosome and recurrent miscarriage.

Gerunda i els ausetans: una qüestió recurrent

Article que tracta del tema recurrent de la pertinença o no de Gerunda a la tribu ibèrica dels ausetans

Equine conceptus development : a mini review

Many aspects of early embryonic development in the horse are unusual or unique; this is of scientific interest and, in some cases, considerable practical significance. During early development the number of different cell types increases rapidly and the organization of these increasingly differentiated cells becomes increasingly intricate as a result of various inter-related processes that occur step-wise or simultaneously in different parts of the conceptus (i.e., the embryo proper and its associated membranes and fluid). Equine conceptus development is of practical interest for many reasons. Most significantly, following a high rate of successful fertilization (71-96%) (Ball, 1988), as many as 30-40% of developing embryos fail to survive beyond the first two weeks of gestation (Ball, 1988), the time at which gastrulation begins. Indeed, despite considerable progress in the development of treatments for common causes of sub-fertility and of assisted reproductive techniques to enhance reproductive efficiency, the need to monitor and rebreed mares that lose a pregnancy or the failure to produce a foal, remain sources of considerable economic loss to the equine breeding industry. Of course, the potential causes of early embryonic death are numerous and varied (e.g. persistent mating induced endometritis, endometrial gland insufficiency, cervical incompetence, corpus luteum (CL) failure, chromosomal, genetic and other unknown factors (LeBlanc, 2004). However, the problem is especially acute in aged mares with a history of poor fertility in which the incidence of embryonic loss between days 2 and 14 after ovulation has been reported to reach 62-73%, and in which embryonic death is due primarily to embryonic defects rather than to uterine pathology (Ball et al., 1989; Carnevale & Ginther, 1995; Ball, 2000).

Streptococcus devriesei sp nov., from equine teeth

Phenotypic and phylogenetic studies were performed on four unidentified Gram-positive staining, catalase-negative, cc-hemolytic Streptococcus-like organisms recovered from the teeth of horses. SDS PAGE analysis of whole-cell proteins and comparative 16S rRNA gene sequencing demonstrated the four strains were highly related to each other but that they did not correspond to any recognised species of the genus Streptococcus. Phylogenetic analysis based on 16S rRNA gene sequences showed the unidentified organisms form a hitherto unknown sub-line within the Streptococcus genus, displaying a close affinity with Streptococcus mutans, Streptococcus ferus and related organisms. Sequence divergence values of > 5% with thew and other reference streptococcal species however demonstrated the organisms from equine sources represent a novel species. Based on the phenotypic distinctiveness of the new bacterium and molecular chemical and molecular genetic evidence, it is proposed that the unknown species be classified as Streptococcus devriesei sp. nov. The type strain of Streptococcus devriesei is CCUG 47155(T) (= CIP 107809T).

Inverse model control using recurrent networks

This paper illustrates how internal model control of nonlinear processes can be achieved by recurrent neural networks, e.g. fully connected Hopfield networks. It is shown that using results developed by Kambhampati et al. (1995), that once a recurrent network model of a nonlinear system has been produced, a controller can be produced which consists of the network comprising the inverse of the model and a filter. Thus, the network providing control for the nonlinear system does not require any training after it has been trained to model the nonlinear system. Stability and other issues of importance for nonlinear control systems are also discussed.

Input/output linearization using dynamic recurrent neural networks

This paper brings together two areas of research that have received considerable attention during the last years, namely feedback linearization and neural networks. A proposition that guarantees the Input/Output (I/O) linearization of nonlinear control affine systems with Dynamic Recurrent Neural Networks (DRNNs) is formulated and proved. The proposition and the linearization procedure are illustrated with the simulation of a single link manipulator.