Activation function study for wavelet network

The main purpose of this paper is to investigate theoretically and experimentally the use of family of Polynomial Powers of the Sigmoid (PPS) Function Networks applied in speech signal representation and function approximation. This paper carries out practical investigations in terms of approximation fitness (LSE), time consuming (CPU Time), computational complexity (FLOP) and representation power (Number of Activation Function) for different PPS activation functions. We expected that different activation functions can provide performance variations and further investigations will guide us towards a class of mappings associating the best activation function to solve a class of problems under certain criteria.

Nonlinear indicial response of complex nonstationary oscillations as pulmonary hypertension responding to step hypoxia

This paper is devoted to the quantization of the degree of nonlinearity of the relationship between two biological variables when one of the variables is a complex nonstationary oscillatory signal. An example of the situation is the indicial responses of pulmonary blood pressure (P) to step changes of oxygen tension (ΔpO2) in the breathing gas. For a step change of ΔpO2 beginning at time t1, the pulmonary blood pressure is a nonlinear function of time and ΔpO2, which can be written as P(t-t1 | ΔpO2). An effective method does not exist to examine the nonlinear function P(t-t1 | ΔpO2). A systematic approach is proposed here. The definitions of mean trends and oscillations about the means are the keys. With these keys a practical method of calculation is devised. We fit the mean trends of blood pressure with analytic functions of time, whose nonlinearity with respect to the oxygen level is clarified here. The associated oscillations about the mean can be transformed into Hilbert spectrum. An integration of the square of the Hilbert spectrum over frequency yields a measure of oscillatory energy, which is also a function of time, whose mean trends can be expressed by analytic functions. The degree of nonlinearity of the oscillatory energy with respect to the oxygen level also is clarified here. Theoretical extension of the experimental nonlinear indicial functions to arbitrary history of hypoxia is proposed. Application of the results to tissue remodeling and tissue engineering of blood vessels is discussed.

Analysis of small signal stability margins using genetic optimization

Power system small signal stability analysis aims to explore different small signal stability conditions and controls, namely: (1) exploring the power system security domains and boundaries in the space of power system parameters of interest, including load flow feasibility, saddle node and Hopf bifurcation ones; (2) finding the maximum and minimum damping conditions; and (3) determining control actions to provide and increase small signal stability. These problems are presented in this paper as different modifications of a general optimization to a minimum/maximum, depending on the initial guesses of variables and numerical methods used. In the considered problems, all the extreme points are of interest. Additionally, there are difficulties with finding the derivatives of the objective functions with respect to parameters. Numerical computations of derivatives in traditional optimization procedures are time consuming. In this paper, we propose a new black-box genetic optimization technique for comprehensive small signal stability analysis, which can effectively cope with highly nonlinear objective functions with multiple minima and maxima, and derivatives that can not be expressed analytically. The optimization result can then be used to provide such important information such as system optimal control decision making, assessment of the maximum network's transmission capacity, etc. (C) 1998 Elsevier Science S.A. All rights reserved.

The role of fixed delays in neuronal rate models

Fixed delays in neuronal interactions arise through synaptic and dendritic processing. Previous work has shown that such delays, which play an important role in shaping the dynamics of networks of large numbers of spiking neurons with continuous synaptic kinetics, can be taken into account with a rate model through the addition of an explicit, fixed delay. Here we extend this work to account for arbitrary symmetric patterns of synaptic connectivity and generic nonlinear transfer functions. Specifically, we conduct a weakly nonlinear analysis of the dynamical states arising via primary instabilities of the stationary uniform state. In this way we determine analytically how the nature and stability of these states depend on the choice of transfer function and connectivity. While this dependence is, in general, nontrivial, we make use of the smallness of the ratio in the delay in neuronal interactions to the effective time constant of integration to arrive at two general observations of physiological relevance. These are: 1 - fast oscillations are always supercritical for realistic transfer functions. 2 - Traveling waves are preferred over standing waves given plausible patterns of local connectivity.

Integration of Notch 1 and calcineurin/NFAT signaling pathways in keratinocyte growth and differentiation control.

The Notch and Calcineurin/NFAT pathways have both been implicated in control of keratinocyte differentiation. Induction of the p21(WAF1/Cip1) gene by Notch 1 activation in differentiating keratinocytes is associated with direct targeting of the RBP-Jkappa protein to the p21 promoter. We show here that Notch 1 activation functions also through a second Calcineurin-dependent mechanism acting on the p21 TATA box-proximal region. Increased Calcineurin/NFAT activity by Notch signaling involves downregulation of Calcipressin, an endogenous Calcineurin inhibitor, through a HES-1-dependent mechanism. Besides control of the p21 gene, Calcineurin contributes significantly to the transcriptional response of keratinocytes to Notch 1 activation, both in vitro and in vivo. In fact, deletion of the Calcineurin B1 gene in the skin results in a cyclic alopecia phenotype, associated with altered expression of Notch-responsive genes involved in hair follicle structure and/or adhesion to the surrounding mesenchyme. Thus, an important interconnection exists between Notch 1 and Calcineurin-NFAT pathways in keratinocyte growth/differentiation control.

A Thermodynamically Consistent Damage Model for Advanced Composites

A continuum damage model for the prediction of damage onset and structural collapse of structures manufactured in fiber-reinforced plastic laminates is proposed. The principal damage mechanisms occurring in the longitudinal and transverse directions of a ply are represented by a damage tensor that is fixed in space. Crack closure under load reversal effects are taken into account using damage variables established as a function of the sign of the components of the stress tensor. Damage activation functions based on the LaRC04 failure criteria are used to predict the different damage mechanisms occurring at the ply level. The constitutive damage model is implemented in a finite element code. The objectivity of the numerical model is assured by regularizing the dissipated energy at a material point using Bazant’s Crack Band Model. To verify the accuracy of the approach, analyses ofcoupon specimens were performed, and the numerical predictions were compared with experimental data

On Recurrent and Deep Neural Networks

L'apprentissage profond est un domaine de recherche en forte croissance en apprentissage automatique qui est parvenu à des résultats impressionnants dans différentes tâches allant de la classification d'images à la parole, en passant par la modélisation du langage. Les réseaux de neurones récurrents, une sous-classe d'architecture profonde, s'avèrent particulièrement prometteurs. Les réseaux récurrents peuvent capter la structure temporelle dans les données. Ils ont potentiellement la capacité d'apprendre des corrélations entre des événements éloignés dans le temps et d'emmagasiner indéfiniment des informations dans leur mémoire interne. Dans ce travail, nous tentons d'abord de comprendre pourquoi la profondeur est utile. Similairement à d'autres travaux de la littérature, nos résultats démontrent que les modèles profonds peuvent être plus efficaces pour représenter certaines familles de fonctions comparativement aux modèles peu profonds. Contrairement à ces travaux, nous effectuons notre analyse théorique sur des réseaux profonds acycliques munis de fonctions d'activation linéaires par parties, puisque ce type de modèle est actuellement l'état de l'art dans différentes tâches de classification. La deuxième partie de cette thèse porte sur le processus d'apprentissage. Nous analysons quelques techniques d'optimisation proposées récemment, telles l'optimisation Hessian free, la descente de gradient naturel et la descente des sous-espaces de Krylov. Nous proposons le cadre théorique des méthodes à région de confiance généralisées et nous montrons que plusieurs de ces algorithmes développés récemment peuvent être vus dans cette perspective. Nous argumentons que certains membres de cette famille d'approches peuvent être mieux adaptés que d'autres à l'optimisation non convexe. La dernière partie de ce document se concentre sur les réseaux de neurones récurrents. Nous étudions d'abord le concept de mémoire et tentons de répondre aux questions suivantes: Les réseaux récurrents peuvent-ils démontrer une mémoire sans limite? Ce comportement peut-il être appris? Nous montrons que cela est possible si des indices sont fournis durant l'apprentissage. Ensuite, nous explorons deux problèmes spécifiques à l'entraînement des réseaux récurrents, à savoir la dissipation et l'explosion du gradient. Notre analyse se termine par une solution au problème d'explosion du gradient qui implique de borner la norme du gradient. Nous proposons également un terme de régularisation conçu spécifiquement pour réduire le problème de dissipation du gradient. Sur un ensemble de données synthétique, nous montrons empiriquement que ces mécanismes peuvent permettre aux réseaux récurrents d'apprendre de façon autonome à mémoriser des informations pour une période de temps indéfinie. Finalement, nous explorons la notion de profondeur dans les réseaux de neurones récurrents. Comparativement aux réseaux acycliques, la définition de profondeur dans les réseaux récurrents est souvent ambiguë. Nous proposons différentes façons d'ajouter de la profondeur dans les réseaux récurrents et nous évaluons empiriquement ces propositions.

Apprentissage des réseaux de neurones profonds et applications en traitement automatique de la langue naturelle

En apprentissage automatique, domaine qui consiste à utiliser des données pour apprendre une solution aux problèmes que nous voulons confier à la machine, le modèle des Réseaux de Neurones Artificiels (ANN) est un outil précieux. Il a été inventé voilà maintenant près de soixante ans, et pourtant, il est encore de nos jours le sujet d'une recherche active. Récemment, avec l'apprentissage profond, il a en effet permis d'améliorer l'état de l'art dans de nombreux champs d'applications comme la vision par ordinateur, le traitement de la parole et le traitement des langues naturelles. La quantité toujours grandissante de données disponibles et les améliorations du matériel informatique ont permis de faciliter l'apprentissage de modèles à haute capacité comme les ANNs profonds. Cependant, des difficultés inhérentes à l'entraînement de tels modèles, comme les minima locaux, ont encore un impact important. L'apprentissage profond vise donc à trouver des solutions, en régularisant ou en facilitant l'optimisation. Le pré-entraînnement non-supervisé, ou la technique du ``Dropout'', en sont des exemples. Les deux premiers travaux présentés dans cette thèse suivent cette ligne de recherche. Le premier étudie les problèmes de gradients diminuants/explosants dans les architectures profondes. Il montre que des choix simples, comme la fonction d'activation ou l'initialisation des poids du réseaux, ont une grande influence. Nous proposons l'initialisation normalisée pour faciliter l'apprentissage. Le second se focalise sur le choix de la fonction d'activation et présente le rectifieur, ou unité rectificatrice linéaire. Cette étude a été la première à mettre l'accent sur les fonctions d'activations linéaires par morceaux pour les réseaux de neurones profonds en apprentissage supervisé. Aujourd'hui, ce type de fonction d'activation est une composante essentielle des réseaux de neurones profonds. Les deux derniers travaux présentés se concentrent sur les applications des ANNs en traitement des langues naturelles. Le premier aborde le sujet de l'adaptation de domaine pour l'analyse de sentiment, en utilisant des Auto-Encodeurs Débruitants. Celui-ci est encore l'état de l'art de nos jours. Le second traite de l'apprentissage de données multi-relationnelles avec un modèle à base d'énergie, pouvant être utilisé pour la tâche de désambiguation de sens.

A comparative study of multilayered and single layered neural network based predictive controllers

The authors compare the performance of two types of controllers one based on the multilayered network and the other based on the single layered CMAC network (cerebellar model articulator controller). The neurons (information processing units) in the multi-layered network use Gaussian activation functions. The control scheme which is considered is a predictive control algorithm, along the lines used by Willis et al. (1991), Kambhampati and Warwick (1991). The process selected as a test bed is a continuous stirred tank reactor. The reaction taking place is an irreversible exothermic reaction in a constant volume reactor cooled by a single coolant stream. This reactor is a simplified version of the first tank in the two tank system given by Henson and Seborg (1989).

The Thyroid Hormone Receptor (TR) beta-Selective Agonist GC-1 Inhibits Proliferation But Induces Differentiation and TR beta mRNA Expression in Mouse and Rat Osteoblast-Like Cells

Previous studies showed anabolic effects of GC-1, a triiodothyronine (T3) analogue that is selective for both binding and activation functions of thyroid hormone receptor (TR) beta 1 over TR alpha 1, on bone tissue in vivo. The aim of this study was to investigate the responsiveness of rat (ROS17/2.8) and mouse (MC3T3-E1) osteoblast-like cells to GC-1. As expected, T3 inhibited cellular proliferation and stimulated mRNA expression of osteocalcin or alkaline phosphatase in both cell lineages. Whereas equimolar doses of T3 and GC-1 equally affected these parameters in ROS17/2.8 cells, the effects of GC-1 were more modest compared to those of T3 in MC3T3-E1 cells. Interestingly, we showed that there is higher expression of TR alpha 1 than TR beta 1 mRNA in rat (similar to 20-90%) and mouse (similar to 90-98%) cell lineages and that this difference is even higher in mouse cells, which highlights the importance of TR alpha 1 to bone physiology and may partially explain the modest effects of GC-1 in comparison with T3 in MC3T3-E1 cells. Nevertheless, we showed that TR beta 1 mRNA expression increases (similar to 2.8- to 4.3-fold) as osteoblastic cells undergo maturation, suggesting a key role of TR beta 1 in mediating T3 effects in the bone forming cells, especially in mature osteoblasts. It is noteworthy that T3 and GC-1 induced TR beta 1 mRNA expression to a similar extent in both cell lineages (similar to 2- to 4-fold), indicating that both ligands may modulate the responsiveness of osteoblasts to T3. Taken together, these data show that TR beta selective T3 analogues have the potential to directly induce the differentiation and activity of osteoblasts.

Multidimensional polynomial powers of sigmoid (PPS) Wavelet neural networks

Wavelet functions have been used as the activation function in feedforward neural networks. An abundance of R&D has been produced on wavelet neural network area. Some successful algorithms and applications in wavelet neural network have been developed and reported in the literature. However, most of the aforementioned reports impose many restrictions in the classical backpropagation algorithm, such as low dimensionality, tensor product of wavelets, parameters initialization, and, in general, the output is one dimensional, etc. In order to remove some of these restrictions, a family of polynomial wavelets generated from powers of sigmoid functions is presented. We described how a multidimensional wavelet neural networks based on these functions can be constructed, trained and applied in pattern recognition tasks. As an example of application for the method proposed, it is studied the exclusive-or (XOR) problem.

Human face verification based on multidimensional Polynomial Powers of Sigmoid (PPS)

In this paper, we described how a multidimensional wavelet neural networks based on Polynomial Powers of Sigmoid (PPS) can be constructed, trained and applied in image processing tasks. In this sense, a novel and uniform framework for face verification is presented. The framework is based on a family of PPS wavelets,generated from linear combination of the sigmoid functions, and can be considered appearance based in that features are extracted from the face image. The feature vectors are then subjected to subspace projection of PPS-wavelet. The design of PPS-wavelet neural networks is also discussed, which is seldom reported in the literature. The Stirling Universitys face database were used to generate the results. Our method has achieved 92 % of correct detection and 5 % of false detection rate on the database.

Growth curves for ostriches (Struthio camelus) in a Brazilian population

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Identificação e controle de sistemas dinâmicos utilizando redes wavelets

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Myc-mediated control of gene transcription in cancer cells

The Myc oncoproteins belong to a family of transcription factors composed by Myc, N-Myc and L-Myc. The most studied components of this family are Myc and N-Myc because their expressions are frequently deregulated in a wide range of cancers. These oncoproteins can act both as activators or repressors of gene transcription. As activators, they heterodimerize with Max (Myc associated X-factor) and the heterodimer recognizes and binds a specific sequence elements (E-Box) onto gene promoters recruiting histone acetylase and inducing transcriptional activation. Myc-mediated transcriptional repression is a quite debated issue. One of the first mechanisms defined for the Myc-mediated transcriptional repression consisted in the interaction of Myc-Max complex Sp1 and/or Miz1 transcription factors already bound to gene promoters. This interaction may interfere with their activation functions by recruiting co-repressors such as Dnmt3 or HDACs. Moreover, in the absence of , Myc may interfere with the Sp1 activation function by direct interaction and subsequent recruitment of HDACs. More recently the Myc/Max complex was also shown to mediate transcriptional repression by direct binding to peculiar E-box. In this study we analyzed the role of Myc overexpression in Osteosarcoma and Neuroblastoma oncogenesis and the mechanisms underling to Myc function. Myc overexpression is known to correlate with chemoresistance in Osteosarcoma cells. We extended this study by demonstrating that c-Myc induces transcription of a panel of ABC drug transporter genes. ABCs are a large family trans-membrane transporter deeply involved in multi drug resistance. Furthermore expression levels of Myc, ABCC1, ABCC4 and ABCF1 were proved to be important prognostic tool to predict conventional therapy failure. N-Myc amplification/overexpression is the most important prognostic factor for Neuroblastoma. Cyclin G2 and Clusterin are two genes often down regulated in neuroblastoma cells. Cyclin G2 is an atypical member of Cyclin family and its expression is associated with terminal differentiation and apoptosis. Moreover it blocks cell cycle progression and induces cell growth arrest. Instead, CLU is a multifunctional protein involved in many physiological and pathological processes. Several lines of evidences support the view that CLU may act as a tumour suppressor in Neuroblastoma. In this thesis I showed that N-Myc represses CCNG2 and CLU transcription by different mechanisms. • N-Myc represses CCNG2 transcription by directly interacting with Sp1 bound in CCNG2 promoter and recruiting HDAC2. Importantly, reactivation of CCNG2 expression through epigenetic drugs partially reduces N-Myc and HDAC2 mediated cell proliferation. • N-Myc/Max complex represses CLU expression by direct binding to a peculiar E-box element on CLU promoter and by recruitment of HDACs and Polycomb Complexes, to the CLU promoter. Overall our findings strongly support the model in which Myc overexpression/amplification may contribute to some aspects of oncogenesis by a dual action: i) transcription activation of genes that confer a multidrug resistant phenotype to cancer cells; ii), transcription repression of genes involved in cell cycle inhibition and cellular differentiation.