Feedforward neural networks based on PPS-wavelet activation functions

Function approximation is a very important task in environments where computation has to be based on extracting information from data samples in real world processes. Neural networks and wavenets have been recently seen as attractive tools for developing efficient solutions for many real world problems in function approximation. In this paper, it is shown how feedforward neural networks can be built using a different type of activation function referred to as the PPS-wavelet. An algorithm is presented to generate a family of PPS-wavelets that can be used to efficiently construct feedforward networks for function approximation.

RB and hbrm cooperate to repress the activation functions of E2F1

Forced expression of the retinoblastoma (RB) gene product inhibits the proliferation of cells in culture. A major target of the RB protein is the S-phase-inducing transcription factor E2F1. RB binds directly to the activation domain of E2F1 and silences it, thereby preventing cells from entering S phase. To induce complete G1 arrest, RB requires the presence of the hbrm/BRG-1 proteins, which are components of the coactivator SWI/SNF complex. This cooperation is mediated through a physical interaction between RB and hbrm/BRG-1. We show here that in transfected cells RB can contact both E2F1 and hbrm at the same time, thereby targeting hbrm to E2F1. E2F1 and hbrm are indeed found within the same complex in vivo. Furthermore, RB and hbrm cooperate to repress E2F1 activity in transient transfection assays. The ability of hbrm to cooperate with RB to repress E2F1 is dependent upon several distinct domains of hbrm, including the RB binding domain and the NTP binding site. However, the bromodomain seems dispensable for this activity. Taken together, our results point out an unexpected role of corepressor for the hbrm protein. The ability of hbrm and RB to cooperate in repressing E2F1 activity could be an underlying mechanism for the observed cooperation between hbrm and RB to induce G1 arrest. Finally, we demonstrate that the domain of hbrm that binds RB has transcriptional activation potential which RB can repress. This suggest that RB not only targets hbrm but also regulates its activity.

Growing Neural Networks Using Nonconventional Activation Functions

In the paper, an ontogenic artificial neural network (ANNs) is proposed. The network uses orthogonal activation functions that allow significant reducing of computational complexity. Another advantage is numerical stability, because the system of activation functions is linearly independent by definition. A learning procedure for proposed ANN with guaranteed convergence to the global minimum of error function in the parameter space is developed. An algorithm for structure network structure adaptation is proposed. The algorithm allows adding or deleting a node in real-time without retraining of the network. Simulation results confirm the efficiency of the proposed approach.

Double-Wavelet Neuron Based on Analytical Activation Functions

In this paper a new double-wavelet neuron architecture obtained by modification of standard wavelet neuron, and its learning algorithm are proposed. The offered architecture allows to improve the approximation properties of wavelet neuron. Double-wavelet neuron and its learning algorithm are examined for forecasting non-stationary chaotic time series.

Fourier Neural Networks: An Approach with Sinusoidal Activation Functions

* Supported by INTAS 2000-626, INTAS YSF 03-55-1969, INTAS INNO 182, and TIC 2003-09319-c03-03.

Variational Quantum Splines: Moving Beyond Linearity for Quantum Activation Functions

Activation functions within neural networks play a crucial role in Deep Learning since they allow to learn complex and non-trivial patterns in the data. However, the ability to approximate non-linear functions is a significant limitation when implementing neural networks in a quantum computer to solve typical machine learning tasks. The main burden lies in the unitarity constraint of quantum operators, which forbids non-linearity and poses a considerable obstacle to developing such non-linear functions in a quantum setting. Nevertheless, several attempts have been made to tackle the realization of the quantum activation function in the literature. Recently, the idea of the QSplines has been proposed to approximate a non-linear activation function by implementing the quantum version of the spline functions. Yet, QSplines suffers from various drawbacks. Firstly, the final function estimation requires a post-processing step; thus, the value of the activation function is not available directly as a quantum state. Secondly, QSplines need many error-corrected qubits and a very long quantum circuits to be executed. These constraints do not allow the adoption of the QSplines on near-term quantum devices and limit their generalization capabilities. This thesis aims to overcome these limitations by leveraging hybrid quantum-classical computation. In particular, a few different methods for Variational Quantum Splines are proposed and implemented, to pave the way for the development of complete quantum activation functions and unlock the full potential of quantum neural networks in the field of quantum machine learning.

Synergistic transcriptional activation by CTF/NF-I and the estrogen receptor involves stabilized interactions with a limiting target factor.

Transcription initiation at eukaryotic protein-coding gene promoters is regulated by a complex interplay of site-specific DNA-binding proteins acting synergistically or antagonistically. Here, we have analyzed the mechanisms of synergistic transcriptional activation between members of the CCAAT-binding transcription factor/nuclear factor I (CTF/NF-I) family and the estrogen receptor. By using cotransfection experiments with HeLa cells, we show that the proline-rich transcriptional activation domain of CTF-1, when fused to the GAL4 DNA-binding domain, synergizes with each of the two estrogen receptor-activating regions. Cooperative DNA binding between the GAL4-CTF-1 fusion and the estrogen receptor does not occur in vitro, and in vivo competition experiments demonstrate that both activators can be specifically inhibited by the overexpression of a proline-rich competitor, indicating that a common limiting factor is mediating their transcriptional activation functions. Furthermore, the two activators functioning synergistically are much more resistant to competition than either factor alone, suggesting that synergism between CTF-1 and the estrogen receptor is the result of a stronger tethering of the limiting target factor(s) to the two promoter-bound activators.

Priors Stabilizers and Basis Functions: From Regularization to Radial, Tensor and Additive Splines

We had previously shown that regularization principles lead to approximation schemes, as Radial Basis Functions, which are equivalent to networks with one layer of hidden units, called Regularization Networks. In this paper we show that regularization networks encompass a much broader range of approximation schemes, including many of the popular general additive models, Breiman's hinge functions and some forms of Projection Pursuit Regression. In the probabilistic interpretation of regularization, the different classes of basis functions correspond to different classes of prior probabilities on the approximating function spaces, and therefore to different types of smoothness assumptions. In the final part of the paper, we also show a relation between activation functions of the Gaussian and sigmoidal type.

Activation function study for wavelet network

The main purpose of this paper is to investigate theoretically and experimentally the use of family of Polynomial Powers of the Sigmoid (PPS) Function Networks applied in speech signal representation and function approximation. This paper carries out practical investigations in terms of approximation fitness (LSE), time consuming (CPU Time), computational complexity (FLOP) and representation power (Number of Activation Function) for different PPS activation functions. We expected that different activation functions can provide performance variations and further investigations will guide us towards a class of mappings associating the best activation function to solve a class of problems under certain criteria.

Approximation of hyperbolic tangent activation function using hybrid methods

Artificial Neural Networks are widely used in various applications in engineering, as such solutions of nonlinear problems. The implementation of this technique in reconfigurable devices is a great challenge to researchers by several factors, such as floating point precision, nonlinear activation function, performance and area used in FPGA. The contribution of this work is the approximation of a nonlinear function used in ANN, the popular hyperbolic tangent activation function. The system architecture is composed of several scenarios that provide a tradeoff of performance, precision and area used in FPGA. The results are compared in different scenarios and with current literature on error analysis, area and system performance. © 2013 IEEE.

Orthotropic active strain models for the numerical simulation of cardiac biomechanics

An active strain formulation for orthotropic constitutive laws arising in cardiac mechanics modeling is introduced and studied. The passive mechanical properties of the tissue are described by the Holzapfel-Ogden relation. In the active strain formulation, the Euler-Lagrange equations for minimizing the total energy are written in terms of active and passive deformation factors, where the active part is assumed to depend, at the cell level, on the electrodynamics and on the specific orientation of the cardiac cells. The well-posedness of the linear system derived from a generic Newton iteration of the original problem is analyzed and different mechanical activation functions are considered. In addition, the active strain formulation is compared with the classical active stress formulation from both numerical and modeling perspectives. Taylor-Hood and MINI finite elements are employed to discretize the mechanical problem. The results of several numerical experiments show that the proposed formulation is mathematically consistent and is able to represent the main key features of the phenomenon, while allowing savings in computational costs.

Integration of Notch 1 and calcineurin/NFAT signaling pathways in keratinocyte growth and differentiation control.

The Notch and Calcineurin/NFAT pathways have both been implicated in control of keratinocyte differentiation. Induction of the p21(WAF1/Cip1) gene by Notch 1 activation in differentiating keratinocytes is associated with direct targeting of the RBP-Jkappa protein to the p21 promoter. We show here that Notch 1 activation functions also through a second Calcineurin-dependent mechanism acting on the p21 TATA box-proximal region. Increased Calcineurin/NFAT activity by Notch signaling involves downregulation of Calcipressin, an endogenous Calcineurin inhibitor, through a HES-1-dependent mechanism. Besides control of the p21 gene, Calcineurin contributes significantly to the transcriptional response of keratinocytes to Notch 1 activation, both in vitro and in vivo. In fact, deletion of the Calcineurin B1 gene in the skin results in a cyclic alopecia phenotype, associated with altered expression of Notch-responsive genes involved in hair follicle structure and/or adhesion to the surrounding mesenchyme. Thus, an important interconnection exists between Notch 1 and Calcineurin-NFAT pathways in keratinocyte growth/differentiation control.

A Thermodynamically Consistent Damage Model for Advanced Composites

A continuum damage model for the prediction of damage onset and structural collapse of structures manufactured in fiber-reinforced plastic laminates is proposed. The principal damage mechanisms occurring in the longitudinal and transverse directions of a ply are represented by a damage tensor that is fixed in space. Crack closure under load reversal effects are taken into account using damage variables established as a function of the sign of the components of the stress tensor. Damage activation functions based on the LaRC04 failure criteria are used to predict the different damage mechanisms occurring at the ply level. The constitutive damage model is implemented in a finite element code. The objectivity of the numerical model is assured by regularizing the dissipated energy at a material point using Bazant’s Crack Band Model. To verify the accuracy of the approach, analyses ofcoupon specimens were performed, and the numerical predictions were compared with experimental data

On Recurrent and Deep Neural Networks

L'apprentissage profond est un domaine de recherche en forte croissance en apprentissage automatique qui est parvenu à des résultats impressionnants dans différentes tâches allant de la classification d'images à la parole, en passant par la modélisation du langage. Les réseaux de neurones récurrents, une sous-classe d'architecture profonde, s'avèrent particulièrement prometteurs. Les réseaux récurrents peuvent capter la structure temporelle dans les données. Ils ont potentiellement la capacité d'apprendre des corrélations entre des événements éloignés dans le temps et d'emmagasiner indéfiniment des informations dans leur mémoire interne. Dans ce travail, nous tentons d'abord de comprendre pourquoi la profondeur est utile. Similairement à d'autres travaux de la littérature, nos résultats démontrent que les modèles profonds peuvent être plus efficaces pour représenter certaines familles de fonctions comparativement aux modèles peu profonds. Contrairement à ces travaux, nous effectuons notre analyse théorique sur des réseaux profonds acycliques munis de fonctions d'activation linéaires par parties, puisque ce type de modèle est actuellement l'état de l'art dans différentes tâches de classification. La deuxième partie de cette thèse porte sur le processus d'apprentissage. Nous analysons quelques techniques d'optimisation proposées récemment, telles l'optimisation Hessian free, la descente de gradient naturel et la descente des sous-espaces de Krylov. Nous proposons le cadre théorique des méthodes à région de confiance généralisées et nous montrons que plusieurs de ces algorithmes développés récemment peuvent être vus dans cette perspective. Nous argumentons que certains membres de cette famille d'approches peuvent être mieux adaptés que d'autres à l'optimisation non convexe. La dernière partie de ce document se concentre sur les réseaux de neurones récurrents. Nous étudions d'abord le concept de mémoire et tentons de répondre aux questions suivantes: Les réseaux récurrents peuvent-ils démontrer une mémoire sans limite? Ce comportement peut-il être appris? Nous montrons que cela est possible si des indices sont fournis durant l'apprentissage. Ensuite, nous explorons deux problèmes spécifiques à l'entraînement des réseaux récurrents, à savoir la dissipation et l'explosion du gradient. Notre analyse se termine par une solution au problème d'explosion du gradient qui implique de borner la norme du gradient. Nous proposons également un terme de régularisation conçu spécifiquement pour réduire le problème de dissipation du gradient. Sur un ensemble de données synthétique, nous montrons empiriquement que ces mécanismes peuvent permettre aux réseaux récurrents d'apprendre de façon autonome à mémoriser des informations pour une période de temps indéfinie. Finalement, nous explorons la notion de profondeur dans les réseaux de neurones récurrents. Comparativement aux réseaux acycliques, la définition de profondeur dans les réseaux récurrents est souvent ambiguë. Nous proposons différentes façons d'ajouter de la profondeur dans les réseaux récurrents et nous évaluons empiriquement ces propositions.