Feedforward neural networks based on PPS-wavelet activation functions

Function approximation is a very important task in environments where computation has to be based on extracting information from data samples in real world processes. Neural networks and wavenets have been recently seen as attractive tools for developing efficient solutions for many real world problems in function approximation. In this paper, it is shown how feedforward neural networks can be built using a different type of activation function referred to as the PPS-wavelet. An algorithm is presented to generate a family of PPS-wavelets that can be used to efficiently construct feedforward networks for function approximation.

On Recurrent and Deep Neural Networks

L'apprentissage profond est un domaine de recherche en forte croissance en apprentissage automatique qui est parvenu à des résultats impressionnants dans différentes tâches allant de la classification d'images à la parole, en passant par la modélisation du langage. Les réseaux de neurones récurrents, une sous-classe d'architecture profonde, s'avèrent particulièrement prometteurs. Les réseaux récurrents peuvent capter la structure temporelle dans les données. Ils ont potentiellement la capacité d'apprendre des corrélations entre des événements éloignés dans le temps et d'emmagasiner indéfiniment des informations dans leur mémoire interne. Dans ce travail, nous tentons d'abord de comprendre pourquoi la profondeur est utile. Similairement à d'autres travaux de la littérature, nos résultats démontrent que les modèles profonds peuvent être plus efficaces pour représenter certaines familles de fonctions comparativement aux modèles peu profonds. Contrairement à ces travaux, nous effectuons notre analyse théorique sur des réseaux profonds acycliques munis de fonctions d'activation linéaires par parties, puisque ce type de modèle est actuellement l'état de l'art dans différentes tâches de classification. La deuxième partie de cette thèse porte sur le processus d'apprentissage. Nous analysons quelques techniques d'optimisation proposées récemment, telles l'optimisation Hessian free, la descente de gradient naturel et la descente des sous-espaces de Krylov. Nous proposons le cadre théorique des méthodes à région de confiance généralisées et nous montrons que plusieurs de ces algorithmes développés récemment peuvent être vus dans cette perspective. Nous argumentons que certains membres de cette famille d'approches peuvent être mieux adaptés que d'autres à l'optimisation non convexe. La dernière partie de ce document se concentre sur les réseaux de neurones récurrents. Nous étudions d'abord le concept de mémoire et tentons de répondre aux questions suivantes: Les réseaux récurrents peuvent-ils démontrer une mémoire sans limite? Ce comportement peut-il être appris? Nous montrons que cela est possible si des indices sont fournis durant l'apprentissage. Ensuite, nous explorons deux problèmes spécifiques à l'entraînement des réseaux récurrents, à savoir la dissipation et l'explosion du gradient. Notre analyse se termine par une solution au problème d'explosion du gradient qui implique de borner la norme du gradient. Nous proposons également un terme de régularisation conçu spécifiquement pour réduire le problème de dissipation du gradient. Sur un ensemble de données synthétique, nous montrons empiriquement que ces mécanismes peuvent permettre aux réseaux récurrents d'apprendre de façon autonome à mémoriser des informations pour une période de temps indéfinie. Finalement, nous explorons la notion de profondeur dans les réseaux de neurones récurrents. Comparativement aux réseaux acycliques, la définition de profondeur dans les réseaux récurrents est souvent ambiguë. Nous proposons différentes façons d'ajouter de la profondeur dans les réseaux récurrents et nous évaluons empiriquement ces propositions.

Multidimensional polynomial powers of sigmoid (PPS) Wavelet neural networks

Wavelet functions have been used as the activation function in feedforward neural networks. An abundance of R&D has been produced on wavelet neural network area. Some successful algorithms and applications in wavelet neural network have been developed and reported in the literature. However, most of the aforementioned reports impose many restrictions in the classical backpropagation algorithm, such as low dimensionality, tensor product of wavelets, parameters initialization, and, in general, the output is one dimensional, etc. In order to remove some of these restrictions, a family of polynomial wavelets generated from powers of sigmoid functions is presented. We described how a multidimensional wavelet neural networks based on these functions can be constructed, trained and applied in pattern recognition tasks. As an example of application for the method proposed, it is studied the exclusive-or (XOR) problem.

Some limit theorems for regenerative queues

We consider a single server queue with the interarrival times and the service times forming a regenerative sequence. This traffic class includes the standard models: lid, periodic, Markov modulated (e.g., BMAP model of Lucantoni [18]) and their superpositions. This class also includes the recently proposed traffic models in high speed networks, exhibiting long range dependence. Under minimal conditions we obtain the rates of convergence to stationary distributions, finiteness of stationary moments, various functional limit theorems and the continuity of stationary distributions and moments. We use the continuity results to obtain approximations for stationary distributions and moments of an MMPP/GI/1 queue where the modulating chain has a countable state space. We extend all our results to feedforward networks where the external arrivals to each queue can be regenerative. In the end we show that the output process of a leaky bucket is regenerative if the input process is and hence our results extend to a queue with arrivals controlled by a leaky bucket.

Synconset Waves and Chains: Spiking Onsets in Synchronous Populations Predict and Are Predicted by Network Structure

Synfire waves are propagating spike packets in synfire chains, which are feedforward chains embedded in random networks. Although synfire waves have proved to be effective quantification for network activity with clear relations to network structure, their utilities are largely limited to feedforward networks with low background activity. To overcome these shortcomings, we describe a novel generalisation of synfire waves, and define `synconset wave' as a cascade of first spikes within a synchronisation event. Synconset waves would occur in `synconset chains', which are feedforward chains embedded in possibly heavily recurrent networks with heavy background activity. We probed the utility of synconset waves using simulation of single compartment neuron network models with biophysically realistic conductances, and demonstrated that the spread of synconset waves directly follows from the network connectivity matrix and is modulated by top-down inputs and the resultant oscillations. Such synconset profiles lend intuitive insights into network organisation in terms of connection probabilities between various network regions rather than an adjacency matrix. To test this intuition, we develop a Bayesian likelihood function that quantifies the probability that an observed synfire wave was caused by a given network. Further, we demonstrate it's utility in the inverse problem of identifying the network that caused a given synfire wave. This method was effective even in highly subsampled networks where only a small subset of neurons were accessible, thus showing it's utility in experimental estimation of connectomes in real neuronal-networks. Together, we propose synconset chains/waves as an effective framework for understanding the impact of network structure on function, and as a step towards developing physiology-driven network identification methods. Finally, as synconset chains extend the utilities of synfire chains to arbitrary networks, we suggest utilities of our framework to several aspects of network physiology including cell assemblies, population codes, and oscillatory synchrony.

Human face verification based on multidimensional Polynomial Powers of Sigmoid (PPS)

In this paper, we described how a multidimensional wavelet neural networks based on Polynomial Powers of Sigmoid (PPS) can be constructed, trained and applied in image processing tasks. In this sense, a novel and uniform framework for face verification is presented. The framework is based on a family of PPS wavelets,generated from linear combination of the sigmoid functions, and can be considered appearance based in that features are extracted from the face image. The feature vectors are then subjected to subspace projection of PPS-wavelet. The design of PPS-wavelet neural networks is also discussed, which is seldom reported in the literature. The Stirling Universitys face database were used to generate the results. Our method has achieved 92 % of correct detection and 5 % of false detection rate on the database.

Determinação automática da porosidade e zoneamento de perfis através da rede neural artificial competitiva

Duas das mais importantes atividades da interpretação de perfis para avaliação de reservatórios de hidrocarbonetos são o zoneamento do perfil (log zonation) e o cálculo da porosidade efetiva das rochas atravessadas pelo poço. O zoneamento é a interpretação visual do perfil para identificação das camadas reservatório e, consequentemente, dos seus limites verticais, ou seja, é a separação formal do perfil em rochas reservatório e rochas selante. Todo procedimento de zoneamento é realizado de forma manual, valendo-se do conhecimento geológico-geofísico e da experiência do intérprete, na avaliação visual dos padrões (características da curva do perfil representativa de um evento geológico) correspondentes a cada tipo litológico específico. O cálculo da porosidade efetiva combina tanto uma atividade visual, na identificação dos pontos representativos de uma particular rocha reservatório no perfil, como a escolha adequada da equação petrofísica que relaciona as propriedades físicas mensuradas da rocha com sua porosidade. A partir do conhecimento da porosidade, será estabelecido o volume eventualmente ocupado por hidrocarboneto. Esta atividade, essencial para a qualificação de reservatórios, requer muito do conhecimento e da experiência do intérprete de perfil para a efetiva avaliação da porosidade efetiva, ou seja, a porosidade da rocha reservatório, isenta do efeito da argila sobre a medida das propriedades físicas da mesma. Uma forma eficiente de automatizar estes procedimentos e auxiliar o geofísico de poço nestas atividades, que particularmente demandam grande dispêndio de tempo, é apresentado nesta dissertação, na forma de um novo perfil, derivado dos perfis tradicionais de porosidade, que apresenta diretamente o zoneamento. Pode-se destacar neste novo perfil as profundidades do topo e da base das rochas reservatório e das rochas selante, escalonado na forma de porosidade efetiva, denominado perfil de porosidade efetiva zoneado. A obtenção do perfil de porosidade efetiva zoneado é baseado no projeto e execução de várias arquiteturas de rede neural artificial, do tipo direta, com treinamento não supervisionado e contendo uma camada de neurônios artificiais, do tipo competitivo. Estas arquiteturas são projetadas de modo a simular o comportamento do intérprete de perfil, quando da utilização do gráfico densidade-neutrônico, para as situações de aplicabilidade do modelo arenito-folhelho. A aplicabilidade e limitações desta metodologia são avaliadas diretamente sobre dados reais, oriundos da bacia do Lago Maracaibo (Venezuela).

The time dimension of neural network models

This review attempts to provide an insightful perspective on the role of time within neural network models and the use of neural networks for problems involving time. The most commonly used neural network models are defined and explained giving mention to important technical issues but avoiding great detail. The relationship between recurrent and feedforward networks is emphasised, along with the distinctions in their practical and theoretical abilities. Some practical examples are discussed to illustrate the major issues concerning the application of neural networks to data with various types of temporal structure, and finally some highlights of current research on the more difficult types of problems are presented.

The 'moving targets' training algorithm

A simple method for training the dynamical behavior of a neural network is derived. It is applicable to any training problem in discrete-time networks with arbitrary feedback. The method resembles back-propagation in that it is a least-squares, gradient-based optimization method, but the optimization is carried out in the hidden part of state space instead of weight space. A straightforward adaptation of this method to feedforward networks offers an alternative to training by conventional back-propagation. Computational results are presented for simple dynamical training problems, with varied success. The failures appear to arise when the method converges to a chaotic attractor. A patch-up for this problem is proposed. The patch-up involves a technique for implementing inequality constraints which may be of interest in its own right.