Experimental design and model construction algorithms for radial basis function networks

New construction algorithms for radial basis function (RBF) network modelling are introduced based on the A-optimality and D-optimality experimental design criteria respectively. We utilize new cost functions, based on experimental design criteria, for model selection that simultaneously optimizes model approximation, parameter variance (A-optimality) or model robustness (D-optimality). The proposed approaches are based on the forward orthogonal least-squares (OLS) algorithm, such that the new A-optimality- and D-optimality-based cost functions are constructed on the basis of an orthogonalization process that gains computational advantages and hence maintains the inherent computational efficiency associated with the conventional forward OLS approach. The proposed approach enhances the very popular forward OLS-algorithm-based RBF model construction method since the resultant RBF models are constructed in a manner that the system dynamics approximation capability, model adequacy and robustness are optimized simultaneously. The numerical examples provided show significant improvement based on the D-optimality design criterion, demonstrating that there is significant room for improvement in modelling via the popular RBF neural network.

Dual-orthogonal radial basis function networks for nonlinear time series prediction

A new structure of Radial Basis Function (RBF) neural network called the Dual-orthogonal RBF Network (DRBF) is introduced for nonlinear time series prediction. The hidden nodes of a conventional RBF network compare the Euclidean distance between the network input vector and the centres, and the node responses are radially symmetrical. But in time series prediction where the system input vectors are lagged system outputs, which are usually highly correlated, the Euclidean distance measure may not be appropriate. The DRBF network modifies the distance metric by introducing a classification function which is based on the estimation data set. Training the DRBF networks consists of two stages. Learning the classification related basis functions and the important input nodes, followed by selecting the regressors and learning the weights of the hidden nodes. In both cases, a forward Orthogonal Least Squares (OLS) selection procedure is applied, initially to select the important input nodes and then to select the important centres. Simulation results of single-step and multi-step ahead predictions over a test data set are included to demonstrate the effectiveness of the new approach.

Desacoplamento de um gerador síncrono através de um controle adaptativo por modelo de referência baseado em funções de Base radial

An alternative nonlinear technique for decoupling and control is presented. This technique is based on a RBF (Radial Basis Functions) neural network and it is applied to the synchronous generator model. The synchronous generator is a coupled system, in other words, a change at one input variable of the system, changes more than one output. The RBF network will perform the decoupling, separating the control of the following outputs variables: the load angle and flux linkage in the field winding. This technique does not require knowledge of the system parameters and, due the nature of radial basis functions, it shows itself stable to parametric uncertainties, disturbances and simpler when it is applied in control. The RBF decoupler is designed in this work for decouple a nonlinear MIMO system with two inputs and two outputs. The weights between hidden and output layer are modified online, using an adaptive law in real time. The adaptive law is developed by Lyapunov s Method. A decoupling adaptive controller uses the errors between system outputs and model outputs, and filtered outputs of the system to produce control signals. The RBF network forces each outputs of generator to behave like reference model. When the RBF approaches adequately control signals, the system decoupling is achieved. A mathematical proof and analysis are showed. Simulations are presented to show the performance and robustness of the RBF network

Regularised mixture density networks for modelling wind direction

This technical report contains all technical information and results from experiments where Mixture Density Networks (MDN) using an RBF network and fixed kernel means and variances were used to infer the wind direction from satellite data from the ersII weather satellite. The regularisation is based on the evidence framework and three different approximations were used to estimate the regularisation parameter. The results were compared with the results by `early stopping'.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Regularisation of mixture density networks

Mixture Density Networks are a principled method to model conditional probability density functions which are non-Gaussian. This is achieved by modelling the conditional distribution for each pattern with a Gaussian Mixture Model for which the parameters are generated by a neural network. This thesis presents a novel method to introduce regularisation in this context for the special case where the mean and variance of the spherical Gaussian Kernels in the mixtures are fixed to predetermined values. Guidelines for how these parameters can be initialised are given, and it is shown how to apply the evidence framework to mixture density networks to achieve regularisation. This also provides an objective stopping criteria that can replace the `early stopping' methods that have previously been used. If the neural network used is an RBF network with fixed centres this opens up new opportunities for improved initialisation of the network weights, which are exploited to start training relatively close to the optimum. The new method is demonstrated on two data sets. The first is a simple synthetic data set while the second is a real life data set, namely satellite scatterometer data used to infer the wind speed and wind direction near the ocean surface. For both data sets the regularisation method performs well in comparison with earlier published results. Ideas on how the constraint on the kernels may be relaxed to allow fully adaptable kernels are presented.

Practical methods of tracking of nonstationary time series applied to real-world data

In this paper, we discuss some practical implications for implementing adaptable network algorithms applied to non-stationary time series problems. Two real world data sets, containing electricity load demands and foreign exchange market prices, are used to test several different methods, ranging from linear models with fixed parameters, to non-linear models which adapt both parameters and model order on-line. Training with the extended Kalman filter, we demonstrate that the dynamic model-order increment procedure of the resource allocating RBF network (RAN) is highly sensitive to the parameters of the novelty criterion. We investigate the use of system noise for increasing the plasticity of the Kalman filter training algorithm, and discuss the consequences for on-line model order selection. The results of our experiments show that there are advantages to be gained in tracking real world non-stationary data through the use of more complex adaptive models.

Probabilistic topographic information visualisation

The focus of this thesis is the extension of topographic visualisation mappings to allow for the incorporation of uncertainty. Few visualisation algorithms in the literature are capable of mapping uncertain data with fewer able to represent observation uncertainties in visualisations. As such, modifications are made to NeuroScale, Locally Linear Embedding, Isomap and Laplacian Eigenmaps to incorporate uncertainty in the observation and visualisation spaces. The proposed mappings are then called Normally-distributed NeuroScale (N-NS), T-distributed NeuroScale (T-NS), Probabilistic LLE (PLLE), Probabilistic Isomap (PIso) and Probabilistic Weighted Neighbourhood Mapping (PWNM). These algorithms generate a probabilistic visualisation space with each latent visualised point transformed to a multivariate Gaussian or T-distribution, using a feed-forward RBF network. Two types of uncertainty are then characterised dependent on the data and mapping procedure. Data dependent uncertainty is the inherent observation uncertainty. Whereas, mapping uncertainty is defined by the Fisher Information of a visualised distribution. This indicates how well the data has been interpolated, offering a level of ‘surprise’ for each observation. These new probabilistic mappings are tested on three datasets of vectorial observations and three datasets of real world time series observations for anomaly detection. In order to visualise the time series data, a method for analysing observed signals and noise distributions, Residual Modelling, is introduced. The performance of the new algorithms on the tested datasets is compared qualitatively with the latent space generated by the Gaussian Process Latent Variable Model (GPLVM). A quantitative comparison using existing evaluation measures from the literature allows performance of each mapping function to be compared. Finally, the mapping uncertainty measure is combined with NeuroScale to build a deep learning classifier, the Cascading RBF. This new structure is tested on the MNist dataset achieving world record performance whilst avoiding the flaws seen in other Deep Learning Machines.

A new RBF neural network with boundary value constraints

We present a novel topology of the radial basis function (RBF) neural network, referred to as the boundary value constraints (BVC)-RBF, which is able to automatically satisfy a set of BVC. Unlike most existing neural networks whereby the model is identified via learning from observational data only, the proposed BVC-RBF offers a generic framework by taking into account both the deterministic prior knowledge and the stochastic data in an intelligent manner. Like a conventional RBF, the proposed BVC-RBF has a linear-in-the-parameter structure, such that it is advantageous that many of the existing algorithms for linear-in-the-parameters models are directly applicable. The BVC satisfaction properties of the proposed BVC-RBF are discussed. Finally, numerical examples based on the combined D-optimality-based orthogonal least squares algorithm are utilized to illustrate the performance of the proposed BVC-RBF for completeness.

Givens rotation based fast backward elimination algorithm for RBF neural network pruning

A fast backward elimination algorithm is introduced based on a QR decomposition and Givens transformations to prune radial-basis-function networks. Nodes are sequentially removed using an increment of error variance criterion. The procedure is terminated by using a prediction risk criterion so as to obtain a model structure with good generalisation properties. The algorithm can be used to postprocess radial basis centres selected using a k-means routine and, in this mode, it provides a hybrid supervised centre selection approach.

Stable Nonlinear Receding Horizon Regulator Using RBF Neural Network Models

The general stability theory of nonlinear receding horizon controllers has attracted much attention over the last fifteen years, and many algorithms have been proposed to ensure closed-loop stability. On the other hand many reports exist regarding the use of artificial neural network models in nonlinear receding horizon control. However, little attention has been given to the stability issue of these specific controllers. This paper addresses this problem and proposes to cast the nonlinear receding horizon control based on neural network models within the framework of an existing stabilising algorithm.

Symmetry insights for design of supercomputer network topologies: roots and weights lattices

We present a family of networks whose local interconnection topologies are generated by the root vectors of a semi-simple complex Lie algebra. Cartan classification theorem of those algebras ensures those families of interconnection topologies to be exhaustive. The global arrangement of the network is defined in terms of integer or half-integer weight lattices. The mesh or torus topologies that network millions of processing cores, such as those in the IBM BlueGene series, are the simplest member of that category. The symmetries of the root systems of an algebra, manifested by their Weyl group, lends great convenience for the design and analysis of hardware architecture, algorithms and programs.

Exciting dynamical behavior in a network of two coupled rings of Chen oscillators

We study exotic patterns appearing in a network of coupled Chen oscillators. Namely, we consider a network of two rings coupled through a “buffer” cell, with Z3×Z5 symmetry group. Numerical simulations of the network reveal steady states, rotating waves in one ring and quasiperiodic behavior in the other, and chaotic states in the two rings, to name a few. The different patterns seem to arise through a sequence of Hopf bifurcations, period-doubling, and halving-period bifurcations. The network architecture seems to explain certain observed features, such as equilibria and the rotating waves, whereas the properties of the chaotic oscillator may explain others, such as the quasiperiodic and chaotic states. We use XPPAUT and MATLAB to compute numerically the relevant states.

Symmetry and order parameter dynamics of the human odometer

Bipedal gaits have been classified on the basis of the group symmetry of the minimal network of identical differential equations (alias cells) required to model them. Primary bipedal gaits (e.g., walk, run) are characterized by dihedral symmetry, whereas secondary bipedal gaits (e.g., gallop-walk, gallop- run) are characterized by a lower, cyclic symmetry. This fact has been used in tests of human odometry (e.g., Turvey et al. in P Roy Soc Lond B Biol 276:4309–4314, 2009, J Exp Psychol Hum Percept Perform 38:1014–1025, 2012). Results suggest that when distance is measured and reported by gaits from the same symmetry class, primary and secondary gaits are comparable. Switching symmetry classes at report compresses (primary to secondary) or inflates (secondary to primary) measured distance, with the compression and inflation equal in magnitude. The present research (a) extends these findings from overground locomotion to treadmill locomotion and (b) assesses a dynamics of sequentially coupled measure and report phases, with relative velocity as an order parameter, or equilibrium state, and difference in symmetry class as an imperfection parameter, or detuning, of those dynamics. The results suggest that the symmetries and dynamics of distance measurement by the human odometer are the same whether the odometer is in motion relative to a stationary ground or stationary relative to a moving ground.

Strange Dynamics in a Fractional Derivative of Complex-Order Network of Chaotic Oscillators

We study the peculiar dynamical features of a fractional derivative of complex-order network. The network is composed of two unidirectional rings of cells, coupled through a "buffer" cell. The network has a Z3 × Z5 cyclic symmetry group. The complex derivative Dα±jβ, with α, β ∈ R+ is a generalization of the concept of integer order derivative, where α = 1, β = 0. Each cell is modeled by the Chen oscillator. Numerical simulations of the coupled cell system associated with the network expose patterns such as equilibria, periodic orbits, relaxation oscillations, quasiperiodic motion, and chaos, in one or in two rings of cells. In addition, fixing β = 0.8, we perceive differences in the qualitative behavior of the system, as the parameter c ∈ [13, 24] of the Chen oscillator and/or the real part of the fractional derivative, α ∈ {0.5, 0.6, 0.7, 0.8, 0.9, 1.0}, are varied. Some patterns produced by the coupled system are constrained by the network architecture, but other features are only understood in the light of the internal dynamics of each cell, in this case, the Chen oscillator. What is more important, architecture and/or internal dynamics?