Two-Stage Orthogonal Least Squares Methods for Neural Network Construction

A number of neural networks can be formulated as the linear-in-the-parameters models. Training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms. Forward selection methods are popular fast subset selection approaches. However, they may only produce suboptimal models and can be trapped into a local minimum. More recently, a two-stage fast recursive algorithm (TSFRA) combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model. This paper proposes unified two-stage orthogonal least squares methods instead of the fast recursive-based methods. In contrast to the TSFRA, this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods. Furthermore, a new term exchanging scheme for backward model refinement is introduced to reduce computational demand. Finally, given the error reduction ratio criterion, effective and efficient forward and backward subset selection procedures are proposed. Extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods.

Two-stage RBF network construction based on particle swarm optimization

The conventional radial basis function (RBF) network optimization methods, such as orthogonal least squares or the two-stage selection, can produce a sparse network with satisfactory generalization capability. However, the RBF width, as a nonlinear parameter in the network, is not easy to determine. In the aforementioned methods, the width is always pre-determined, either by trial-and-error, or generated randomly. Furthermore, all hidden nodes share the same RBF width. This will inevitably reduce the network performance, and more RBF centres may then be needed to meet a desired modelling specification. In this paper we investigate a new two-stage construction algorithm for RBF networks. It utilizes the particle swarm optimization method to search for the optimal RBF centres and their associated widths. Although the new method needs more computation than conventional approaches, it can greatly reduce the model size and improve model generalization performance. The effectiveness of the proposed technique is confirmed by two numerical simulation examples.

Fast automatic two-stage nonlinear model identification based on the extreme learning machine

It is convenient and effective to solve nonlinear problems with a model that has a linear-in-the-parameters (LITP) structure. However, the nonlinear parameters (e.g. the width of Gaussian function) of each model term needs to be pre-determined either from expert experience or through exhaustive search. An alternative approach is to optimize them by a gradient-based technique (e.g. Newton’s method). Unfortunately, all of these methods still need a lot of computations. Recently, the extreme learning machine (ELM) has shown its advantages in terms of fast learning from data, but the sparsity of the constructed model cannot be guaranteed. This paper proposes a novel algorithm for automatic construction of a nonlinear system model based on the extreme learning machine. This is achieved by effectively integrating the ELM and leave-one-out (LOO) cross validation with our two-stage stepwise construction procedure [1]. The main objective is to improve the compactness and generalization capability of the model constructed by the ELM method. Numerical analysis shows that the proposed algorithm only involves about half of the computation of orthogonal least squares (OLS) based method. Simulation examples are included to confirm the efficacy and superiority of the proposed technique.

A two-stage algorithm for identification of nonlinear dynamic systems

This paper investigates the two-stage stepwise identification for a class of nonlinear dynamic systems that can be described by linear-in-the-parameters models, and the model has to be built from a very large pool of basis functions or model terms. The main objective is to improve the compactness of the model that is obtained by the forward stepwise methods, while retaining the computational efficiency. The proposed algorithm first generates an initial model using a forward stepwise procedure. The significance of each selected term is then reviewed at the second stage and all insignificant ones are replaced, resulting in an optimised compact model with significantly improved performance. The main contribution of this paper is that these two stages are performed within a well-defined regression context, leading to significantly reduced computational complexity. The efficiency of the algorithm is confirmed by the computational complexity analysis, and its effectiveness is demonstrated by the simulation results.

Modelling the MAPK signalling pathway using a two-stage identification algorithm

Signal transduction pathways describe the dynamics of cellular response to input signalling molecules at receptors on the cell membrane. The Mitogen-Activated Protein Kinase (MAPK) cascade is one of such pathways that are involved in many important cellular processes including cell growth and proliferation. This paper describes a black-box model of this pathway created using an advanced two-stage identification algorithm. Identification allows us to capture the unique features and dynamics of the pathway and also opens up the possibility of regulatory control design. In the approach described, an optimal model is obtained by performing model subset selection in two stages, where the terms are first determined by a forward selection method and then modified using a backward selection model refinement. The simulation results demonstrate that the model selected using the two-stage algorithm performs better than with the forward selection method alone.

Two-stage mixed discrete-continuous identification of radial basis function (RBF) neural models for nonlinear systems

The identification of nonlinear dynamic systems using radial basis function (RBF) neural models is studied in this paper. Given a model selection criterion, the main objective is to effectively and efficiently build a parsimonious compact neural model that generalizes well over unseen data. This is achieved by simultaneous model structure selection and optimization of the parameters over the continuous parameter space. It is a mixed-integer hard problem, and a unified analytic framework is proposed to enable an effective and efficient two-stage mixed discrete-continuous; identification procedure. This novel framework combines the advantages of an iterative discrete two-stage subset selection technique for model structure determination and the calculus-based continuous optimization of the model parameters. Computational complexity analysis and simulation studies confirm the efficacy of the proposed algorithm.

Locally regularised two-stage learning algorithm for RBF network centre selection

Nonlinear models constructed from radial basis function (RBF) networks can easily be over-fitted due to the noise on the data. While information criteria, such as the final prediction error (FPE), can provide a trade-off between training error and network complexity, the tunable parameters that penalise a large size of network model are hard to determine and are usually network dependent. This article introduces a new locally regularised, two-stage stepwise construction algorithm for RBF networks. The main objective is to produce a parsomous network that generalises well over unseen data. This is achieved by utilising Bayesian learning within a two-stage stepwise construction procedure to penalise centres that are mainly interpreted by the noise.

Two-Stage Transmitter Precoding Based on Data-Driven Code Hopping and Partial Zero Forcing Beamforming for MC-DCMA Communications

This paper proposes a hybrid transmission technique based on adaptive code-to-user allocation and linear precoding for the downlink of phase shift keying (PSK) based multi-carrier code division multiple access (MC-CDMA) systems. The proposed scheme is based on the separation of the instantaneous multiple access interference (MAI) into constructive and destructive components taking into account the dependency on both the channel variation and the instantaneous symbol values of the active users. The first stage of the proposed technique is to adaptively distribute the available spreading sequences to the users on a symbol-by-symbol basis in the form of codehopping with the objective to steer the users' instantaneous crosscorrelations to yield a favourable constructive to destructive MAI ratio. The second stage is to employ a partial transmitter based zero forcing (ZF) scheme specifically designed for the exploitation of constructive MAI. The partial ZF processing decorrelates destructive interferers, while users that interfere constructively remain correlated. This results in a signal to interference-plus-noise ratio (SINR) enhancement without the need for additional power-per-user investment. It will be shown in the results section that significant bit error rate (BER) performance benefits can be achieved with this technique.

Allele resolution of HLA-A using oligonucleotide probes in a two-stage typing strategy

High-resolution polymerase chain reaction using sequence-specific oligonucleotide probes (PCR-SSOP) typing methods for HLA-A identification have been established. The four systems, which operate independently of each other, are intended for use as secondary typing systems following HLA-A identification with a medium-resolution PCR-SSOP technique. The systems, all using digoxigenin-labelled probes, are based on group specific amplifications for resolution of: i) HLA-A*29 & -A*33; ii) HLA-A*24 & -A*30; and iii) HLA-A*26, -A*25, -A*11, -A*34, -A*66 and -A*68 alleles, respectively. The fourth system, for the detection of HLA-A*02 alleles, is a modification of a previously reported PCR-SSOP subtyping system. The methods have been applied to individuals from the local bone marrow registry and HLA-A allele frequencies for the Northern Ireland population have been established.