Regularization and complexity control in feed-forward networks

In this paper we consider four alternative approaches to complexity control in feed-forward networks based respectively on architecture selection, regularization, early stopping, and training with noise. We show that there are close similarities between these approaches and we argue that, for most practical applications, the technique of regularization should be the method of choice.

Curvature-driven smoothing: a learning algorithm for feed-forward networks

The performance of feed-forward neural networks in real applications can be often be improved significantly if use is made of a-priori information. For interpolation problems this prior knowledge frequently includes smoothness requirements on the network mapping, and can be imposed by the addition to the error function of suitable regularization terms. The new error function, however, now depends on the derivatives of the network mapping, and so the standard back-propagation algorithm cannot be applied. In this paper, we derive a computationally efficient learning algorithm, for a feed-forward network of arbitrary topology, which can be used to minimize the new error function. Networks having a single hidden layer, for which the learning algorithm simplifies, are treated as a special case.

Topographic mappings and feed-forward neural networks

This thesis is a study of the generation of topographic mappings - dimension reducing transformations of data that preserve some element of geometric structure - with feed-forward neural networks. As an alternative to established methods, a transformational variant of Sammon's method is proposed, where the projection is effected by a radial basis function neural network. This approach is related to the statistical field of multidimensional scaling, and from that the concept of a 'subjective metric' is defined, which permits the exploitation of additional prior knowledge concerning the data in the mapping process. This then enables the generation of more appropriate feature spaces for the purposes of enhanced visualisation or subsequent classification. A comparison with established methods for feature extraction is given for data taken from the 1992 Research Assessment Exercise for higher educational institutions in the United Kingdom. This is a difficult high-dimensional dataset, and illustrates well the benefit of the new topographic technique. A generalisation of the proposed model is considered for implementation of the classical multidimensional scaling (¸mds}) routine. This is related to Oja's principal subspace neural network, whose learning rule is shown to descend the error surface of the proposed ¸mds model. Some of the technical issues concerning the design and training of topographic neural networks are investigated. It is shown that neural network models can be less sensitive to entrapment in the sub-optimal global minima that badly affect the standard Sammon algorithm, and tend to exhibit good generalisation as a result of implicit weight decay in the training process. It is further argued that for ideal structure retention, the network transformation should be perfectly smooth for all inter-data directions in input space. Finally, there is a critique of optimisation techniques for topographic mappings, and a new training algorithm is proposed. A convergence proof is given, and the method is shown to produce lower-error mappings more rapidly than previous algorithms.

Novel real-time homodyne coherent receiver using a feed-forward based carrier extraction scheme for phase modulated signals

We report a novel real-time homodyne coherent receiver based on a DPSK optical-electrical-optical (OEO) regenerator used to extract a carrier from carrier-less phase modulated signals based on feed-forward based modulation stripping. The performance of this non-DSP based coherent receiver was evaluated for 10.66Gbit/s BPSK signals. Self-homodyne coherent detection and homodyne detection with an injection-locked local oscillator laser was demonstrated. The performance was evaluated by measuring the electrical signal-to-noise (SNR) and recording the eye diagrams. Using injection-locking for the LO improves the performance and enables homodyne detection with optical injection-locking to operate with carrier-less BPSK signals without the need for polarization multiplexed pilot-tones.

Novel synchronous DPSK optical regenerator based on a feed-forward based carrier extraction scheme

We experimentally demonstrate a novel synchronous 10.66Gbit/s DPSK OEO regenerator which uses a feed-forward carrier extraction scheme with an injection-locked laser to synchronize the regenerated signal wavelength to the incoming signal wavelength. After injection-locking, a low-cost DFB laser used at the regenerator exhibited the same linewidth characteristics as the narrow line-width transmitter laser. The phase regeneration properties of the regenerator were evaluated by emulating random Gaussian phase noise applied to the DPSK signal before the regenerator using a phase modulator driven by an arbitrary waveform generator. The overall performance was evaluated in terms of electrical eye-diagrams, BER measurements, and constellation diagrams.

Full-field feed-forward equalizer with adaptive system optimization

We numerically investigate the combination of full-field detection and feed-forward equalizer (FFE) for adaptive chromatic dispersion compensation up to 2160 km in a 10 Gbit/s on-off keyed optical transmission system. The technique, with respect to earlier reports, incorporates several important implementation modules, including the algorithm for adaptive equalization of the gain imbalance between the two receiver chains, compensation of phase misalignment of the asymmetric Mach-Zehnder interferometer, and simplified implementation of field calculation. We also show that in addition to enabling fast adaptation and simplification of field calculation, full-field FFE exhibits enhanced tolerance to the sampling phase misalignment and reduced sampling rate when compared to the full-field implementation using a dispersive transmission line.

A learning feed-forward current controller for linear reciprocating vapor compressors

Direct-drive linear reciprocating compressors offer numerous advantages over conventional counterparts which are usually driven by a rotary induction motor via a crank shaft. However, to ensure efficient and reliable operation under all conditions, it is essential that motor current of a linear compressor follows a sinusoidal current command with a frequency which matches the system resonant frequency. The design of a high-performance current controller for linear compressor drive presents a challenge since the system is highly nonlinear, and an effective solution must be low cost. In this paper, a learning feed-forward current controller for the linear compressors is proposed. It comprises a conventional feedback proportional-integral controller and a feed-forward B-spline neural network (BSNN). The feed-forward BSNN is trained online and in real time in order to minimize the current tracking error. Extensive simulation and experiment results with a prototype linear compressor show that the proposed current controller exhibits high steady state and transient performance. © 2009 IEEE.

Feed-forward neural networks and topographic mappings for exploratory data analysis

A recent novel approach to the visualisation and analysis of datasets, and one which is particularly applicable to those of a high dimension, is discussed in the context of real applications. A feed-forward neural network is utilised to effect a topographic, structure-preserving, dimension-reducing transformation of the data, with an additional facility to incorporate different degrees of associated subjective information. The properties of this transformation are illustrated on synthetic and real datasets, including the 1992 UK Research Assessment Exercise for funding in higher education. The method is compared and contrasted to established techniques for feature extraction, and related to topographic mappings, the Sammon projection and the statistical field of multidimensional scaling.

Full-field feed-forward equalizer with adaptive system optimization

We numerically investigate the combination of full-field detection and feed-forward equalizer (FFE) for adaptive chromatic dispersion compensation up to 2160 km in a 10 Gbit/s on-off keyed optical transmission system. The technique, with respect to earlier reports, incorporates several important implementation modules, including the algorithm for adaptive equalization of the gain imbalance between the two receiver chains, compensation of phase misalignment of the asymmetric Mach-Zehnder interferometer, and simplified implementation of field calculation. We also show that in addition to enabling fast adaptation and simplification of field calculation, full-field FFE exhibits enhanced tolerance to the sampling phase misalignment and reduced sampling rate when compared to the full-field implementation using a dispersive transmission line.

A mixture density network approach to modelling and exploiting uncertainty in nonlinear control problems

In nonlinear and stochastic control problems, learning an efficient feed-forward controller is not amenable to conventional neurocontrol methods. For these approaches, estimating and then incorporating uncertainty in the controller and feed-forward models can produce more robust control results. Here, we introduce a novel inversion-based neurocontroller for solving control problems involving uncertain nonlinear systems which could also compensate for multi-valued systems. The approach uses recent developments in neural networks, especially in the context of modelling statistical distributions, which are applied to forward and inverse plant models. Provided that certain conditions are met, an estimate of the intrinsic uncertainty for the outputs of neural networks can be obtained using the statistical properties of networks. More generally, multicomponent distributions can be modelled by the mixture density network. Based on importance sampling from these distributions a novel robust inverse control approach is obtained. This importance sampling provides a structured and principled approach to constrain the complexity of the search space for the ideal control law. The developed methodology circumvents the dynamic programming problem by using the predicted neural network uncertainty to localise the possible control solutions to consider. A nonlinear multi-variable system with different delays between the input-output pairs is used to demonstrate the successful application of the developed control algorithm. The proposed method is suitable for redundant control systems and allows us to model strongly non-Gaussian distributions of control signal as well as processes with hysteresis. © 2004 Elsevier Ltd. All rights reserved.

Fast response static var generator for improving the power system performance of an electric arc furnace (EAF)

Cascaded multilevel inverters-based Static Var Generators (SVGs) are FACTS equipment introduced for active and reactive power flow control. They eliminate the need for zigzag transformers and give a fast response. However, with regard to their application for flicker reduction in using Electric Arc Furnace (EAF), the existing multilevel inverter-based SVGs suffer from the following disadvantages. (1) To control the reactive power, an off-line calculation of Modulation Index (MI) is required to adjust the SVG output voltage. This slows down the transient response to the changes of reactive power; and (2) Random active power exchange may cause unbalance to the voltage of the d.c. link (HBI) capacitor when the reactive power control is done by adjusting the power angle d alone. To resolve these problems, a mathematical model of 11-level cascaded SVG, was developed. A new control strategy involving both MI (modulation index) and power angle (d) is proposed. A selected harmonics elimination method (SHEM) is taken for switching pattern calculations. To shorten the response time and simplify the controls system, feed forward neural networks are used for on-line computation of the switching patterns instead of using look-up tables. The proposed controller updates the MI and switching patterns once each line-cycle according to the sampled reactive power Qs. Meanwhile, the remainder reactive power (compensated by the MI) and the reactive power variations during the line-cycle will be continuously compensated by adjusting the power angles, d. The scheme senses both variables MI and d, and takes action through the inverter switching angle, qi. As a result, the proposed SVG is expected to give a faster and more accurate response than present designs allow. In support of the proposal there is a mathematical model for reactive powered distribution and a sensitivity matrix for voltage regulation assessment, MATLAB simulation results are provided to validate the proposed schemes. The performance with non-linear time varying loads is analysed and refers to a general review of flicker, of methods for measuring flickers due to arc furnace and means for mitigation.

Minimum description length, regularisation and multi-modal data

Conventional feed forward Neural Networks have used the sum-of-squares cost function for training. A new cost function is presented here with a description length interpretation based on Rissanen's Minimum Description Length principle. It is a heuristic that has a rough interpretation as the number of data points fit by the model. Not concerned with finding optimal descriptions, the cost function prefers to form minimum descriptions in a naive way for computational convenience. The cost function is called the Naive Description Length cost function. Finding minimum description models will be shown to be closely related to the identification of clusters in the data. As a consequence the minimum of this cost function approximates the most probable mode of the data rather than the sum-of-squares cost function that approximates the mean. The new cost function is shown to provide information about the structure of the data. This is done by inspecting the dependence of the error to the amount of regularisation. This structure provides a method of selecting regularisation parameters as an alternative or supplement to Bayesian methods. The new cost function is tested on a number of multi-valued problems such as a simple inverse kinematics problem. It is also tested on a number of classification and regression problems. The mode-seeking property of this cost function is shown to improve prediction in time series problems. Description length principles are used in a similar fashion to derive a regulariser to control network complexity.

Bayesian methods for neural networks

Bayesian techniques have been developed over many years in a range of different fields, but have only recently been applied to the problem of learning in neural networks. As well as providing a consistent framework for statistical pattern recognition, the Bayesian approach offers a number of practical advantages including a potential solution to the problem of over-fitting. This chapter aims to provide an introductory overview of the application of Bayesian methods to neural networks. It assumes the reader is familiar with standard feed-forward network models and how to train them using conventional techniques.

Bayesian methods for neural networks

Bayesian techniques have been developed over many years in a range of different fields, but have only recently been applied to the problem of learning in neural networks. As well as providing a consistent framework for statistical pattern recognition, the Bayesian approach offers a number of practical advantages including a potential solution to the problem of over-fitting. This chapter aims to provide an introductory overview of the application of Bayesian methods to neural networks. It assumes the reader is familiar with standard feed-forward network models and how to train them using conventional techniques.

Transients and asymptotics of natural gradient learning

We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.