The role of biases in on-line learning of two-layer networks

The influence of biases on the learning dynamics of a two-layer neural network, a normalized soft-committee machine, is studied for on-line gradient descent learning. Within a statistical mechanics framework, numerical studies show that the inclusion of adjustable biases dramatically alters the learning dynamics found previously. The symmetric phase which has often been predominant in the original model all but disappears for a non-degenerate bias task. The extended model furthermore exhibits a much richer dynamical behavior, e.g. attractive suboptimal symmetric phases even for realizable cases and noiseless data.

On-line learning with adaptive back-propagation in two-layer networks

An adaptive back-propagation algorithm parameterized by an inverse temperature 1/T is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, we analyse these learning algorithms in both the symmetric and the convergence phase for finite learning rates in the case of uncorrelated teachers of similar but arbitrary length T. These analyses show that adaptive back-propagation results generally in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

On-line learning in radial basis functions networks

An analytic investigation of the average case learning and generalization properties of Radial Basis Function Networks (RBFs) is presented, utilising on-line gradient descent as the learning rule. The analytic method employed allows both the calculation of generalization error and the examination of the internal dynamics of the network. The generalization error and internal dynamics are then used to examine the role of the learning rate and the specialization of the hidden units, which gives insight into decreasing the time required for training. The realizable and over-realizable cases are studied in detail; the phase of learning in which the hidden units are unspecialized (symmetric phase) and the phase in which asymptotic convergence occurs are analyzed, and their typical properties found. Finally, simulations are performed which strongly confirm the analytic results.

Incorporating curvature information into on-line learning

We analyse the dynamics of a number of second order on-line learning algorithms training multi-layer neural networks, using the methods of statistical mechanics. We first consider on-line Newton's method, which is known to provide optimal asymptotic performance. We determine the asymptotic generalization error decay for a soft committee machine, which is shown to compare favourably with the result for standard gradient descent. Matrix momentum provides a practical approximation to this method by allowing an efficient inversion of the Hessian. We consider an idealized matrix momentum algorithm which requires access to the Hessian and find close correspondence with the dynamics of on-line Newton's method. In practice, the Hessian will not be known on-line and we therefore consider matrix momentum using a single example approximation to the Hessian. In this case good asymptotic performance may still be achieved, but the algorithm is now sensitive to parameter choice because of noise in the Hessian estimate. On-line Newton's method is not appropriate during the transient learning phase, since a suboptimal unstable fixed point of the gradient descent dynamics becomes stable for this algorithm. A principled alternative is to use Amari's natural gradient learning algorithm and we show how this method provides a significant reduction in learning time when compared to gradient descent, while retaining the asymptotic performance of on-line Newton's method.

On-line learning of unrealizable tasks

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

On-line learning in neural networks

On-line learning is one of the most powerful and commonly used techniques for training large layered networks and has been used successfully in many real-world applications. Traditional analytical methods have been recently complemented by ones from statistical physics and Bayesian statistics. This powerful combination of analytical methods provides more insight and deeper understanding of existing algorithms and leads to novel and principled proposals for their improvement. This book presents a coherent picture of the state-of-the-art in the theoretical analysis of on-line learning. An introduction relates the subject to other developments in neural networks and explains the overall picture. Surveys by leading experts in the field combine new and established material and enable non-experts to learn more about the techniques and methods used. This book, the first in the area, provides a comprehensive view of the subject and will be welcomed by mathematicians, scientists and engineers, whether in industry or academia.

A Bayesian approach to on-line learning

Online learning is discussed from the viewpoint of Bayesian statistical inference. By replacing the true posterior distribution with a simpler parametric distribution, one can define an online algorithm by a repetition of two steps: An update of the approximate posterior, when a new example arrives, and an optimal projection into the parametric family. Choosing this family to be Gaussian, we show that the algorithm achieves asymptotic efficiency. An application to learning in single layer neural networks is given.

Effect of β-adrenoceptor antagonists on autonomic control of ciliary smooth muscle

Purpose: Pharmacological intervention with peripheral sympathetic transmission at ciliary smooth muscle neuro-receptor junctions has been used against a background of controlled parasympathetic activity to investigate the characteristics of autonomic control of ocular accommodation. Methods: A continuously recording infrared optometer was used to measure accommodation on a group of five visually normal emmetropic subjects under open- and closed-loop conditions. A double-blind protocol between saline, timolol and betaxolol was used to differentiate between the localised action on ciliary smooth muscle and effects induced by changes in stimulus conditions. Data were collected before and 45 min following the instillation of saline, timolol or betaxolol. Open-loop post-task decay was investigated following 3 min sustained near fixation of a stimulus placed 3 D above the subject's pre-task tonic accommodation level. Closed-loop dynamic responses were recorded for each treatment condition while subjects viewed sinusoidally (0.05-0.6 Hz) or stepwise vergence-modulated targets over a 2 D range (2-4 D). Results: Open-loop data demonstrate a rapid post-task regression to pre-task tonic accommodation levels for saline and betaxolol control conditions. A slow positive post-task shift was induced by timolol indicating that sympathetic inhibition contributes to accommodative adaptation during sustained near vision. Closed-loop accommodation responses to temporally modulated sinusoidal stimuli showed characteristic features for both saline and betaxolol control conditions. Timolol induced a reduced gain for low- and mid-temporal frequencies (< 0.3 Hz) but did not affect the response at higher temporal frequencies. Response times to stepwise stimuli increased following the instillation of timolol for the near-to-far fixation condition compared with the controls and was related to the period of sustained prior fixation. Conclusions: Modulation of accommodation under open- and closed-loop conditions by a non-selective β-blocker is consistent with the temporal and inhibitory features of sympathetic innervation to ciliary smooth muscle. Although parasympathetic innervation predominates there is evidence to support a role for sympathetic innervation in the control of ocular accommodation. © 2002 The College of Optometrists.

High stability interleaved fibre Michelson interferometer for on-line precision displacement measurements

A self-reference fiber Michelson interferometer measurement system, which employs fiber Bragg gratings (FBGs) as in-fiber reflective mirrors and interleaves together two fiber Michelson interferometers that share the common-interferometric-optical path, is presented. One of the fiber interferometers is used to stabilise the system by the use of an electronic feedback loop to compensate the influences resulting from the environmental disturbances, while the other one is used to perform the measurement task. The influences resulting from the environmental disturbances have been eliminated by the compensating action of the electronic feedback loop, this makes the system suitable for on-line precision measurement. By means of the homodyne phase-tracking technique, the linearity of the measurement results of displacement measurements has been very high.

Investigation of some critical aspects of on-line surface measurement by a wavelength-division-multiplexing technique

Some critical aspects of a new kind of on-line measurement technique for micro and nanoscale surface measurements are described. This attempts to use spatial light-wave scanning to replace mechanical stylus scanning, and an optical fibre interferometer to replace optically bulky interferometers for measuring the surfaces. The basic principle is based on measuring the phase shift of a reflected optical signal. Wavelength-division-multiplexing and fibre Bragg grating techniques are used to carry out wavelength-to-field transformation and phase-to-depth detection, allowing a large dynamic measurement ratio (range/resolution) and high signal-to-noise ratio with remote access. In effect the paper consists of two parts: multiplexed fibre interferometry and remote on-machine surface detection sensor (an optical dispersive probe). This paper aims to investigate the metrology properties of a multiplexed fibre interferometer and to verify its feasibility by both theoretical and experimental studies. Two types of optical probes, using a dispersive prism and a blazed grating, respectively, are introduced to realize wavelength-to-spatial scanning.