Dynamics of on-line gradient descent learning for multilayer neural networks

We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

On-line learning in multilayer neural networks

We present an analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks. The technique, demonstrated here for the case of adaptive input-to-hidden weights, becomes exact as the dimensionality of the input space increases.

Adaptive back-propagation in on-line learning of multilayer networks

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

Learning with noise and regularizers in multilayer neural networks

We study the effect of two types of noise, data noise and model noise, in an on-line gradient-descent learning scenario for general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labeled by a two-layer teacher network with an arbitrary number of hidden units. Data is then corrupted by Gaussian noise affecting either the output or the model itself. We examine the effect of both types of noise on the evolution of order parameters and the generalization error in various phases of the learning process.

Finite-size effects in on-line learning of multilayer neural networks

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.

Learning with regularizers in multilayer neural networks

We study the effect of regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labelled by a two-layer teacher network with an arbitrary number of hidden units which may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical evolution of the order parameters and generalization error in various phases of the learning process, in both noiseless and noisy scenarios.

Globally optimal parameters for on-line learning in multilayer neural networks

We present a framework for calculating globally optimal parameters, within a given time frame, for on-line learning in multilayer neural networks. We demonstrate the capability of this method by computing optimal learning rates in typical learning scenarios. A similar treatment allows one to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule as well as to compare different training methods.

Globally optimal learning rates in multilayer neural networks

A method for calculating the globally optimal learning rate in on-line gradient-descent training of multilayer neural networks is presented. The method is based on a variational approach which maximizes the decrease in generalization error over a given time frame. We demonstrate the method by computing optimal learning rates in typical learning scenarios. The method can also be employed when different learning rates are allowed for different parameter vectors as well as to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule.

Enterprise networking:multilayer switching and applications

Primarily targeted toward the network or MIS manager who wants to stay abreast of the latest networking technology, Enterprise Networking: Multilayer Switching and Applications offers up to date information relevant for the design of modern corporate networks and for the evaluation of new networking equipment. The book describes the architectures and standards of switching across the various protocol layers and will also address issues such as multicast quality of service, high-availability and network policies that are requirements of modern switched networks.

Nonlinearity in all-night sleep EEG recorded with foramen ovale electrodes in a patient with temporal lobe epilepsy

Objective: It is investigated to which extent measures of nonlinearity derived from surrogate data analysis are capable to quantify the changes of epileptic activity related to varying vigilance levels. Methods: Surface and intracranial EEG from foramen ovale (FO-)electrodes was recorded from a patient with temporal lobe epilepsy under presurgical evaluation over one night. Different measures of nonlinearity were estimated for non-overlapping 30-s segments for selected channels from surface and intracranial EEG. Additionally spectral measures were calculated. Sleep stages were scored according to Rechtschaffen/Kales and epileptic transients were counted and classified by visual inspection. Results: In the intracranial recordings stronger nonlinearity was found ipsilateral to the epileptogenic focus, more pronounced in NREM sleep, weaker in REM sleep. The dynamics within the NREM episodes varied with the different nonlinearity measures. Some nonlinearity measures showed variations with the sleep cycle also in the intracranial recordings contralateral to the epileptic focus and in the surface EEG. It is shown that the nonlinearity is correlated with short-term fluctuations of the delta power. The higher frequency of occurrence of clinical relevant epileptic spikes in the first NREM episode was not clearly reflected in the nonlinearity measures. Conclusions: It was confirmed that epileptic activity renders the EEG nonlinear. However, it was shown that the sleep dynamics itself also effects the nonlinearity measures. Therefore, at the present stage it is not possible to establish a unique connection between the studied nonlinearity measures and specific types of epileptic activity in sleep EEG recordings.

Exploitation of multilayer coatings for infrared surface plasmon resonance fiber sensors

We demonstrate surface plasmon resonance (SPR) fiber devices based upon ultraviolet inscription of a grating-type structure into both single-layered and multilayered thin films deposited on the flat side of a lapped D-shaped fiber. The single-layered devices were fabricated from germanium, while the multilayered ones comprised layers of germanium, silica, and silver. Some of the devices operated in air with high coupling efficiency in excess of 40 dB and an estimated index sensitivity of Delta lambda/Delta n = 90 mn from 1 to 1.15 index range, while others provided an index sensitivity of Delta lambda/Delta n = 6790 mn for refractive indices from 1.33 to 1.37. (C) 2009 Optical Society of America