Dynamics of on-line gradient descent learning for multilayer neural networks

We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

On-line learning in multilayer neural networks

We present an analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks. The technique, demonstrated here for the case of adaptive input-to-hidden weights, becomes exact as the dimensionality of the input space increases.

Adaptive back-propagation in on-line learning of multilayer networks

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

Learning with noise and regularizers in multilayer neural networks

We study the effect of two types of noise, data noise and model noise, in an on-line gradient-descent learning scenario for general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labeled by a two-layer teacher network with an arbitrary number of hidden units. Data is then corrupted by Gaussian noise affecting either the output or the model itself. We examine the effect of both types of noise on the evolution of order parameters and the generalization error in various phases of the learning process.

Finite-size effects in on-line learning of multilayer neural networks

We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating fluctuations possessed by finite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time.

Learning with regularizers in multilayer neural networks

We study the effect of regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labelled by a two-layer teacher network with an arbitrary number of hidden units which may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical evolution of the order parameters and generalization error in various phases of the learning process, in both noiseless and noisy scenarios.

Globally optimal parameters for on-line learning in multilayer neural networks

We present a framework for calculating globally optimal parameters, within a given time frame, for on-line learning in multilayer neural networks. We demonstrate the capability of this method by computing optimal learning rates in typical learning scenarios. A similar treatment allows one to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule as well as to compare different training methods.

Globally optimal learning rates in multilayer neural networks

A method for calculating the globally optimal learning rate in on-line gradient-descent training of multilayer neural networks is presented. The method is based on a variational approach which maximizes the decrease in generalization error over a given time frame. We demonstrate the method by computing optimal learning rates in typical learning scenarios. The method can also be employed when different learning rates are allowed for different parameter vectors as well as to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule.

Separation of n-paraffins by selective adsorption

A study has been undertaken of the vapor-phase adsorptive separation of n-alkanes from Kuwait kerosene (Kuwait National Petroleum Company, heavy kerosene) using zeolite molecular sieves. Due to the shortage of information on the adsorption of multicomponent systems in the open literature, the present investigation was initiated to study the effect of feed flowrate, temperature, and zeolite particle size on the height of mass transfer zone (MTZ) and the dynamic capacity of the adsorbent for multicomponent n-alkanes adsorption on a fixed-bed of zeolite type-5A. The optimum operating conditions for separation of the n-alkanes has been identified so that the effluent would also be of marketable quality. The effect of multicycle adsorption-desorption stages on the dynamic behaviour of zeolite using steam as a desorbing agent has been studied and compared with n-pentane and n-hexane as desorbing agents. The separation process comprised one cycle of adsorption using a fixed-bed of zeolite type-5A. The bed was fed with vaporized kerosene until saturation had been achieved whereby the n-alkanes were adsorbed and the denormalized material eluted. The process of adsorption-desorption was carried out isobarically at one atmosphere. A mathematical model has been developed to predict the breakthrough time using the method of characteristics. The results were in a reasonable agreement with the experimental values. This model has also been utilized to develop the equilibrium isotherm. Optimum operating conditions were achieved at a feed flowrate of 33.33 x 10-9 m3/s, a temperature of 643 K, and a particle size of (1.0 - 2.0) x 10-3 m. This yielded an HMTZ value and a dynamic capacity of 0.206 m and 9.6S3 x 10-2 kg n-alkanes/kg of zeolite respectively. These data will serve as a basis for design of a commercial plant. The purity of liquid-paraffin product desorbed using steam was 83.24 wt%. The dynamic capacity was noticed to decrease sharply with the cycle number, without intermediate reactivation of zeolite, while it was kept unchanged by intermediate reactivation. Normal hexane was found to be the best desorbing agent, the efficiency of which was mounted to 88.2%.

Enterprise networking:multilayer switching and applications

Primarily targeted toward the network or MIS manager who wants to stay abreast of the latest networking technology, Enterprise Networking: Multilayer Switching and Applications offers up to date information relevant for the design of modern corporate networks and for the evaluation of new networking equipment. The book describes the architectures and standards of switching across the various protocol layers and will also address issues such as multicast quality of service, high-availability and network policies that are requirements of modern switched networks.