The role of biases in on-line learning of two-layer networks

The influence of biases on the learning dynamics of a two-layer neural network, a normalized soft-committee machine, is studied for on-line gradient descent learning. Within a statistical mechanics framework, numerical studies show that the inclusion of adjustable biases dramatically alters the learning dynamics found previously. The symmetric phase which has often been predominant in the original model all but disappears for a non-degenerate bias task. The extended model furthermore exhibits a much richer dynamical behavior, e.g. attractive suboptimal symmetric phases even for realizable cases and noiseless data.

On-line learning with adaptive back-propagation in two-layer networks

An adaptive back-propagation algorithm parameterized by an inverse temperature 1/T is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, we analyse these learning algorithms in both the symmetric and the convergence phase for finite learning rates in the case of uncorrelated teachers of similar but arbitrary length T. These analyses show that adaptive back-propagation results generally in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

The structure and dynamics of 2-dimensional fluids in swelling clays

The interlayer pores of swelling 2:1 clays provide an ideal 2-dimensional environment in which to study confined fluids. In this paper we discuss our understanding of the structure and dynamics of interlayer fluid species in expanded clays, based primarily on the outcome of recent molecular modelling and neutron scattering studies. Counterion solvation is compared with that measured in bulk solutions, and at a local level the cation-oxygen coordination is found to be remarkably similar in these two environments. However, for the monovalent ions the contribution to the first coordination shell from the clay surfaces increases with counterion radius. This gives rise to inner-sphere (surface) complexes in the case of potassium and caesium. In this context, the location of the negative clay surface charge (i.e. arising from octahedral or tetrahedral substitution) is also found to be of major importance. Divalent cations, such as calcium, eagerly solvate to form outer-sphere complexes. These complexes are able to pin adjacent clay layers together, and thereby prevent colloidal swelling. Confined water molecules form hydrogen bonds to each other and to the clays' surfaces. In this way their local environment relaxes to close to the bulk water structure within two molecular layers of the clay surface. Finally, we discuss the way in which the simple organic molecules methane, methanol and ethylene glycol behave in the interlayer region of hydrated clays. Quasi-elastic neutron scattering of isotopically labelled interlayer CH 3OD and (CH2OD)2 in deuterated clay allows us to measure the diffusion of the CH3- and CH2-groups in both clay and liquid environments. We find that in both the one-layer methanol solvates and the two-layer glycol solvates the diffusion of the most mobile organic molecules is close to that in the bulk solution.

Efficient THz radiation from a nanocrystalline silicon-based multi-layer photomixer

In this paper we propose a novel type of multiple-layer photomixer based on amorphous/nano-crystalline-Si. Such a device implies that it could be possible to enhance the conversion efficiency from optical power to THz emission by increasing the absorption length and by reducing the device overheating through the use of substrates with higher thermal conductivity compared to GaAs. Our calculations show that the output power from a two-layer Si-based photomixer is at least ten times higher than that from conventional LT-GaAs photomixers at 1 THz.

Dynamics of on-line gradient descent learning for multilayer neural networks

We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

On-line learning in multilayer neural networks

We present an analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks. The technique, demonstrated here for the case of adaptive input-to-hidden weights, becomes exact as the dimensionality of the input space increases.

Adaptive back-propagation in on-line learning of multilayer networks

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

Learning with noise and regularizers in multilayer neural networks

We study the effect of two types of noise, data noise and model noise, in an on-line gradient-descent learning scenario for general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labeled by a two-layer teacher network with an arbitrary number of hidden units. Data is then corrupted by Gaussian noise affecting either the output or the model itself. We examine the effect of both types of noise on the evolution of order parameters and the generalization error in various phases of the learning process.

The learning dynamics of a universal approximator

The learning properties of a universal approximator, a normalized committee machine with adjustable biases, are studied for on-line back-propagation learning. Within a statistical mechanics framework, numerical studies show that this model has features which do not exist in previously studied two-layer network models without adjustable biases, e.g., attractive suboptimal symmetric phases even for realizable cases and noiseless data.

Learning with regularizers in multilayer neural networks

We study the effect of regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labelled by a two-layer teacher network with an arbitrary number of hidden units which may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical evolution of the order parameters and generalization error in various phases of the learning process, in both noiseless and noisy scenarios.

Transients and asymptotics of natural gradient learning

We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.

On-line learning of unrealizable tasks

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

Natural gradient matrix momentum

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.

Monte Carlo and molecular dynamics simulations of methane in potassium montmorillonite clay hydrates at elevated pressures and temperatures

The structure and dynamics of methane in hydrated potassium montmorillonite clay have been studied under conditions encountered in sedimentary basin and compared to those of hydrated sodium montmorillonite clay using computer simulation techniques. The simulated systems contain two molecular layers of water and followed gradients of 150 barkm-1 and 30 Kkm-1 up to a maximum burial depth of 6 km. Methane particle is coordinated to about 19 oxygen atoms, with 6 of these coming from the clay surface oxygen. Potassium ions tend to move away from the center towards the clay surface, in contrast to the behavior observed with the hydrated sodium form. The clay surface affinity for methane was found to be higher in the hydrated K-form. Methane diffusion in the two-layer hydrated K-montmorillonite increases from 0.39×10-9 m2s-1 at 280 K to 3.27×10-9 m2s-1 at 460 K compared to 0.36×10-9 m2s-1 at 280 K to 4.26×10-9 m2s-1 at 460 K in Na-montmorillonite hydrate. The distributions of the potassium ions were found to vary in the hydrates when compared to those of sodium form. Water molecules were also found to be very mobile in the potassium clay hydrates compared to sodium clay hydrates. © 2004 Elsevier Inc. All All rights reserved.

Rutting-induced permeability loss of open graded friction course mixtures

Functionality of an open graded friction course (OGFC) depends on the high interconnected air voids or pores of the OGFC mixture. The authors' previous study indicated that the pores in the OGFC mixture were easily clogged by rutting deformation. Such a deformation-related clogging can cause a significant rutting-induced permeability loss in the OGFC mixture. The objective of this study was to control and reduce the rutting-induced permeability loss of the OGFC based on mixture design and layer thickness. Eight types of the OGFC mixtures with different air void contents, gradations, and nominal maximum aggregate sizes were fabricated in the laboratory. Wheel-tracking rutting tests were conducted on the OGFC slabs to simulate the deformation-related clogging. Permeability tests after different wheel load applications were performed on the rutted OGFC slabs using a falling head permeameter developed in the authors' previous study. The relationships between permeability loss and rutting depth as well as dynamic stability were developed based on the eight OGFC mixtures' test results. The thickness effects of the single-layer and the two-layer OGFC slabs were also discussed in terms of deformation-related clogging and the rutting-induced permeability loss. Results showed that the permeability coefficient decreases linearly with an increasing rutting depth of the OGFC mixtures. Rutting depth was recommended as a design index to control permeability loss of the OGFC mixture rather than the dynamic stability. Permeability loss due to deformation-related clogging can be effectively reduced by using a thicker single-layer OGFC or two-layer OGFC.