Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

Study of hexane adsorption in nanoporous MCM-41 silica

We study here the adsorption of hexane on nanoporous MCM-41 silica at 303, 313, and 323 K, for various pore diameters between 2.40 and 4.24 nm. Adsorption equilibria, measured thermogravimetrically, show that all the isotherms, that are somewhat akin to those of type V, exhibit remarkably sharp capillary adsorption phase transition steps and are reversible. The position of the phase transition step gradually shifts from low to high relative pressure with an increase in the temperature as well as the pore sizes. The isosteric heats of adsorption derived from the equilibrium information using the Clapeyron equation reveal a gradual decrease with increasing adsorbed amount because of the surface heterogeneity but approach a constant value near the phase transition. A decrease in the pore size results in an increase in the isosteric heat of adsorption because of the increased dispersion forces. A simple strategy, based on the Broekhoff and De Boer adsorption theory, successfully interprets the hexane adsorption isotherms for the different pore size MCM-41 samples. The parameters of an empirical expression, used to represent the potential of interaction between the adsorbate and adsorbent, are obtained by fitting the monolayer region prior to capillary condensation and the experimental phase transition simultaneously, for some pore sizes. Subsequently, the parameters are used to predict the adsorption isotherm on other pore size samples, which showed good agreement with experimental data.

Structure and thermal stability of Langmuir-Blodgett films of barium arachidate

The structures of multilayer Langmuir-Blodgett films of barium arachidate before and after heat treatment have been investigated using both atomic force microscopy (AFM) and grazing incidence synchrotron X-ray diffraction (GIXD). AFM gave information on surface morphology at molecular resolution while GIXD provided quantitative details of the lattice structures of the films with their crystal symmetries and lattice constants. As-prepared films contained three coexisting structures: two triclinic structures with the molecularchains tilted by about 20degrees from the film normal and with 3 x 1 or 2 x 2 super-lattice features arising from height modulation of the molecules in the films; a rectangular structure with molecules perpendicular to the film surface. Of these, the 3 x 1 structure is dominant with a loose correlation between the bilayers. In the film plane both superstructures are commensurate with the local structures, having different oblique symmetries. The lattice constants for the 3 x 1 structure are a(s) = 3a = 13.86 Angstrom, b(s) = b = 4.31 Angstrom and gamma(s) = gamma = 82.7degrees; for the 2 x 2 structure a(s) = 2a = 16.54 Angstrom, b(s) = 2b = 9.67 Angstrom, gamma(s) = gamma = 88degrees. For the rectangular structure the lattice constants are a = 7.39 Angstrom, b = 4.96 Angstrom and gamma = 90degrees. After annealing, the 2 x 2 and rectangular structures were not observed, while the 3 x 1 structure had developed over the entire film. For the annealed films the correlation length in the film plane is about twice that in the unheated films, and in the out-of-plane direction covers two bilayers. The above lattice parameters, determined by GIXD, differed significantly from the values obtained by AFM, due possibly to distortion of the films by the scanning action of the AFM tip. (C) 2004 Published by Elsevier B.V.

The structure of ultrathin Langmuir-Blodgett films of cadmium behenate

Grazing incidence x-ray-diffraction investigations of the structures of Langmuir-Blodgett films of cadmium behenate with 1, 2, 3, 5, and 21 monolayers are reported. The single monolayer film, deposited on a hydrophilic substrate, showed a hexagonal structure, whereas the bilayer film, deposited on a hydrophobic substrate, had a rectangular structure with herringbone orientation of the acyl chains. With multilayer films formed on a hydrophilic substrate, it was possible to detect that the hexagonal structure of the first layer was retained when additional layers were deposited and that the additional layers had the same rectangular structure as the bilayer. (c) 2005 American Institute of Physics.

Interfacial interactions and structure of polyurethane intercalated nanocomposite

Understanding the interfacial interactions between the nanofiller and polymer matrix is important to improve the design and manufacture of polymer nanocomposites. This paper reports a molecular dynamic Study on the interfacial interactions and structure of a clay-based polyurethane intercalated nanocomposite. The results show that the intercalation of surfactant (i.e. dioctadecyldlmethyl ammonium) and polyurethane (PU) into the nanoconfined gallery of clay leads to the multilayer structure for both surfactant and PU, and the absence of phase separation for PU chains. Such structural characteristics are attributed to the result of competitive interactions among the surfactant, PU and the clay surface, including van der Waals, electrostatic and hydrogen bonding.

Application of a generalized thermodynamic approach to characterize mesoporous materials

Equilibrium adsorption and desorption in mesoporous adsorbents is considered on the basis of rigorous thermodynamic analysis, in which the curvature-dependent solid-fluid potential and the compressibility of the adsorbed phase are accounted for. The compressibility of the adsorbed phase is considered for the first time in the literature in the framework of a rigorous thermodynamic approach. Our model is a further development of continuum thermodynamic approaches proposed by Derjaguin and Broekhoff and de Boer, and it is based on a reference isotherm of a non-porous material having the same chemical structure as that of the pore wall. In this improved thermodynamic model, we incorporated a prescription for transforming the solid-fluid potential exerted by the flat reference surface to the potential inside cylindrical and spherical pores. We relax the assumption that the adsorbed film density is constant and equal to that of the saturated liquid. Instead, the density of the adsorbed fluid is allowed to vary over the adsorbed film thickness and is calculated by an equation of state. As a result, the model is capable to describe the adsorption-desorption reversibility in cylindrical pores having diameter less than 2 nm. The generalized thermodynamic model may be applied to the pore size characterization of mesoporous materials instead of much more time-consuming molecular approaches. (c) 2005 Elsevier B.V. All rights reserved.

Adsorption of benzene on graphitized thermal carbon black: Reduction of the quadrupole moment in the adsorbed phase

The performance of intermolecular potential models on the adsorption of benzene on graphitized thermal carbon black at various temperatures is investigated. Two models contain only dispersive sites, whereas the other two models account explicitly for the dispersive and electrostatic sites. Using numerous data in the literature on benzene adsorption on graphitized thermal carbon black at various temperatures, we have found that the effect of surface mediation on interaction between adsorbed benzene molecules must be accounted for to describe correctly the adsorption isotherm as well as the isosteric heat. Among the two models with partial charges tested, the WSKS model of Wick et at. I that has only six dispersive sites and three discrete partial charges is better than the very expensive all-atom model of Jorgensen and Severance.(2) Adsorbed benzene molecules on graphitized thermal carbon black have a complex orientation with respect to distance from the surface and also with respect to loading. At low loadings, they adopt the parallel configuration relative to the graphene surface, whereas at higher loadings (still less than monolayer coverage) some molecules adopt a slant orientation to maximize the fluid-fluid interaction. For loadings in the multilayer region, the orientation of molecules in the first layer is influenced by the presence of molecules in the second layer. The data that are used in this article come from the work of Isirikyan and Kiselev,(3) Pierotti and Smallwood,(4) Pierce and Ewing,(5) Belyakova, Kiselev, and Kovaleva,(6) and Carrott et al.(7)

Biological polarized light reflectors in stomatopod crustaceans

Body parts that can reflect highly polarized light have been found in several species of stomatopod crustaceans (mantis shrimps). These polarized light reflectors can be grossly divided into two major types. The first type, usually red or pink in color to the human visual system, is located within an animal’s cuticle. Reflectors of the second type, showing iridescent blue, are located beneath the exoskeleton and thus are unaffected by the molt cycle. We used reflection spectropolarimetry and transmission electron microscopy (TEM) to study the reflective properties and the structures that reflect highly polarized light in stomatopods. For the first type of reflector, the degree of polarization usually changes dramatically, from less than 20% to over 70%, with a change in viewing angle. TEM examination indicates that the polarization reflection is generated by multilayer thin-film interference. The second type of reflector, the blue colored ones, reflects highly polarized light to all viewing angles. However, these reflectors show a slight chromatic change with different viewing angles. TEM sections have revealed that streams of oval-shaped vesicles might be responsible for the production of the polarized light reflection. In all the reflectors we have examined so far, the reflected light is always maximally polarized at around 500 nm, which is close to the wavelength best transmitted by sea water. This suggests that the polarized light reflectors found in stomatopods are well adapted to the underwater environment. We also found that most reflectors produce polarized light with a horizontal e-vector. How these polarized light reflectors are used in stomatopod signaling remains unknown.

Neural networks

An overview of neural networks, covering multilayer perceptrons, radial basis functions, constructive algorithms, Kohonen and K-means unupervised algorithms, RAMnets, first and second order training methods, and Bayesian regularisation methods.

Capacity of the upstart algorithm

The storage capacity of multilayer networks with overlapping receptive fields is investigated for a constructive algorithm within a one-step replica symmetry breaking (RSB) treatment. We find that the storage capacity increases logarithmically with the number of hidden units K without saturating the Mitchison-Durbin bound. The slope of the logarithmic increase decays exponentionally with the stability with which the patterns have been stored.

Classification of oximetry signals using Bayesian neural networks to assist in the detection of obstructive sleep apnoea syndrome

In the present study, multilayer perceptron (MLP) neural networks were applied to help in the diagnosis of obstructive sleep apnoea syndrome (OSAS). Oxygen saturation (SaO2) recordings from nocturnal pulse oximetry were used for this purpose. We performed time and spectral analysis of these signals to extract 14 features related to OSAS. The performance of two different MLP classifiers was compared: maximum likelihood (ML) and Bayesian (BY) MLP networks. A total of 187 subjects suspected of suffering from OSAS took part in the study. Their SaO2 signals were divided into a training set with 74 recordings and a test set with 113 recordings. BY-MLP networks achieved the best performance on the test set with 85.58% accuracy (87.76% sensitivity and 82.39% specificity). These results were substantially better than those provided by ML-MLP networks, which were affected by overfitting and achieved an accuracy of 76.81% (86.42% sensitivity and 62.83% specificity). Our results suggest that the Bayesian framework is preferred to implement our MLP classifiers. The proposed BY-MLP networks could be used for early OSAS detection. They could contribute to overcome the difficulties of nocturnal polysomnography (PSG) and thus reduce the demand for these studies.

Noise, regularizers, and unrealizable scenarios in online learning from restricted training sets

We study the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is carried out using the dynamical replica method aimed at obtaining a closed set of coupled equations for a set of macroscopic variables from which both training and generalization errors can be calculated. We focus on scenarios whereby training examples are corrupted by additive Gaussian output noise and regularizers are introduced to improve the network performance. The dependence of the dynamics on the noise level, with and without regularizers, is examined, as well as that of the asymptotic values obtained for both training and generalization errors. We also demonstrate the ability of the method to approximate the learning dynamics in structurally unrealizable scenarios. The theoretical results show good agreement with those obtained by computer simulations.

The theory of on-line learning: a statistical physics approach

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

DVMS 1.5: A user manual (the data visualization and modeling system)

The data available during the drug discovery process is vast in amount and diverse in nature. To gain useful information from such data, an effective visualisation tool is required. To provide better visualisation facilities to the domain experts (screening scientist, biologist, chemist, etc.),we developed a software which is based on recently developed principled visualisation algorithms such as Generative Topographic Mapping (GTM) and Hierarchical Generative Topographic Mapping (HGTM). The software also supports conventional visualisation techniques such as Principal Component Analysis, NeuroScale, PhiVis, and Locally Linear Embedding (LLE). The software also provides global and local regression facilities . It supports regression algorithms such as Multilayer Perceptron (MLP), Radial Basis Functions network (RBF), Generalised Linear Models (GLM), Mixture of Experts (MoE), and newly developed Guided Mixture of Experts (GME). This user manual gives an overview of the purpose of the software tool, highlights some of the issues to be taken care while creating a new model, and provides information about how to install & use the tool. The user manual does not require the readers to have familiarity with the algorithms it implements. Basic computing skills are enough to operate the software.

Techniques to improve forecasting models: applications to energy demand and price

This thesis is a study of three techniques to improve performance of some standard fore-casting models, application to the energy demand and prices. We focus on forecasting demand and price one-day ahead. First, the wavelet transform was used as a pre-processing procedure with two approaches: multicomponent-forecasts and direct-forecasts. We have empirically compared these approaches and found that the former consistently outperformed the latter. Second, adaptive models were introduced to continuously update model parameters in the testing period by combining ?lters with standard forecasting methods. Among these adaptive models, the adaptive LR-GARCH model was proposed for the fi?rst time in the thesis. Third, with regard to noise distributions of the dependent variables in the forecasting models, we used either Gaussian or Student-t distributions. This thesis proposed a novel algorithm to infer parameters of Student-t noise models. The method is an extension of earlier work for models that are linear in parameters to the non-linear multilayer perceptron. Therefore, the proposed method broadens the range of models that can use a Student-t noise distribution. Because these techniques cannot stand alone, they must be combined with prediction models to improve their performance. We combined these techniques with some standard forecasting models: multilayer perceptron, radial basis functions, linear regression, and linear regression with GARCH. These techniques and forecasting models were applied to two datasets from the UK energy markets: daily electricity demand (which is stationary) and gas forward prices (non-stationary). The results showed that these techniques provided good improvement to prediction performance.