Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

A uniform white-noise model for fixed-point roundoff errors in digital systems

Fixed-point roundoff noise in digital implementation of linear systems arises due to overflow, quantization of coefficients and input signals, and arithmetical errors. In uniform white-noise models, the last two types of roundoff errors are regarded as uniformly distributed independent random vectors on cubes of suitable size. For input signal quantization errors, the heuristic model is justified by a quantization theorem, which cannot be directly applied to arithmetical errors due to the complicated input-dependence of errors. The complete uniform white-noise model is shown to be valid in the sense of weak convergence of probabilistic measures as the lattice step tends to zero if the matrices of realization of the system in the state space satisfy certain nonresonance conditions and the finite-dimensional distributions of the input signal are absolutely continuous.

The highly rearranged mitochondrial genome of the plague thrips, Thrips imaginis (Insecta : thysanoptera): Convergence of two novel gene boundaries and an extraordinary arrangement of rRNA genes

To help understand the mechanisms of gene rearrangement in the mitochondrial (mt) genomes of hemipteroid insects, we sequenced the mt genome of the plague thrips, Thrips imaginis (Thysanoptera). This genome is circular, 15,407 by long, and has many unusual features, including (1) rRNA genes inverted and distant from one another, (2) an extra gene for tRNA-Ser, (3) a tRNA-Val lacking a D-arm, (4) two pseudo-tRNA genes, (5) duplicate control regions, and (6) translocations and/or inversions of 24 of the 37 genes. The mechanism of rRNA gene transcription in T. imaginis may be different from that of other arthropods since the two rRNA genes have inverted and are distant from one another. Further, the rRNA genes are not adjacent or even close to either of the two control regions. Tandem duplication and deletion is a plausible model for the evolution of duplicate control regions and for the gene translocations, but intramitochondrial recombination may account for the gene inversions in T. imaginis. All the 18 genes between control regions #1 and #2 have translocated and/or inverted, whereas only six of the 20 genes outside this region have translocated and/or inverted. Moreover, the extra tRNA gene and the two pseudo-tRNA genes are either in this region or immediately adjacent to one of the control regions. These observations suggest that tandem duplication and deletion may be facilitated by the duplicate control regions and may have occurred a number of times in the lineage leading to T. imaginis. T. imaginis shares two novel gene boundaries with a lepidopsocid species from another order of hemipteroid insects, the Psocoptera. The evidence available suggests that these shared gene boundaries evolved by convergence and thus are not informative for the interordinal phylogeny of hemipteroid insects. We discuss the potential of hemipteroid insects as a model system for studies of the evolution of animal rut genomes and outline some fundamental questions that may be addressed with this system.

Speeding up the EM algorithm for mixture model-based segmentation of magnetic resonance images

Mixture models implemented via the expectation-maximization (EM) algorithm are being increasingly used in a wide range of problems in pattern recognition such as image segmentation. However, the EM algorithm requires considerable computational time in its application to huge data sets such as a three-dimensional magnetic resonance (MR) image of over 10 million voxels. Recently, it was shown that a sparse, incremental version of the EM algorithm could improve its rate of convergence. In this paper, we show how this modified EM algorithm can be speeded up further by adopting a multiresolution kd-tree structure in performing the E-step. The proposed algorithm outperforms some other variants of the EM algorithm for segmenting MR images of the human brain. (C) 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Subgeometric rates of convergence for a class of continuous-time Markov process

Let (Phi(t))(t is an element of R+) be a Harris ergodic continuous-time Markov process on a general state space, with invariant probability measure pi. We investigate the rates of convergence of the transition function P-t(x, (.)) to pi; specifically, we find conditions under which r(t) vertical bar vertical bar P-t (x, (.)) - pi vertical bar vertical bar -> 0 as t -> infinity, for suitable subgeometric rate functions r(t), where vertical bar vertical bar - vertical bar vertical bar denotes the usual total variation norm for a signed measure. We derive sufficient conditions for the convergence to hold, in terms of the existence of suitable points on which the first hitting time moments are bounded. In particular, for stochastically ordered Markov processes, explicit bounds on subgeometric rates of convergence are obtained. These results are illustrated in several examples.

Income Ranking and convergence with physical and human capital and income equality

This paper presents an overlapping generations model with physical and human capital and income inequality. It shows that inequality impedes output growth by directly harming capital accumulation and indirectly raising the ratio of physical to human capital. The convergence speed of output growth equals the lower of the convergence speeds of the relative capital ratio and inequality, and varies with initial states. Among economies with the same balanced growth rate but different initial income levels, the ranking of income can switch in favor of those starting from low inequality and a low ratio of physical to human capital, particularly if the growth rate converges slowly. (C) 2004 Elsevier B.V. All rights reserved.

Searching for convergence in phylogenetic Markov chain Monte Carlo

Markov chain Monte Carlo (MCMC) is a methodology that is gaining widespread use in the phylogenetics community and is central to phylogenetic software packages such as MrBayes. An important issue for users of MCMC methods is how to select appropriate values for adjustable parameters such as the length of the Markov chain or chains, the sampling density, the proposal mechanism, and, if Metropolis-coupled MCMC is being used, the number of heated chains and their temperatures. Although some parameter settings have been examined in detail in the literature, others are frequently chosen with more regard to computational time or personal experience with other data sets. Such choices may lead to inadequate sampling of tree space or an inefficient use of computational resources. We performed a detailed study of convergence and mixing for 70 randomly selected, putatively orthologous protein sets with different sizes and taxonomic compositions. Replicated runs from multiple random starting points permit a more rigorous assessment of convergence, and we developed two novel statistics, delta and epsilon, for this purpose. Although likelihood values invariably stabilized quickly, adequate sampling of the posterior distribution of tree topologies took considerably longer. Our results suggest that multimodality is common for data sets with 30 or more taxa and that this results in slow convergence and mixing. However, we also found that the pragmatic approach of combining data from several short, replicated runs into a metachain to estimate bipartition posterior probabilities provided good approximations, and that such estimates were no worse in approximating a reference posterior distribution than those obtained using a single long run of the same length as the metachain. Precision appears to be best when heated Markov chains have low temperatures, whereas chains with high temperatures appear to sample trees with high posterior probabilities only rarely. [Bayesian phylogenetic inference; heating parameter; Markov chain Monte Carlo; replicated chains.]

Making Fedora easier to implement with Fez: A free open source content model and workflow management front-end to Fedora

The University of Queensland, Australia has developed Fez, a world-leading user-interface and management system for Fedora-based institutional repositories, which bridges the gap between a repository and users. Christiaan Kortekaas, Andrew Bennett and Keith Webster will review this open source software that gives institutions the power to create a comprehensive repository solution without the hassle..

A modified discrete random pore model allowing for different initial surface reactivity

We investigate here a modification of the discrete random pore model [Bhatia SK, Vartak BJ, Carbon 1996;34:1383], by including an additional rate constant which takes into account the different reactivity of the initial pore surface having attached functional groups and hydrogens, relative to the subsequently exposed surface. It is observed that the relative initial reactivity has a significant effect on the conversion and structural evolution, underscoring the importance of initial surface chemistry. The model is tested against experimental data on chemically controlled char oxidation and steam gasification at various temperatures. It is seen that the variations of the reaction rate and surface area with conversion are better represented by the present approach than earlier random pore models. The results clearly indicate the improvement of model predictions in the low conversion region, where the effect of the initially attached functional groups and hydrogens is more significant, particularly for char oxidation. It is also seen that, for the data examined, the initial surface chemistry is less important for steam gasification as compared to the oxidation reaction. Further development of the approach must also incorporate the dynamics of surface complexation, which is not considered here.

Adsorption in mesopores: A molecular-continuum model with application to MCM-41

The classical model of surface layering followed by capillary condensation during adsorption in mesopores, is modified here by consideration of the adsorbate solid interaction potential. The new theory accurately predicts the capillary coexistence curve as well as pore criticality, matching that predicted by density functional theory. The model also satisfactorily predicts the isotherm for nitrogen adsorption at 77.4 K on MCM-41 material of various pore sizes, synthesized and characterized in our laboratory, including the multilayer region, using only data on the variation of condensation pressures with pore diameter. The results indicate a minimum mesopore diameter for the surface layering model to hold as 14.1 Å, below which size micropore filling must occur, and a minimum pore diameter for mechanical stability of the hemispherical meniscus during desorption as 34.2 Å. For pores in-between these two sizes reversible condensation is predicted to occur, in accord with the experimental data for nitrogen adsorption on MCM-41 at 77.4 K.

A Nonstationary Model of Newborn EEG

The detection of seizure in the newborn is a critical aspect of neurological research. Current automatic detection techniques are difficult to assess due to the problems associated with acquiring and labelling newborn electroencephalogram (EEG) data. A realistic model for newborn EEG would allow confident development, assessment and comparison of these detection techniques. This paper presents a model for newborn EEG that accounts for its self-similar and non-stationary nature. The model consists of background and seizure sub-models. The newborn EEG background model is based on the short-time power spectrum with a time-varying power law. The relationship between the fractal dimension and the power law of a power spectrum is utilized for accurate estimation of the short-time power law exponent. The newborn EEG seizure model is based on a well-known time-frequency signal model. This model addresses all significant time-frequency characteristics of newborn EEG seizure which include; multiple components or harmonics, piecewise linear instantaneous frequency laws and harmonic amplitude modulation. Estimates of the parameters of both models are shown to be random and are modelled using the data from a total of 500 background epochs and 204 seizure epochs. The newborn EEG background and seizure models are validated against real newborn EEG data using the correlation coefficient. The results show that the output of the proposed models has a higher correlation with real newborn EEG than currently accepted models (a 10% and 38% improvement for background and seizure models, respectively).

Parliament House Competition Model, Canberra

View of model for competition entry.

Parliament House Competition Model, Darwin

View of model for competition entry.

Parliament House Competition Model, Darwin

View of model for competition entry.

Entanglement between a qubit and the environment in the spin-boson model

The quantitative description of the quantum entanglement between a qubit and its environment is considered. Specifically, for the ground state of the spin-boson model, the entropy of entanglement of the spin is calculated as a function of α, the strength of the ohmic coupling to the environment, and ɛ, the level asymmetry. This is done by a numerical renormalization group treatment of the related anisotropic Kondo model. For ɛ=0, the entanglement increases monotonically with α, until it becomes maximal for α→1-. For fixed ɛ>0, the entanglement is a maximum as a function of α for a value, α=αM