Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

Generalization by symbolic abstraction in cascaded recurrent networks

Generalization performance in recurrent neural networks is enhanced by cascading several networks. By discretizing abstractions induced in one network, other networks can operate on a coarse symbolic level with increased performance on sparse and structural prediction tasks. The level of systematicity exhibited by the cascade of recurrent networks is assessed on the basis of three language domains. (C) 2004 Elsevier B.V. All rights reserved.

An implementation of genetic algorithms as a basis for a trading system on the foreign exchange market

Foreign exchange trading has emerged in recent times as a significant activity in many countries. As with most forms of trading, the activity is influenced by many random parameters so that the creation of a system that effectively emulates the trading process is very helpful. In this paper, we try to create such a system with a genetic algorithm engine to emulate trader behaviour on the foreign exchange market and to find the most profitable trading strategy.

Genetic basis of inosine triphosphate pyrophosphohydrolase (ITPA) deficiency

Inosine triphosphate pyrophosphohydrolase (ITPase) deficiency is a common inherited condition characterized by the abnormal accumulation of inosine triphosphate (ITP) in erythrocytes. The genetic basis and pathological consequences of ITPase deficiency are unknown. We have characterized the genomic structure of the ITPA gene, showing that it has eight exons. Five single nucleotide polymorphisms were identified, three silent (138GMA, 561GMA, 708GMA) and two associated with ITPase deficiency (94CMA, IVS2+21AMC). Homozygotes for the 94CMA missense mutation (Pro32 to Thr) had zero erythrocyte ITPase activity, whereas 94CMA heterozygotes averaged 22.5% of the control mean, a level of activity consistent with impaired subunit association of a dimeric enzyme. ITPase activity of IVS2+21AMC homozygotes averaged 60% of the control mean. In order to explore further the relationship between mutations and enzyme activity, we examined the association between genotype and ITPase activity in 100 healthy controls. Ten subjects were heterozygous for 94CMA (allele frequency: 0.06), 24 were heterozygotes for IVS2+21AMC (allele frequency: 0.13) and two were compound heterozygous for these mutations. The activities of IVS2+21AMC heterozygotes and 94CMA/IVS2+21AMC compound heterozygotes were 60% and 10%, respectively, of the normal control mean, suggesting that the intron mutation affects enzyme activity. In all cases when ITPase activity was below the normal range, one or both mutations were found. The ITPA genotype did not correspond to any identifiable red cell phenotype. A possible relationship between ITPase deficiency and increased drug toxicity of purine analogue drugs is proposed.

Biomechanical consequences of impairment: A taxonomically valid basis for classification in a unified disability athletics system

Developing a unified classification system to replace four of the systems currently used in disability athletics (i.e., track and field) has been widely advocated. The diverse impairments to be included in a unified system require severed assessment methods, results of which cannot be meaningfully compared. Therefore, the taxonomic basis of current classification systems is invalid in a unified system. Biomechanical analysis establishes that force, a vector described in terms of magnitude and direction, is a key determinant of success in all athletic disciplines. It is posited that all impairments to be included in a unified system may be classified as either force magnitude impairments (FMI) or force control impairments (FCI). This framework would provide a valid taxonomic basis for a unified system, creating the opportunity to decrease the number of classes and enhance the viability of disability athletics.

The Scientific Basis for High-Intensity Interval Training: Optimising Training Programmes and Maximising Performance in Highly Trained Endurance Athletes

While the physiological adaptations that occur following endurance training in previously sedentary and recreationally active individuals are relatively well understood, the adaptations to training in already highly trained endurance athletes remain unclear. While significant improvements in endurance performance and corresponding physiological markers are evident following submaximal endurance training in sedentary and recreationally active groups, an additional increase in submaximal training (i.e. volume) in highly trained individuals does not appear to further enhance either endurance performance or associated physiological variables [e.g. peak oxygen uptake (V-dot O2peak), oxidative enzyme activity]. It seems that, for athletes who are already trained, improvements in endurance performance can be achieved only through high-intensity interval training (HIT). The limited research which has examined changes in muscle enzyme activity in highly trained athletes, following HIT, has revealed no change in oxidative or glycolytic enzyme activity, despite significant improvements in endurance performance (p < 0.05). Instead, an increase in skeletal muscle buffering capacity may be one mechanism responsible for an improvement in endurance performance. Changes in plasma volume, stroke volume, as well as muscle cation pumps, myoglobin, capillary density and fibre type characteristics have yet to be investigated in response to HIT with the highly trained athlete. Information relating to HIT programme optimisation in endurance athletes is also very sparse. Preliminary work using the velocity at which V-dot O2max is achieved (Vmax) as the interval intensity, and fractions (50 to 75%) of the time to exhaustion at Vmax (Tmax) as the interval duration has been successful in eliciting improvements in performance in long-distance runners. However, Vmax and Tmax have not been used with cyclists. Instead, HIT programme optimisation research in cyclists has revealed that repeated supramaximal sprinting may be equally effective as more traditional HIT programmes for eliciting improvements in endurance performance. Further examination of the biochemical and physiological adaptations which accompany different HIT programmes, as well as investigation into the optimal HIT programme for eliciting performance enhancements in highly trained athletes is required.

Selection bias in gene extraction on the basis of microarray gene-expression data

In the context of cancer diagnosis and treatment, we consider the problem of constructing an accurate prediction rule on the basis of a relatively small number of tumor tissue samples of known type containing the expression data on very many (possibly thousands) genes. Recently, results have been presented in the literature suggesting that it is possible to construct a prediction rule from only a few genes such that it has a negligible prediction error rate. However, in these results the test error or the leave-one-out cross-validated error is calculated without allowance for the selection bias. There is no allowance because the rule is either tested on tissue samples that were used in the first instance to select the genes being used in the rule or because the cross-validation of the rule is not external to the selection process; that is, gene selection is not performed in training the rule at each stage of the cross-validation process. We describe how in practice the selection bias can be assessed and corrected for by either performing a cross-validation or applying the bootstrap external to the selection process. We recommend using 10-fold rather than leave-one-out cross-validation, and concerning the bootstrap, we suggest using the so-called. 632+ bootstrap error estimate designed to handle overfitted prediction rules. Using two published data sets, we demonstrate that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes.

A fast cross-entropy method for estimating buffer overflows in queuing networks

In this paper, we propose a fast adaptive importance sampling method for the efficient simulation of buffer overflow probabilities in queueing networks. The method comprises three stages. First, we estimate the minimum cross-entropy tilting parameter for a small buffer level; next, we use this as a starting value for the estimation of the optimal tilting parameter for the actual (large) buffer level. Finally, the tilting parameter just found is used to estimate the overflow probability of interest. We study various properties of the method in more detail for the M/M/1 queue and conjecture that similar properties also hold for quite general queueing networks. Numerical results support this conjecture and demonstrate the high efficiency of the proposed algorithm.

A hybrid planning method for transmission networks in a deregulated environment

The reconstruction of power industries has brought fundamental changes to both power system operation and planning. This paper presents a new planning method using multi-objective optimization (MOOP) technique, as well as human knowledge, to expand the transmission network in open access schemes. The method starts with a candidate pool of feasible expansion plans. Consequent selection of the best candidates is carried out through a MOOP approach, of which multiple objectives are tackled simultaneously, aiming at integrating the market operation and planning as one unified process in context of deregulated system. Human knowledge has been applied in both stages to ensure the selection with practical engineering and management concerns. The expansion plan from MOOP is assessed by reliability criteria before it is finalized. The proposed method has been tested with the IEEE 14-bus system and relevant analyses and discussions have been presented.

Hypercapitalism: New media, language, and social perceptions of value

Every day trillions of dollars circulate the globe in a digital data space and new forms of property and ownership emerge. Massive corporate entities with a global reach are formed and disappear with breathtaking speed, making and breaking personal fortunes the size of which defy imagination. Fictitious commodities abound. The genomes of entire nations have become corporately owned. Relationships have become the overt basis of economic wealth and political power. Hypercapitalism explores the problems of understanding this emergent form of global political economic organization by focusing on the internal relations between language, new media networks, and social perceptions of value. Taking an historical approach informed by Marx, Phil Graham draws upon writings in political economy, media studies, sociolinguistics, anthropology, and critical social science to understand the development, roots, and trajectory of the global system in which every possible aspect of human existence, including imagined futures, has become a commodity form.

Protein fold recognition without Boltzmann statistics or explicit physical basis

We present a fast method for finding optimal parameters for a low-resolution (threading) force field intended to distinguish correct from incorrect folds for a given protein sequence. In contrast to other methods, the parameterization uses information from >10(7) misfolded structures as well as a set of native sequence-structure pairs. In addition to testing the resulting force field's performance on the protein sequence threading problem, results are shown that characterize the number of parameters necessary for effective structure recognition.

Matrix elements of U(2n) generators in a multishell spin-orbit basis. I. The del-operator MEs in a two-shell composite Gelfand-Paldus basis

This is the first in a series of three articles which aimed to derive the matrix elements of the U(2n) generators in a multishell spin-orbit basis. This is a basis appropriate to many-electron systems which have a natural partitioning of the orbital space and where also spin-dependent terms are included in the Hamiltonian. The method is based on a new spin-dependent unitary group approach to the many-electron correlation problem due to Gould and Paldus [M. D. Gould and J. Paldus, J. Chem. Phys. 92, 7394, (1990)]. In this approach, the matrix elements of the U(2n) generators in the U(n) x U(2)-adapted electronic Gelfand basis are determined by the matrix elements of a single Ll(n) adjoint tensor operator called the del-operator, denoted by Delta(j)(i) (1 less than or equal to i, j less than or equal to n). Delta or del is a polynomial of degree two in the U(n) matrix E = [E-j(i)]. The approach of Gould and Paldus is based on the transformation properties of the U(2n) generators as an adjoint tensor operator of U(n) x U(2) and application of the Wigner-Eckart theorem. Hence, to generalize this approach, we need to obtain formulas for the complete set of adjoint coupling coefficients for the two-shell composite Gelfand-Paldus basis. The nonzero shift coefficients are uniquely determined and may he evaluated by the methods of Gould et al. [see the above reference]. In this article, we define zero-shift adjoint coupling coefficients for the two-shell composite Gelfand-Paldus basis which are appropriate to the many-electron problem. By definition, these are proportional to the corresponding two-shell del-operator matrix elements, and it is shown that the Racah factorization lemma applies. Formulas for these coefficients are then obtained by application of the Racah factorization lemma. The zero-shift adjoint reduced Wigner coefficients required for this procedure are evaluated first. All these coefficients are needed later for the multishell case, which leads directly to the two-shell del-operator matrix elements. Finally, we discuss an application to charge and spin densities in a two-shell molecular system. (C) 1998 John Wiley & Sons.