Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass Classification

The expectation-maximization (EM) algorithm has been of considerable interest in recent years as the basis for various algorithms in application areas of neural networks such as pattern recognition. However, there exists some misconceptions concerning its application to neural networks. In this paper, we clarify these misconceptions and consider how the EM algorithm can be adopted to train multilayer perceptron (MLP) and mixture of experts (ME) networks in applications to multiclass classification. We identify some situations where the application of the EM algorithm to train MLP networks may be of limited value and discuss some ways of handling the difficulties. For ME networks, it is reported in the literature that networks trained by the EM algorithm using iteratively reweighted least squares (IRLS) algorithm in the inner loop of the M-step, often performed poorly in multiclass classification. However, we found that the convergence of the IRLS algorithm is stable and that the log likelihood is monotonic increasing when a learning rate smaller than one is adopted. Also, we propose the use of an expectation-conditional maximization (ECM) algorithm to train ME networks. Its performance is demonstrated to be superior to the IRLS algorithm on some simulated and real data sets.

An algorithm to predict 3 ' intron splice sites in Plasmodium falciparum genomic sequences

A new algorithm, PfAGSS, for predicting 3' splice sites in Plasmodium falciparum genomic sequences is described. Application of this program to the published P. falciparum chromosome 2 and 3 data suggests that existing programs result in a high error rate in assigning 3' intron boundaries. (C) 2001 Elsevier Science B.V. All rights reserved.

An EM-based Semi-Parametric Mixture Model Approach to the Regression Analysis of Competing-Risks Data

We consider a mixture model approach to the regression analysis of competing-risks data. Attention is focused on inference concerning the effects of factors on both the probability of occurrence and the hazard rate conditional on each of the failure types. These two quantities are specified in the mixture model using the logistic model and the proportional hazards model, respectively. We propose a semi-parametric mixture method to estimate the logistic and regression coefficients jointly, whereby the component-baseline hazard functions are completely unspecified. Estimation is based on maximum likelihood on the basis of the full likelihood, implemented via an expectation-conditional maximization (ECM) algorithm. Simulation studies are performed to compare the performance of the proposed semi-parametric method with a fully parametric mixture approach. The results show that when the component-baseline hazard is monotonic increasing, the semi-parametric and fully parametric mixture approaches are comparable for mildly and moderately censored samples. When the component-baseline hazard is not monotonic increasing, the semi-parametric method consistently provides less biased estimates than a fully parametric approach and is comparable in efficiency in the estimation of the parameters for all levels of censoring. The methods are illustrated using a real data set of prostate cancer patients treated with different dosages of the drug diethylstilbestrol. Copyright (C) 2003 John Wiley Sons, Ltd.

Speeding up the EM algorithm for mixture model-based segmentation of magnetic resonance images

Mixture models implemented via the expectation-maximization (EM) algorithm are being increasingly used in a wide range of problems in pattern recognition such as image segmentation. However, the EM algorithm requires considerable computational time in its application to huge data sets such as a three-dimensional magnetic resonance (MR) image of over 10 million voxels. Recently, it was shown that a sparse, incremental version of the EM algorithm could improve its rate of convergence. In this paper, we show how this modified EM algorithm can be speeded up further by adopting a multiresolution kd-tree structure in performing the E-step. The proposed algorithm outperforms some other variants of the EM algorithm for segmenting MR images of the human brain. (C) 2004 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Convergence analysis of a discrete-time single-unit gradient ICA algorithm

We revisit the one-unit gradient ICA algorithm derived from the kurtosis function. By carefully studying properties of the stationary points of the discrete-time one-unit gradient ICA algorithm, with suitable condition on the learning rate, convergence can be proved. The condition on the learning rate helps alleviate the guesswork that accompanies the problem of choosing suitable learning rate in practical computation. These results may be useful to extract independent source signals on-line.

The parameterless self-organizing map algorithm

The parameterless self-organizing map (PLSOM) is a new neural network algorithm based on the self-organizing map (SOM). It eliminates the need for a learning rate and annealing schemes for learning rate and neighborhood size. We discuss the relative performance of the PLSOM and the SOM and demonstrate some tasks in which the SOM fails but the PLSOM performs satisfactory. Finally we discuss some example applications of the PLSOM and present a proof of ordering under certain limited conditions.

Lingual kinematic strategies used to increase speech rate: Comparisons between younger and older adults

The primary objective of this study was to assess the lingual kinematic strategies used by younger and older adults to increase rate of speech. It was hypothesised that the strategies used by the older adults would differ from the young adults either as a direct result of, or in response to a need to compensate for, age-related changes in the tongue. Electromagnetic articulography was used to examine the tongue movements of eight young (M526.7 years) and eight older (M567.1 years) females during repetitions of /ta/ and /ka/ at a controlled moderate rate and then as fast as possible. The younger and older adults were found to significantly reduce consonant durations and increase syllable repetition rate by similar proportions. To achieve these reduced durations both groups appeared to use the same strategy, that of reducing the distances travelled by the tongue. Further comparisons at each rate, however, suggested a speed-accuracy trade-off and increased speech monitoring in the older adults. The results may assist in differentiating articulatory changes associated with normal aging from pathological changes found in disorders that affect the older population.

Blood pressure and heart rate during tonic immobility in the black tipped reef shark, Carcharhinus melanoptera

Tonic immobility was induced in black tipped reef sharks (Carcharhinus melanoptera) and heart rate and ventral aortic blood pressure recorded. Without branchial irrigation, tonic immobility was correlated with a significant depression in blood pressure and heart rate irrespective of the sharks being in air or in water. Tonic immobility with branchial irrigation resulted in a significant increase in blood pressure in sharks in air, but not in water. Heart rate was unchanged when the gills were irrigated. Intra-arterial injections of atropine abolished the bradycardia and blood pressure rise associated with tonic immobility. We conclude that, during tonic immobility, sharks are able to receive afferent information from the ventilatory system and make appropriate responses via the vagus nerve.

Embedded Voxel Colouring

The reconstruction of a complex scene from multiple images is a fundamental problem in the field of computer vision. Volumetric methods have proven to be a strong alternative to traditional correspondence-based methods due to their flexible visibility models. In this paper we analyse existing methods for volumetric reconstruction and identify three key properties of voxel colouring algorithms: a water-tight surface model, a monotonic carving order, and causality. We present a new Voxel Colouring algorithm which embeds all reconstructions of a scene into a single output. While modelling exact visibility for arbitrary camera locations, Embedded Voxel Colouring removes the need for a priori threshold selection present in previous work. An efficient implementation is given along with results demonstrating the advantages of posteriori threshold selection.

An improved seeded region growing algorithm

Recently Adams and Bischof (1994) proposed a novel region growing algorithm for segmenting intensity images. The inputs to the algorithm are the intensity image and a set of seeds - individual points or connected components - that identify the individual regions to be segmented. The algorithm grows these seed regions until all of the image pixels have been assimilated. Unfortunately the algorithm is inherently dependent on the order of pixel processing. This means, for example, that raster order processing and anti-raster order processing do not, in general, lead to the same tessellation. In this paper we propose an improved seeded region growing algorithm that retains the advantages of the Adams and Bischof algorithm fast execution, robust segmentation, and no tuning parameters - but is pixel order independent. (C) 1997 Elsevier Science B.V.

Prediction of MHC class II-binding peptides using an evolutionary algorithm and artificial neural network

Motivation: Prediction methods for identifying binding peptides could minimize the number of peptides required to be synthesized and assayed, and thereby facilitate the identification of potential T-cell epitopes. We developed a bioinformatic method for the prediction of peptide binding to MHC class II molecules. Results: Experimental binding data and expert knowledge of anchor positions and binding motifs were combined with an evolutionary algorithm (EA) and an artificial neural network (ANN): binding data extraction --> peptide alignment --> ANN training and classification. This method, termed PERUN, was implemented for the prediction of peptides that bind to HLA-DR4(B1*0401). The respective positive predictive values of PERUN predictions of high-, moderate-, low- and zero-affinity binder-a were assessed as 0.8, 0.7, 0.5 and 0.8 by cross-validation, and 1.0, 0.8, 0.3 and 0.7 by experimental binding. This illustrates the synergy between experimentation and computer modeling, and its application to the identification of potential immunotheraaeutic peptides.

Relationships between blood pressure and heart rate in the saltwater crocodile Crocodylus porosus

The cardiac limb of the baroreflex loop was studied in the saltwater crocodile Crocodylus porosus, The classical pharmacological methodology using phenylephrine and sodium nitroprusside was used to trigger blood pressure changes, and the resulting alterations in heart rate were analysed quantitatively using a logistic function. Interindividual differences in resting heart rates and blood pressures were observed, but all seven animals displayed clear baroreflex responses. Atropine and sotalol greatly attenuated the response. A maximal baroreflex gain of 7.2 beats min(-1) kPa(-1) was found at a mean aortic pressure of 6.1 kPa, indicating the active role of the baroreflex in a wide pressure range encompassing hypotensive and hypertensive states. At the lowest mean aortic pressures (5.0 kPa), the synergistic role of the pulmonary-to-systemic shunt in buffering the blood pressure drop also contributes to blood pressure regulation, Pulse pressure showed a better correlation,vith heart rate and also a higher gain than mean aortic, systolic or diastolic pressures, and this is taken as an indicator of the existence of a differential control element working simultaneously with a linear proportional element.

Heart rate, blood lactate and kinematic data of elite colts (under-19) rugby union players during competition

Physiological and kinematic data were collected from elite under-19 rugby union players to provide a greater understanding of the physical demands of rugby union. Heart rate, blood lactate and time-motion analysis data were collected from 24 players (mean +/- s((x) over bar): body mass 88.7 +/- 9.9 kg, height 185 +/- 7 cm, age 18.4 +/- 0.5 years) during six competitive premiership fixtures. Six players were chosen at random from each of four groups: props and locks, back row forwards, inside backs, outside backs. Heart rate records were classified based on percent time spent in four zones (>95%, 85-95%, 75-84%, <75% HRmax). Blood lactate concentration was measured periodically throughout each match, with movements being classified as standing, walking, jogging, cruising, sprinting, utility, rucking/mauling and scrummaging. The heart rate data indicated that props and locks (58.4%) and back row forwards (56.2%) spent significantly more time in high exertion (85-95% HRmax) than inside backs (40.5%) and outside backs (33.9%) (P < 0.001). Inside backs (36.5%) and outside backs (38.5%) spent significantly more time in moderate exertion (75-84% HRmax) than props and locks (22.6%) and back row forwards (19.8%) (P < 0.05). Outside backs (20.1%) spent significantly more time in low exertion (< 75% HRmax) than props and locks (5.8%) and back row forwards (5.6%) (P < 0.05). Mean blood lactate concentration did not differ significantly between groups (range: 4.67 mmol.l(-1) for outside backs to 7.22 mmol.l(-1) for back row forwards; P < 0.05). The motion analysis data indicated that outside backs (5750 m) covered a significantly greater total distance than either props and locks or back row forwards (4400 and 4080 m, respectively; P < 0.05). Inside backs and outside backs covered significantly greater distances walking (1740 and 1780 m, respectively; P < 0.001), in utility movements (417 and 475 m, respectively; P < 0.001) and sprinting (208 and 340 m, respectively; P < 0.001) than either props and locks or back row forwards (walking: 1000 and 991 m; utility movements: 106 and 154 m; sprinting: 72 and 94 m, respectively). Outside backs covered a significantly greater distance sprinting than inside backs (208 and 340 m, respectively; P < 0.001). Forwards maintained a higher level of exertion than backs, due to more constant motion and a large involvement in static high-intensity activities. A mean blood lactate concentration of 4.8-7.2 mmol.l(-1) indicated a need for 'lactate tolerance' training to improve hydrogen ion buffering and facilitate removal following high-intensity efforts. Furthermore, the large distances (4.2-5.6 km) covered during, and intermittent nature of, match-play indicated a need for sound aerobic conditioning in all groups (particularly backs) to minimize fatigue and facilitate recovery between high-intensity efforts.

A consistent point-searching algorithm for solution interpolation in unstructured meshes consisting of 4-node bilinear quadrilateral elements

To translate and transfer solution data between two totally different meshes (i.e. mesh 1 and mesh 2), a consistent point-searching algorithm for solution interpolation in unstructured meshes consisting of 4-node bilinear quadrilateral elements is presented in this paper. The proposed algorithm has the following significant advantages: (1) The use of a point-searching strategy allows a point in one mesh to be accurately related to an element (containing this point) in another mesh. Thus, to translate/transfer the solution of any particular point from mesh 2 td mesh 1, only one element in mesh 2 needs to be inversely mapped. This certainly minimizes the number of elements, to which the inverse mapping is applied. In this regard, the present algorithm is very effective and efficient. (2) Analytical solutions to the local co ordinates of any point in a four-node quadrilateral element, which are derived in a rigorous mathematical manner in the context of this paper, make it possible to carry out an inverse mapping process very effectively and efficiently. (3) The use of consistent interpolation enables the interpolated solution to be compatible with an original solution and, therefore guarantees the interpolated solution of extremely high accuracy. After the mathematical formulations of the algorithm are presented, the algorithm is tested and validated through a challenging problem. The related results from the test problem have demonstrated the generality, accuracy, effectiveness, efficiency and robustness of the proposed consistent point-searching algorithm. Copyright (C) 1999 John Wiley & Sons, Ltd.

Increased ovulation rate in Merino ewes immunized against small synthetic peptide fragments of the inhibin alpha subunit

Four experiments were carried out in Merino ewes during a period of 4 years to determine the long-term effects of immunization against different synthetic peptides mimicking the amine terminal of the or subunit of porcine inhibin. Peptides were conjugated to human serum albumin and 100-200 mu g emulsified in Freund's complete adjuvant for the primary immunization. Usually two booster injections were given at monthly intervals with 50-100 mu g conjugated peptide using either incomplete Freund's adjuvant or Montanide : Marcel. In some experiments a further immunization was carried in the next year. Blood samples were taken 10 days after each immunization, during the luteal phase, for estimation of gonadotrophin concentrations and determination of inhibin antibody titres. One day after blood sampling cloprostenol was used to induce luteolysis and laparoscopy was performed in the subsequent oestrous cycle. Immunization of ewes with synthetic peptides 1-32, 1-26, 7-26 and 8-30 resulted in large increases in the ovulation rate (OR). An approximately two-fold increase in OR was observed following the first booster immunization with these peptides and a three- to five-fold increase after the second booster immunization. Immunization with these large peptides resulted in a sustained increase in OR for a period of at least 1 year after the second booster immunization. Of the shorter peptides, peptides 10-26 and 13-26 gave a reasonable ovulatory response, although it was more difficult to obtain a response with peptides 1-16, 8-22, 13-25, 8-19 and 10-19; peptides 7-13 and 1-6 gave no response (but were examined for one breeding season only). The smaller peptides led to lower inhibin antibody titres that were not necessarily associated with increased follicle-stimulating hormone (FSH) or OR. More intensive blood sampling in one experiment showed that following primary immunization against peptide 1-32 there was a transient increase in plasma FSH which did not lead to an increased OR. Moreover, a prolonged period of raised FSH after the first booster was significantly correlated with increased OR. In these animals antibody titres were only slightly increased after primary immunization, but after the first booster immunization higher titres were observed that were significantly correlated with trough FSH values and the subsequent OR. These results are interpreted as showing that (1) to obtain an increase in OR peptides 1-32, 1-26 and 7-26 are suitable as immunogens; (2) smaller peptides are less reliable, often require multiple injections, and the response may be delayed; and (3) an extended period of raised plasma FSH is needed to give a large ovulatory response.