Time trials on second-order and variable-learning-rate algorithms

The performance of seven minimization algorithms are compared on five neural network problems. These include a variable-step-size algorithm, conjugate gradient, and several methods with explicit analytic or numerical approximations to the Hessian.

Bayesian training of mixture density networks

Mixture Density Networks (MDNs) are a well-established method for modelling the conditional probability density which is useful for complex multi-valued functions where regression methods (such as MLPs) fail. In this paper we extend earlier research of a regularisation method for a special case of MDNs to the general case using evidence based regularisation and we show how the Hessian of the MDN error function can be evaluated using R-propagation. The method is tested on two data sets and compared with early stopping.

Incorporating curvature information into on-line learning

We analyse the dynamics of a number of second order on-line learning algorithms training multi-layer neural networks, using the methods of statistical mechanics. We first consider on-line Newton's method, which is known to provide optimal asymptotic performance. We determine the asymptotic generalization error decay for a soft committee machine, which is shown to compare favourably with the result for standard gradient descent. Matrix momentum provides a practical approximation to this method by allowing an efficient inversion of the Hessian. We consider an idealized matrix momentum algorithm which requires access to the Hessian and find close correspondence with the dynamics of on-line Newton's method. In practice, the Hessian will not be known on-line and we therefore consider matrix momentum using a single example approximation to the Hessian. In this case good asymptotic performance may still be achieved, but the algorithm is now sensitive to parameter choice because of noise in the Hessian estimate. On-line Newton's method is not appropriate during the transient learning phase, since a suboptimal unstable fixed point of the gradient descent dynamics becomes stable for this algorithm. A principled alternative is to use Amari's natural gradient learning algorithm and we show how this method provides a significant reduction in learning time when compared to gradient descent, while retaining the asymptotic performance of on-line Newton's method.

The dynamics of matrix momentum

We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.

Bayesian error bars for regression

Regression problems are concerned with predicting the values of one or more continuous quantities, given the values of a number of input variables. For virtually every application of regression, however, it is also important to have an indication of the uncertainty in the predictions. Such uncertainties are expressed in terms of the error bars, which specify the standard deviation of the distribution of predictions about the mean. Accurate estimate of error bars is of practical importance especially when safety and reliability is an issue. The Bayesian view of regression leads naturally to two contributions to the error bars. The first arises from the intrinsic noise on the target data, while the second comes from the uncertainty in the values of the model parameters which manifests itself in the finite width of the posterior distribution over the space of these parameters. The Hessian matrix which involves the second derivatives of the error function with respect to the weights is needed for implementing the Bayesian formalism in general and estimating the error bars in particular. A study of different methods for evaluating this matrix is given with special emphasis on the outer product approximation method. The contribution of the uncertainty in model parameters to the error bars is a finite data size effect, which becomes negligible as the number of data points in the training set increases. A study of this contribution is given in relation to the distribution of data in input space. It is shown that the addition of data points to the training set can only reduce the local magnitude of the error bars or leave it unchanged. Using the asymptotic limit of an infinite data set, it is shown that the error bars have an approximate relation to the density of data in input space.

Ecological studies on river pollution control

This collection of papers records a series of studies, carried out over a period of some 50 years, on two aspects of river pollution control - the prevention of pollution by sewage biological filtration and the monitoring of river pollution by biological surveillance. The earlier studies were carried out to develop methods of controlling flies which bred in the filters and caused serious nuisance and possible public health hazard, when they dispersed to surrounding villages. Although the application of insecticides proved effective as an alleviate measure, because it resulted in only a temporary disturbance of the ecological balance, it was considered ecologically unsound as a long-term solution. Subsequent investigations showed that the fly populations in filters were largely determined by the amount of food available to the grazing larval stage in the form of filter film. It was also established that the winter deterioration in filter performance was due to the excessive accumulation of film. Subsequent investigations were therefore carried out to determine the factors responsible for the accumulation of film in different types of filter. Methods of filtration which were considered to control film accumulation by increasing the flushing action of the sewage, were found to control fungal film by creating nutrient limiting conditions. In some filters increasing the hydraulic flushing reduced the grazing fauna population in the surface layers and resulted in an increase in film. The results of these investigations were successfully applied in modifying filters and in the design of a Double Filtration process. These studies on biological filters lead to the conclusion that they should be designed and operated as ecological systems and not merely as hydraulic ones. Studies on the effects of sewage effluents on Birmingham streams confirmed the findings of earlier workers justifying their claim for using biological methods for detecting and assessing river pollution. Further ecological studies showed the sensitivity of benthic riffle communities to organic pollution. Using experimental channels and laboratory studies the different environmental conditions associated with organic pollution were investigated. The degree and duration of the oxygen depletion during the dark hours were found to be a critical factor. The relative tolerance of different taxa to other pollutants, such as ammonia, differed. Although colonisation samplers proved of value in sampling difficult sites, the invertebrate data generated were not suitable for processing as any of the commonly used biotic indexes. Several of the papers, which were written by request for presentation at conferences etc., presented the biological viewpoint on river pollution and water quality issues at the time and advocated the use of biological methods. The information and experiences gained in these investigations was used as the "domain expert" in the development of artificial intelligence systems for use in the biological surveillance of river water quality.

L-carnosine affects the growth of Saccharomyces cerevisiae in a metabolism-dependent manner

The dipeptide L-carnosine (β-alanyl-L-histidine) has been described as enigmatic: it inhibits growth of cancer cells but delays senescence in cultured human fibroblasts and extends the lifespan of male fruit flies. In an attempt to understand these observations, the effects of L-carnosine on the model eukaryote, Saccharomyces cerevisiae, were examined on account of its unique metabolic properties; S. cerevisiae can respire aerobically, but like some tumor cells, it can also exhibit a metabolism in which aerobic respiration is down regulated. L-Carnosine exhibited both inhibitory and stimulatory effects on yeast cells, dependent upon the carbon source in the growth medium. When yeast cells were not reliant on oxidative phosphorylation for energy generation (e.g. when grown on a fermentable carbon source such as 2% glucose), 10-30 mM L-carnosine slowed growth rates in a dose-dependent manner and increased cell death by up to 17%. In contrast, in media containing a non-fermentable carbon source in which yeast are dependent on aerobic respiration (e.g. 2% glycerol), L-carnosine did not provoke cell death. This latter observation was confirmed in the respiratory yeast, Pichia pastoris. Moreover, when deletion strains in the yeast nutrient-sensing pathway were treated with L-carnosine, the cells showed resistance to its inhibitory effects. These findings suggest that L-carnosine affects cells in a metabolism-dependent manner and provide a rationale for its effects on different cell types. © 2012 Cartwright et al.

Saccharomyces Cerevisiae as a biotechnological tool for ageing research:studies on translation and metabolism

The yeast Saccharomyces cerevisiae is an important model organism for the study of cell biology. The similarity between yeast and human genes and the conservation of fundamental pathways means it can be used to investigate characteristics of healthy and diseased cells throughout the lifespan. Yeast is an equally important biotechnological tool that has long been the organism of choice for the production of alcoholic beverages, bread and a large variety of industrial products. For example, yeast is used to manufacture biofuels, lubricants, detergents, industrial enzymes, food additives and pharmaceuticals such as anti-parasitics, anti-cancer compounds, hormones (including insulin), vaccines and nutraceuticals. Its function as a cell factory is possible because of the speed with which it can be grown to high cell yields, the knowledge that it is generally recognized as safe (GRAS) and the ease with which metabolism and cellular pathways, such as translation can be manipulated. In this thesis, these two pathways are explored in the context of their biotechnological application to ageing research: (i) understanding translational processes during the high-yielding production of membrane protein drug targets and (ii) the manipulation of yeast metabolism to study the molecule, L-carnosine, which has been proposed to have anti-ageing properties. In the first of these themes, the yeast strains, spt3?, srb5?, gcn5? and yTHCBMS1, were examined since they have been previously demonstrated to dramatically increase the yields of a target membrane protein (the aquaporin, Fps1) compared to wild-type cells. The mechanisms underlying this discovery were therefore investigated. All high yielding strains were shown to have an altered translational state (mostly characterised by an initiation block) and constitutive phosphorylation of the translational initiation factor, eIF2a. The relevance of the initiation block was further supported by the finding that other strains, with known initiation blocks, are also high yielding for Fps1. A correlation in all strains between increased Fps1 yields and increased production of the transcriptional activator protein, Gcn4, suggested that yields are subject to translational control. Analysis of the 5´ untranslated region (UTR) of FPS1 revealed two upstream open reading frames (uORFs). Mutagenesis data suggest that high yielding strains may circumvent these control elements through either a leaky scanning or a re-initiation mechanism. In the second theme, the dipeptide L-carnosine (ß-alanyl-L-histidine) was investigated: it has previously been shown to inhibit the growth of cancer cells but delay senescence in cultured human fibroblasts and extend the lifespan of male fruit flies. To understand these apparently contradictory properties, the effects of L-carnosine on yeast were studied. S. cerevisiae can respire aerobically when grown on a non-fermentable carbon source as a substrate but has a respiro-fermentative metabolism when grown on a fermentable carbon source; these metabolisms mimic normal cell and cancerous cell metabolisms, respectively. When yeast were grown on fermentable carbon sources, in the presence of L-carnosine, a reduction in cell growth and viability was observed, which was not apparent for cells grown on a non-fermentable carbon source. The metabolism-dependent mechanism was confirmed in the respiratory yeast species Pichia pastoris. Further analysis of S. cerevisiae yeast strains with deletions in their nutrient-sensing pathway, which result in an increase in respiratory metabolism, confirmed the metabolism-dependent effects of L-carnosine.

The interaction between metabolic disease and ageing

Two of the greatest crises that civilisation faces in the 21st century are the predicted rapid increases in the ageing population and levels of metabolic disorders such as obesity and type 2 diabetes. A growing amount of evidence now supports the notion that energy balance is a key determinant not only in metabolism but also in the process of cellular ageing. Much of genetic evidence for a metabolic activity-driven ageing process has come from model organisms such as worms and flies where inactivation of the insulin receptor signalling cascade prolongs lifespan. At its most simplistic, this poses a conundrum for ageing in humans – can reduced insulin receptor signalling really promote lifespan and does this relate to insulin resistance seen in ageing? In higher animals, caloric restriction studies have confirmed a longer lifespan when daily calorie intake is reduced to 60% of normal energy requirement. This suggests that for humans, it is energy excess which is a likely driver of metabolic ageing. Interventions that interfere with the metabolic fate of nutrients offer a potentially important target for delaying biological ageing.

Isolation and characterisation of bacteria associated with flying insects in hospitals, with particualr emphasis on Clostridum difficile

Clostridium difficile is a bacterial healthcare-associated infection, which houseflies Musca domestica may transfer due to their synanthropic nature. The aims of this thesis were to determine the ability of M. domestica to transfer C. difficile mechanically and to collect and identify flying insects in UK hospitals and classify any associated bacteria. M. domestica exposed to independent suspensions of vegetative cells and spores of C. difficile were able to mechanically transfer the bacteria on to agar for up to 4 hours following exposure. C. difficile could be recovered from fly excreta for 96hrs and was isolated from the M. domestica alimentary canal. Also confirmed was the carriage of C. difficile by M. domestica larvae, although it was not retained in the pupae or in the adults that subsequently developed. Flying insects were collected from ultra-violet light flytraps in hospitals. Flies (order Diptera) were the most commonly identified. Chironomidae were the most common flies, Calliphora vicina were the most common synanthropic fly and ‘drain flies’ were surprisingly numerous and represent an emerging problem in hospitals. External washings and macerates of flying insects were prepared and inoculated onto a variety of agars and following incubation bacterial colonies identified by biochemical tests. A variety of flying insects, including synanthropic flies (e.g. M. domestica and C. vicina) collected from UK hospitals harboured pathogenic bacteria of different species. Enterobacteriaceae were the group of bacteria most commonly isolated, followed by Bacillus spp, Staphylococci, Clostridia, Streptococci and Micrococcus spp. This study highlights the potential for M. domestica to contribute to environmental persistence and spread of C. difficile in hospitals. Also illustrated is the potential for flying insects to contribute to environmental persistence and spread of other pathogenic bacteria in hospitals and therefore the need to implement pest control as part of infection control strategies.

The housefly Musca domestica as a mechanical vector of Clostridium difficile

Background - Clostridium difficile is a bacterial healthcare-associated infection that may be transferred by houseflies (Musca domestica) due to their close ecological association with humans and cosmopolitan nature. Aim - To determine the ability of M. domestica to transfer C. difficile both mechanically and following ingestion. Methods - M. domestica were exposed to independent suspensions of vegetative cells and spores of C. difficile, then sampled on to selective agar plates immediately postexposure and at 1-h intervals to assess the mechanical transfer of C. difficile. Fly excreta was cultured and alimentary canals were dissected to determine internalization of cells and spores. Findings - M. domestica exposed to vegetative cell suspensions and spore suspensions of C. difficile were able to transfer the bacteria mechanically for up to 4 h upon subsequent contact with surfaces. The greatest numbers of colony-forming units (CFUs) per fly were transferred immediately following exposure (mean CFUs 123.8 +/− 66.9 for vegetative cell suspension and 288.2 +/− 83.2 for spore suspension). After 1 h, this had reduced (21.2 +/− 11.4 for vegetative cell suspension and 19.9 +/− 9 for spores). Mean C. difficile CFUs isolated from the M. domestica alimentary canal was 35 +/− 6.5, and mean C. difficile CFUs per faecal spot was 1.04 +/− 0.58. C. difficile could be recovered from fly excreta for up to 96 h. Conclusion - This study describes the potential for M. domestica to contribute to environmental persistence and spread of C. difficile in hospitals, highlighting flies as realistic vectors of this micro-organism in clinical areas.