Training with noise is equivalent to Tikhonov regularization

It is well known that the addition of noise to the input data of a neural network during training can, in some circumstances, lead to significant improvements in generalization performance. Previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function. However, the regularization term, which involves second derivatives of the error function, is not bounded below, and so can lead to difficulties if used directly in a learning algorithm based on error minimization. In this paper we show that, for the purposes of network training, the regularization term can be reduced to a positive definite form which involves only first derivatives of the network mapping. For a sum-of-squares error function, the regularization term belongs to the class of generalized Tikhonov regularizers. Direct minimization of the regularized error function provides a practical alternative to training with noise.

Regularization and complexity control in feed-forward networks

In this paper we consider four alternative approaches to complexity control in feed-forward networks based respectively on architecture selection, regularization, early stopping, and training with noise. We show that there are close similarities between these approaches and we argue that, for most practical applications, the technique of regularization should be the method of choice.

Infer user interests via link structure regularization

Learning user interests from online social networks helps to better understand user behaviors and provides useful guidance to design user-centric applications. Apart from analyzing users' online content, it is also important to consider users' social connections in the social Web. Graph regularization methods have been widely used in various text mining tasks, which can leverage the graph structure information extracted from data. Previously, graph regularization methods operate under the cluster assumption that nearby nodes are more similar and nodes on the same structure (typically referred to as a cluster or a manifold) are likely to be similar. We argue that learning user interests from complex, sparse, and dynamic social networks should be based on the link structure assumption under which node similarities are evaluated based on the local link structures instead of explicit links between two nodes. We propose a regularization framework based on the relation bipartite graph, which can be constructed from any type of relations. Using Twitter as our case study, we evaluate our proposed framework from social networks built from retweet relations. Both quantitative and qualitative experiments show that our proposed method outperforms a few competitive baselines in learning user interests over a set of predefined topics. It also gives superior results compared to the baselines on retweet prediction and topical authority identification. © 2014 ACM.

Curvature-driven smoothing: a learning algorithm for feed-forward networks

The performance of feed-forward neural networks in real applications can be often be improved significantly if use is made of a-priori information. For interpolation problems this prior knowledge frequently includes smoothness requirements on the network mapping, and can be imposed by the addition to the error function of suitable regularization terms. The new error function, however, now depends on the derivatives of the network mapping, and so the standard back-propagation algorithm cannot be applied. In this paper, we derive a computationally efficient learning algorithm, for a feed-forward network of arbitrary topology, which can be used to minimize the new error function. Networks having a single hidden layer, for which the learning algorithm simplifies, are treated as a special case.

Learning with regularizers in multilayer neural networks

We study the effect of regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labelled by a two-layer teacher network with an arbitrary number of hidden units which may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical evolution of the order parameters and generalization error in various phases of the learning process, in both noiseless and noisy scenarios.

Gaussian regression and optimal finite dimensional linear models

The problem of regression under Gaussian assumptions is treated generally. The relationship between Bayesian prediction, regularization and smoothing is elucidated. The ideal regression is the posterior mean and its computation scales as O(n³), where n is the sample size. We show that the optimal m-dimensional linear model under a given prior is spanned by the first m eigenfunctions of a covariance operator, which is a trace-class operator. This is an infinite dimensional analogue of principal component analysis. The importance of Hilbert space methods to practical statistics is also discussed.

Hierarchical GTM: Constructing localized non-linear projection manifolds in a principled way

It has been argued that a single two-dimensional visualization plot may not be sufficient to capture all of the interesting aspects of complex data sets, and therefore a hierarchical visualization system is desirable. In this paper we extend an existing locally linear hierarchical visualization system PhiVis ¸iteBishop98a in several directions: bf(1) We allow for em non-linear projection manifolds. The basic building block is the Generative Topographic Mapping. bf(2) We introduce a general formulation of hierarchical probabilistic models consisting of local probabilistic models organized in a hierarchical tree. General training equations are derived, regardless of the position of the model in the tree. bf(3) Using tools from differential geometry we derive expressions for local directional curvatures of the projection manifold. Like PhiVis, our system is statistically principled and is built interactively in a top-down fashion using the EM algorithm. It enables the user to interactively highlight those data in the parent visualization plot which are captured by a child model. We also incorporate into our system a hierarchical, locally selective representation of magnification factors and directional curvatures of the projection manifolds. Such information is important for further refinement of the hierarchical visualization plot, as well as for controlling the amount of regularization imposed on the local models. We demonstrate the principle of the approach on a toy data set and apply our system to two more complex 12- and 19-dimensional data sets.

Using directional curvatures to visualize folding patterns of the GTM projection manifolds

In data visualization, characterizing local geometric properties of non-linear projection manifolds provides the user with valuable additional information that can influence further steps in the data analysis. We take advantage of the smooth character of GTM projection manifold and analytically calculate its local directional curvatures. Curvature plots are useful for detecting regions where geometry is distorted, for changing the amount of regularization in non-linear projection manifolds, and for choosing regions of interest when constructing detailed lower-level visualization plots.

Hierarchical GTM: constructing localized non-linear projection manifolds in a principled way

It has been argued that a single two-dimensional visualization plot may not be sufficient to capture all of the interesting aspects of complex data sets, and therefore a hierarchical visualization system is desirable. In this paper we extend an existing locally linear hierarchical visualization system PhiVis ¸iteBishop98a in several directions: bf(1) We allow for em non-linear projection manifolds. The basic building block is the Generative Topographic Mapping (GTM). bf(2) We introduce a general formulation of hierarchical probabilistic models consisting of local probabilistic models organized in a hierarchical tree. General training equations are derived, regardless of the position of the model in the tree. bf(3) Using tools from differential geometry we derive expressions for local directional curvatures of the projection manifold. Like PhiVis, our system is statistically principled and is built interactively in a top-down fashion using the EM algorithm. It enables the user to interactively highlight those data in the ancestor visualization plots which are captured by a child model. We also incorporate into our system a hierarchical, locally selective representation of magnification factors and directional curvatures of the projection manifolds. Such information is important for further refinement of the hierarchical visualization plot, as well as for controlling the amount of regularization imposed on the local models. We demonstrate the principle of the approach on a toy data set and apply our system to two more complex 12- and 18-dimensional data sets.

Constructing localized non-linear projection manifolds in a principled way:hierarchical generative topographic mapping

It has been argued that a single two-dimensional visualization plot may not be sufficient to capture all of the interesting aspects of complex data sets, and therefore a hierarchical visualization system is desirable. In this paper we extend an existing locally linear hierarchical visualization system PhiVis (Bishop98a) in several directions: 1. We allow for em non-linear projection manifolds. The basic building block is the Generative Topographic Mapping. 2. We introduce a general formulation of hierarchical probabilistic models consisting of local probabilistic models organized in a hierarchical tree. General training equations are derived, regardless of the position of the model in the tree. 3. Using tools from differential geometry we derive expressions for local directionalcurvatures of the projection manifold. Like PhiVis, our system is statistically principled and is built interactively in a top-down fashion using the EM algorithm. It enables the user to interactively highlight those data in the parent visualization plot which are captured by a child model.We also incorporate into our system a hierarchical, locally selective representation of magnification factors and directional curvatures of the projection manifolds. Such information is important for further refinement of the hierarchical visualization plot, as well as for controlling the amount of regularization imposed on the local models. We demonstrate the principle of the approach on a toy data set andapply our system to two more complex 12- and 19-dimensional data sets.

Liposome-mediated DNA immunisation via the subcutaneous route

Compared to naked DNA immunisation, entrapment of plasmid-based DNA vaccines into liposomes by the dehydration-rehydration method has shown to enhance both humoural and cell-mediated immune responses to encoded antigens administered by a variety of routes. In this paper, we have investigated the application of liposome-entrapped DNA and their cationic lipid composition on such potency after subcutaneous immunisation. Plasmid pI.18Sfi/NP containing the nucleoprotein (NP) gene of A/Sichuan/2/87 (H3N2) influenza virus in the pI.18 expression vector was incorporated by the dehydration-rehydration method into liposomes composed of 16 μmol egg phosphatidylcholine (PC), 8 μmoles dioleoyl phosphatidylethanolamine (DOPE) or cholesterol (Chol) and either the cationic lipid 1,2-diodeoyl-3-(trimethylammonium) propane (DOTAP) or cholesteryl 3-N-(dimethyl amino ethyl) carbamate (DC-Chol). This method, entailing mixing of small unilamellar vesicles (SUV) with DNA, followed by dehydration and rehydration, yielded incorporation values of 90-94% of the DNA used. Mixing or rehydration of preformed cationic liposomes with 100 μg plasmid DNA also led to similarly high complexation values (92-94%). In an attempt to establish differences in the nature of DNA association with these various liposome preparations their physico-chemical characteristics were investigated. Studies on vesicle size, zeta potential and gel electrophoresis in the presence of the anion sodium dodecyl sulphate (SDS) indicate that, under the conditions employed, formulation of liposomal DNA by the dehydration-rehydration generated submicron size liposomes incorporating most of the DNA in a manner that prevents DNA displacement through anion competition. The bilayer composition of these dehydration-rehydration vesicles (DRV(DNA)) can also further influence these physicochemical characteristics with the presence of DOPE within the liposome bilayer resulting in a reduced vesicle zeta potential. Subcutaneous liposome-mediated DNA immunisation employing two DRV(DNA) formulations as well as naked DNA revealed that humoural responses (immunoglobulin total IgG, and subclasses IgG1 and 1gG2a) engendered by the plasmid encoded NP were substantially higher after dosing twice, 28 days apart with 10 μg liposome-entrapped DNA compared to naked DNA. At all time points measured, mice immunised with naked DNA showed no greater immune response compared to the control, non-immunised group. In contrast, as early as day 49, responses were significantly higher in mice injected with DNA entrapped in DRV liposomes containing DOTAP compared to the control group and mice immunised with naked DNA. By day 56, all total IgG responses from mice immunised with both DRV formulations were significantly higher. Comparison between the DRV formulations revealed no significant difference in immune responses elicited except at day 114, where the humoural responses of the group injected with liposomal formulation containing DC-Chol dropped to significantly lower levels that those measured in mice which received the DOTAP formulation. Similar results were found when the IgG1 and IgG2a subclass responses were determined. These results suggest that, not only can DNA be effectively entrapped within liposomes using the DRV method but that such DRV liposomes containing DNA may be a useful system for subcutaneous delivery of DNA vaccines. © 2003 Taylor & Francis Ltd.

Hydrophobicity and surface electrostatic charge of conidia of the mycoparasite Coniothyrium minitans

The effect of increasing culture age on cell surface hydrophobicity (CSH) and cell surface electrostatic charge (measured as zeta potential) of conidia from five isolates of Coniothyrium minitans representing three different morphological types was examined. Conidial CSH of three isolates (A2 960/1, CH1 and CH2) decreased with culture age, whereas CSH of two others (B 1300/2 and IMI 134523) remained high for the whole 42 day experimental period. In contrast, cell surface electrostatic charge decreased uniformly in conidia of all five isolates for the first 34 d and then rose slightly at 42 d. The variation in cell surface electrostatic charge (spectrum width) of the sampled conidia decreased with age for all five isolates. In all five isolates cell surface electrostatic charge of conidia became increasingly negative as the pH of the buffer used to suspend conidia was increased from pH 3.0 to 9.0. No relationship between colony morphology of C. minitans and conidial CSH and cell surface electrostatic charge was found.

Characterisation of a harvesting step for monoclonal antibodies from mammalian cell culture

The focus of this research was defined by a poorly characterised filtration train employed to clarify culture broth containing monoclonal antibodies secreted by GS-NSO cells: the filtration train blinded unpredictably and the ability of the positively charged filters to adsorb DNA from process material was unknown. To direct the development of an assay to quantify the ability of depth filters to adsorb DNA, the molecular weight of DNA from a large-scale, fed-batch, mammalian cell culture vessel was evaluated as process material passed through the initial stages of the purification scheme. High molecular weight DNA was substantially cleared from the broth after passage through a disc stack centrifuge and the remaining low molecular weight DNA was largely unaffected by passage through a series of depth filters and a sterilising grade membrane. Removal of high molecular weight DNA was shown to be coupled with clarification of the process stream. The DNA from cell culture supernatant showed a pattern of internucleosomal cleavage of chromatin when fractionated by electrophoresis but the presence of both necrotic and apoptotic cells throughout the fermentation meant that the origin of the fragmented DNA could not be unequivocally determined. An intercalating fluorochrome, PicoGreen, was elected for development of a suitable DNA assay because of its ability to respond to low molecular weight DNA. It was assessed for its ability to determine the concentration of DNA in clarified mammalian cell culture broths containing pertinent monoclonal antibodies. Fluorescent signal suppression was ameliorated by sample dilution or by performing the assay above the pI of secreted IgG. The source of fluorescence in clarified culture broth was validated by incubation with RNase A and DNase I. At least 89.0 % of fluorescence was attributable to nucleic acid and pre-digestion with RNase A was shown to be a requirement for successful quantification of DNA in such samples. Application of the fluorescence based assay resulted in characterisation of the physical parameters governing adsorption of DNA by various positively charged depth filters and membranes in test solutions and the DNA adsorption profile of the manufacturing scale filtration train. Buffers that reduced or neutralised the depth filter or membrane charge, and those that impeded hydrophobic interactions were shown to affect their operational capacity, demonstrating that DNA was adsorbed by a combination of electrostatic and hydrophobic interactions. Production-scale centrifugation of harvest broth containing therapeutic protein resulted in the reduction of total DNA in the process stream from 79.8 μg m1-1 to 9.3 μg m1-1 whereas the concentration of DNA in the supernatant of pre-and post-filtration samples had only marginally reduced DNA content: from 6.3 to 6.0 μg m1-1 respectively. Hence the filtration train was shown to ineffective in DNA removal. Historically, blinding of the depth filters had been unpredictable with data such as numbers of viable cells, non-viable cells, product titre, or process shape (batch, fed-batch, or draw and fill) failing to inform on the durability of depth filters in the harvest step. To investigate this, key fouling contaminants were identified by challenging depth filters with the same mass of one of the following: viable healthy cells, cells that had died by the process of apoptosis, and cells that had died through the process of necrosis. The pressure increase across a Cuno Zeta Plus 10SP depth filter was 2.8 and 16.5 times more sensitive to debris from apoptotic and necrotic cells respectively, when compared to viable cells. The condition of DNA released into the culture broth was assessed. Necrotic cells released predominantly high molecular weight DNA in contrast to apoptotic cells which released chiefly low molecular weight DNA. The blinding of the filters was found to be largely unaffected by variations in the particle size distribution of material in, and viscosity of, solutions with which they were challenged. The exceptional response of the depth filters to necrotic cells may suggest the cause of previously noted unpredictable filter blinding whereby a number of necrotic cells have a more significant impact on the life of a depth filter than a similar number of viable or apoptotic cells. In a final set of experiments the pressure drop caused by non-viable necrotic culture broths which had been treated with DNase I or benzonase was found to be smaller when compared to untreated broths: the abilities of the enzyme treated cultures to foul the depth filter were reduced by 70.4% and 75.4% respectively indicating the importance of DNA in the blinding of the depth filter studied.

Liposomes as carriers for polymyxins in the treatment of cystic fibrosis lung infections

Cystic fibrosis (CF) is the most common autosomal recessive disorder affecting Caucasian populations. The pathophysiology of this disorder predisposes the lungs of affected patients to chronic infection, typically by Pseudomonas aeruginosa, which is the main cause of morbidity and mortality. Recently, attention has focused on aerosolised polymyxins, which are given prophylactically in an effort to limit infection and subsequent lung damage. This class of antimicrobial compounds is highly active against P. aeruginosa and possess the advantage that resistance rarely develops. However, the rapid lung clearance of antibiotics is a well documented phenomenon and it was postulated that polymyxin treatment could be further improved by liposomal encapsulation. As part of the development of liposomal polymyxin B, analytical methodology (radiolabelling, HPLC and protein assay) applicable to liposomal formulations was established. Liposomes were prepared by the dehydration-rehydration method and encapsulation efficiencies were determined for a number of phospholipid compositions. Vesicles were characterised with respect to size, zeta potential, morphology and release characteristics. The surface hydrophobicity of vesicles was quantified by hydrophobic interaction chromatography and it was found that this method produced comparable results to techniques conventionally used to assess this property. In vivo testing of liposomal polymyxins demonstrated that encapsulation successfully prevented the rapid pulmonary clearance of PXB. Antimicrobial activity of liposomal formulations was quantified and found to be dependent on both the vesicle surface characteristics and their release profile. Investigation of the interaction of PXB with lipopolysaccharide was undertaken and results demonstrated that PXB caused significant structural distortion of the lipid A region. This may be sufficient to abrogate the potentiating action of LPS in the inflammatory cascade.

Electrodeposition of cermets

Modern engineering requirements are frequently near the limits of application of conventional materials. For many purposes, particularly tribological, the most satisfactory solution is frequently the application of a resistant coating to the surface of a common metal. Electrodeposited cermet coatings have proved very satisfactory: some of the factors underlying the cernet electrodeposition process have been investigated. A ceramic particle in contact with an electrolyte solution will carry a charge which may affect the kinetics of the suspended particle under electroplating conditions. Measurerment has been made of this charge on particles of silicon carbide, chrornium diboride and quartz, in contiact with solutions of copper sulphate/ sulphuric acid in terms of the electrokinetic (zeta) potential and also as surface charge density. The methocl used was that of streaming potential and streaming current measurement