Dynamics of on-line gradient descent learning for multilayer neural networks

We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

Transients and asymptotics of natural gradient learning

We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.

Natural gradient matrix momentum

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.

Blurred edges look faint, and faint edges look sharp:The effect of a gradient threshold in a multi-scale edge coding model

A multi-scale model of edge coding based on normalized Gaussian derivative filters successfully predicts perceived scale (blur) for a wide variety of edge profiles [Georgeson, M. A., May, K. A., Freeman, T. C. A., & Hesse, G. S. (in press). From filters to features: Scale-space analysis of edge and blur coding in human vision. Journal of Vision]. Our model spatially differentiates the luminance profile, half-wave rectifies the 1st derivative, and then differentiates twice more, to give the 3rd derivative of all regions with a positive gradient. This process is implemented by a set of Gaussian derivative filters with a range of scales. Peaks in the inverted normalized 3rd derivative across space and scale indicate the positions and scales of the edges. The edge contrast can be estimated from the height of the peak. The model provides a veridical estimate of the scale and contrast of edges that have a Gaussian integral profile. Therefore, since scale and contrast are independent stimulus parameters, the model predicts that the perceived value of either of these parameters should be unaffected by changes in the other. This prediction was found to be incorrect: reducing the contrast of an edge made it look sharper, and increasing its scale led to a decrease in the perceived contrast. Our model can account for these effects when the simple half-wave rectifier after the 1st derivative is replaced by a smoothed threshold function described by two parameters. For each subject, one pair of parameters provided a satisfactory fit to the data from all the experiments presented here and in the accompanying paper [May, K. A. & Georgeson, M. A. (2007). Added luminance ramp alters perceived edge blur and contrast: A critical test for derivative-based models of edge coding. Vision Research, 47, 1721-1731]. Thus, when we allow for the visual system's insensitivity to very shallow luminance gradients, our multi-scale model can be extended to edge coding over a wide range of contrasts and blurs. © 2007 Elsevier Ltd. All rights reserved.

Enzymatic production of oligosaccharides in centrifugal fields

The aims of this work have been to identify an enzymatic reaction system suitable to investigate and develop the high-speed centrifuge as a novel reaction system for performing such reactions. The production of galacto-oligosaccharides by the trans-galactosyl activity of the enzyme β-galactosidase on lactose monohydrate was identified as a model enzymatic system to elucidate the principles of this type of process. Galacto-oligosaccharides have attracted considerable commercial interest as food additives which have been shown to be beneficial to the health of the human gastrointestinal tract. The development of a single unit operation capable of controlling the biosynthesis of galacto-oligosaccharides whilst simultaneously separating the enzyme from the reaction products would reduce downstream processing costs. This thesis shows for the first time that by using a combination of (a) immobilised or insolubilised β-galactosidase , (b) a rate-zonal centrifugation technique, and (c) various applied centrifugal fields, that a high-speed centrifuge could be used to control the formation of galacto-oligosaccharides whilst removing the enzyme from the reaction products. By layering a suspension of insolubilised β-galactosidase on top of a lactose monohydrate density gradient and centrifuging, the applied centrifugal fields generated produced sedimentation of the enzyme particles through the substrate. The higher sedimentation rate of the enzyme compared to those of the reaction products allowed for separation to take place. Complete sedimentation, or pelleting of the enzyme permits the possible recovery and re-use. Insolubilisation of the enzyme allowed it to be sedimented through the substrate gradient using much lower applied centrifugal fields than that required to sediment free soluble enzyme and this allowed for less expensive centrifugation equipment to be used. Using free soluble and insolubilised β-galactosidase stirred-batch reactions were performed to investigate the kinetics of lactose monohydrate hydrolysis and galacto-oligosaccharide formation. Based on these results a preliminary mathematical model based on Michaelis-Menten kinetics was produced. It was found that the enzyme insolubilisation process using a chemical cross-linking agent did not affect the process of galacto-oligosaccharide formation. Centrifugation experiments were performed and it was found that by varying the applied centrifugal fields that the yield of galacto-oligosaccharides could be controlled. The higher the applied centrifugal fields the lower the yield of galacto-oligosaccharides. By increasing the applied centrifugal fields the 'contact time' between the sedimenting enzyme and the substrate was reduced, which produced lower yields. A novel technique involving pulsing the insolubilised enzyme through the substrate gradient was developed and this was found to produce higher yields of galacto-oligosaccharide compared to using a single enzyme loading equivalent to the total combined activity of the pulses. Comparison of the galacto-oligosaccharide yields between stirred-batch and centrifugation reactions showed that the applied centrifugal fields did not adversely affect the transgalactosyl activity of the insolubilised enzyme.

Continuous chromatographic biochemical reaction-separation

Combined bioreaction separation studies have been carried out for the first time on a moving port semi-continuous counter-current chromatographic reactor-separator (SCCR-S1) consisting of twelve 5.4cm id x 75cm long columns packed with calcium charged cross-linked polystyrene resin (KORELA V07C). The inversion of sucrose to glucose and fructose in the presence of the enzyme invertase and the biochemIcal synthesis of dextran and fructose from sucrose in the presence of the enzyme dextransucrase were investigated. A dilute stream of the appropriate enzyme in deionised water was used as the eluent stream. The effect of switch time, feed concentration, enzyme activity, eluent rate and enzyme to feed concentration ratio on the combined bioreaction-separation were investigated. For the invertase reaction, at 20.77% w/v sucrose feed concentrations complete conversions were achieved. The enzyme usage was 34% of the theoretical enzyme amount needed to convert an equivalent amount of sucrose over the same time period when using a conventional fermenter. The fructose rich (FRP) and glucose rich (GRP) product purities obtained were over 90%. By operating at 35% w/v sucrose feed concentration and employing the product splitting and recycling techniques, the total concentration and purity of the GRP increased from 32% w/v to 4.6% and from 92.3% to 95% respectively. The FRP concentration also increased from 1.82% w/v to 2.88% w/v. A mathematical model was developed for the combined reaction-separation and used to simulate the continuous inversion of sucrose and product separation using the SCCR-S1. In the biosynthesis of dextran studies, 52% conversion of a 2% w/v sucrose concentration feed was achieved. An average dextran molecular weight of 4 millIon was obtained in the dextran rich (DRP) product stream. The enzyme dextransucrase was purifed successfully using centrifugation and ultrafiltration techniques.

The chromatographic separation of carbohydrate mixtures

The separation performance of a semicontinuous counter-current chromatographic refiner (SCCR7), consisting of twelve 5.4 cm id x 75cm long columns packed with calcium charged cross-linked polysytrene resin (KORELA VO7C), was optimised. An industrial barley syrup was used containing 42% fructose, 52% glucose and 6% maltose and oligosaccharides. The effects of temperature, flow rates and concentration on the distribution coefficients were evaluated and quantified by deriving general relationships. The effects of flow rates, feed composition and concentration on the separation performance of the SCCR7 were identified and general relationships between them and the switch time, which was found to be the controlling parameter, were developed. Fructose rich (FRP) and glucose rich (GRP) product purities of 99.9% were obtained at 18.6% w/v feed concentrations. When a 66% w/v feed concentration was used and product splitting technique was employed, the throughput was 32.1 kg sugar solids/m3 resin/hr. The GRP contained less than 4.5% fructose, the FRP was over 95% pure, and the respective concentrations were 22.56 and 11.29% w/v. Over 94% of the glucose and 95.78% of the fructose in the feed were recovered in the GRP and FRP respectively. By recycling the dilute product split fractions, the GRP and FRP concentrations were increased to 25.4 and 12.96% w/v; the FRP was 90.2% pure and the GRP contained 6.69% w/v fructose. A theoretical link between batch and semicontinuous chromatographic equipments has been determined. A computer simulation was developed predicting successfully the purging concentration profiles at `pseudo-equilibrium', and also certain system design parameters. An important further aspect of the work has been to study the behaviour of chromatographic bioreactor-separators. Such batch systems of 5.4cm id and lengths varying between 30 and 230cm, were used to investigate the effect of scaling up on the conversion of sucrose into dextran and fructose in the presence of the dextransucrase enzyme. Conversions of over 80% were achieved at 4 hr sucrose residence times. The crude dextransucrase was purified using centrifugation, ultrafiltration and cross-flow microfiltration techniques. Better enzyme stability was obtained by first separating the non-solid impurities using cross-flow microfiltration, and then removing the cells from the enzyme immediately before use by continuous centrifugation.

Resonant gradient force on atom interacting with laser field

The gradient force, as a function of position and velocity, is derived for a two-level atom interacting with a standing-wave laser field. Basing on optical Bloch equations, the numerical solutions for the gradient force f_(|_;n) (n = 0, 1, 2, 3, 4, ...) pointing in the direction of the transverse of the laser beam are given. It is shown the higher order gradient force plays important role at strong intensity (G = 64), the contribution of them can not be neglected.

The investigation of the sensitivity gradient of the visual field, as a function of stimulus dynamic range, in the normal and abnormal eye

The study utilized the advanced technology provided by automated perimeters to investigate the hypothesis that patients with retinitis pigmentosa behave atypically over the dynamic range and to concurrently determine the influence of extraneous factors on the format of the normal perimetric sensitivity profile. The perimetric processing of some patients with retinitis pigmentosa was considered to be abnormal in either the temporal and/or the spatial domain. The standard size III stimulus saturated the central regions and was thus ineffective in detecting early depressions in sensitivity in these areas. When stimulus size was scaled in inverse proportion to the square root of ganglion cell receptive field density (M-scaled), isosensitive profiles did not result, although cortical representation was theoretically equivalent across the visual field. It was conjectured that this was due to variations in the ganglion cell characteristics with increasing peripheral angle, most notably spatial summation. It was concluded that the development of perimetric routines incorporating stimulus sizes adjusted in proportion to the coverage factor of retinal ganglion cells would enhance the diagnostic capacity of perimetry. Good general and local correspondence was found between perimetric sensitivity and the available retinal cell counts. Intraocular light scatter arising both from simulations and media opacities depressed perimetric sensitivity. Attenuation was greater centrally for the smaller LED stimuli, whereas the reverse was true for the larger projected stimuli. Prior perimetric experience and pupil size also demonstrated eccentricity-dependent effect on sensitivity. Practice improved perimetric sensitivity for projected stimuli at eccentricities greater than or equal to 30o; particularly in the superior region. Increase in pupil size for LED stimuli enhanced sensitivity at eccentricities greater than 10o. Conversely, microfluctuation in the accommodative response during perimetric examination and the correction of peripheral refractive error had no significant influence on perimetric sensitivity.

Computationally efficient natural gradient descent

DUE TO COPYRIGHT RESTRICTIONS ONLY AVAILABLE FOR CONSULTATION AT ASTON UNIVERSITY LIBRARY AND INFORMATION SERVICES WITH PRIOR ARRANGEMENT

Microarray-formatted clinical biomarker assay development using peptide aptamers to anterior gradient-2

Anterior gradient-2 protein was identified using proteomic technologies as a p53 inhibitor which is overexpressed in human cancers, and this protein presents a novel pro-oncogenic target with which to develop diagnostic assays for biomarker detection in clinical tissue. Combinatorial phage-peptide libraries were used to select 12 amino acid polypeptide aptamers toward anterior gradient-2 to determine whether methods can be developed to affinity purify the protein from clinical biopsies. Selecting phage aptamers through four rounds of screening on recombinant human anterior gradient-2 protein identified two classes of peptide ligand that bind to distinct epitopes on anterior gradient-2 protein in an immunoblot. Synthetic biotinylated peptide aptamers bound in an ELISA format to anterior gradient-2, and substitution mutagenesis further minimized one polypeptide aptamer to a hexapeptide core. Aptamers containing this latter consensus sequence could be used to affinity purify to homogeneity human anterior gradient-2 protein from a single clinical biopsy. The spotting of a panel of peptide aptamers onto a protein microarray matrix could be used to quantify anterior gradient-2 protein from crude clinical biopsy lysates, providing a format for quantitative screening. These data highlight the utility of peptide combinatorial libraries to acquire rapidly a high-affinity ligand that can selectively bind a target protein from a clinical biopsy and provide a technological approach for clinical biomarker assay development in an aptamer microarray format.

A variational conjugate gradient method for determining the fluid velocity of a slow viscous flow

The problem considered is that of determining the fluid velocity for linear hydrostatics Stokes flow of slow viscous fluids from measured velocity and fluid stress force on a part of the boundary of a bounded domain. A variational conjugate gradient iterative procedure is proposed based on solving a series of mixed well-posed boundary value problems for the Stokes operator and its adjoint. In order to stabilize the Cauchy problem, the iterations are ceased according to an optimal order discrepancy principle stopping criterion. Numerical results obtained using the boundary element method confirm that the procedure produces a convergent and stable numerical solution.

A Gradient-Type Optimization Technique for the Optimal Control for Schrodinger Equations

In this paper, we are considered with the optimal control of a schrodinger equation. Based on the formulation for the variation of the cost functional, a gradient-type optimization technique utilizing the finite difference method is then developed to solve the constrained optimization problem. Finally, a numerical example is given and the results show that the method of solution is robust.

Sucrose cryo-protection facilitates imaging of whole eye sections by MALDI mass spectrometry

Sucrose is used as a cryo-preservation agent on large mammalian eyes post formalin fixation and is shown to reduce freezing artefacts allowing the collection of 12-μm thick sections from these large aqueous samples. The suitability of this technique for use in MALDI imaging experiments is demonstrated by the acquisition of the first images of lipid distributions within whole sagittal porcine eye sections. © 2012 John Wiley & Sons, Ltd.

Modified lipids from LDL in the blood during mid-life increase blood brain barrier permeability

Low density lipoprotein levels (LDL) are consistently elevated in cardiovascular disease. It has been suggested that those with high circulating LDL levels in mid-life may be susceptible to develop neurodegenerative diseases in later life. In the circulation, high levels of LDL are associated with increased oxidative modification (oxLDL) and nitration. We have investigated the hypothesis that disruption of blood brain barrier function by oxLDL and their lipids may increase risk of neurodegeneration in later life and that statin intervention in mid-life can mitigate the neurodegenerative effects of hyperlipidaemia. Blood from statin-naïve, normo- and hyperlipidaemic subjects (n=10/group) was collected at baseline. Hyperlipidaemic subjects received statin-intervention whereas normolipidaemic subjects did not prior to a second blood sampling, taken after 3 months. The intervention will be completed in June 2013. Plasma was separated by centrifugation (200g, 30min) and LDL was isolated by potassium bromide density gradient ultracentrifugation. Total homocysteine, LDL cholesterol, 8-isoprostane F2α levels were measured in plasma using commercial kits. LDL were analysed by agarose gel electrophoresis. LDL-lipids were extracted by partitioning in 1:1 chloroform:methanol (v/v) and conjugated to fatty acid free-BSA in serum-free EGM-2 medium (4hrs, 370C) for co-culture with human microvascular endothelial cells (HMVEC). HMVEC were maintained on polycarbonate inserts for two weeks to create a microvascular barrier. Change in barrier permeability was measured by trans-endothelial electrical resistance (TER), FITC-dextran permeability and immunohistochemistry. HMVEC glutathione (GSH) levels were measured after 2 hours by GSH-glo assay. LDL isolated from statin-naïve hyperlipidaemic subjects had higher mobility by agarose gel electrophoresis (Rf;0.53±0.06) and plasma 8-isoprostane F2α (43.5±8.42 pg/ml) compared to control subjects (0.46±0.05 and 24.2±5.37 pg/ml; p<0.05). Compared to HMVEC treatment with the LDL-lipids (5μM) from normolipidaemic subjects, LDL-lipids from hyperlipidaemic subjects increased barrier permeability (103.4±12.5 Ωcm2 v 66.7±7.3 Ωcm2,P<0.01) and decreased GSH (18.5 nmol/mg v 12.3 nmol/mg; untreated cells 26.2±3.6 nmol/mg).