Dynamics of on-line gradient descent learning for multilayer neural networks

We consider the problem of on-line gradient descent learning for general two-layer neural networks. An analytic solution is presented and used to investigate the role of the learning rate in controlling the evolution and convergence of the learning process.

Transients and asymptotics of natural gradient learning

We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.

Natural gradient matrix momentum

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.

A generative model for separating illumination and reflectance from images

It is well known that even slight changes in nonuniform illumination lead to a large image variability and are crucial for many visual tasks. This paper presents a new ICA related probabilistic model where the number of sources exceeds the number of sensors to perform an image segmentation and illumination removal, simultaneously. We model illumination and reflectance in log space by a generalized autoregressive process and Hidden Gaussian Markov random field, respectively. The model ability to deal with segmentation of illuminated images is compared with a Canny edge detector and homomorphic filtering. We apply the model to two problems: synthetic image segmentation and sea surface pollution detection from intensity images.

Blurred edges look faint, and faint edges look sharp:The effect of a gradient threshold in a multi-scale edge coding model

A multi-scale model of edge coding based on normalized Gaussian derivative filters successfully predicts perceived scale (blur) for a wide variety of edge profiles [Georgeson, M. A., May, K. A., Freeman, T. C. A., & Hesse, G. S. (in press). From filters to features: Scale-space analysis of edge and blur coding in human vision. Journal of Vision]. Our model spatially differentiates the luminance profile, half-wave rectifies the 1st derivative, and then differentiates twice more, to give the 3rd derivative of all regions with a positive gradient. This process is implemented by a set of Gaussian derivative filters with a range of scales. Peaks in the inverted normalized 3rd derivative across space and scale indicate the positions and scales of the edges. The edge contrast can be estimated from the height of the peak. The model provides a veridical estimate of the scale and contrast of edges that have a Gaussian integral profile. Therefore, since scale and contrast are independent stimulus parameters, the model predicts that the perceived value of either of these parameters should be unaffected by changes in the other. This prediction was found to be incorrect: reducing the contrast of an edge made it look sharper, and increasing its scale led to a decrease in the perceived contrast. Our model can account for these effects when the simple half-wave rectifier after the 1st derivative is replaced by a smoothed threshold function described by two parameters. For each subject, one pair of parameters provided a satisfactory fit to the data from all the experiments presented here and in the accompanying paper [May, K. A. & Georgeson, M. A. (2007). Added luminance ramp alters perceived edge blur and contrast: A critical test for derivative-based models of edge coding. Vision Research, 47, 1721-1731]. Thus, when we allow for the visual system's insensitivity to very shallow luminance gradients, our multi-scale model can be extended to edge coding over a wide range of contrasts and blurs. © 2007 Elsevier Ltd. All rights reserved.

What is second-order vision for? Discriminating illumination versus material changes

The human visual system is sensitive to second-order modulations of the local contrast (CM) or amplitude (AM) of a carrier signal. Second-order cues are detected independently of first-order luminance signals; however, it is not clear why vision should benet from second-order sensitivity. Analysis of the first-and second-order contents of natural images suggests that these cues tend to occur together, but their phase relationship varies. We have shown that in-phase combinations of LM and AM are perceived as a shaded corrugated surface whereas the anti-phase combination can be seen as corrugated when presented alone or as a flat material change when presented in a plaid containing the in-phase cue. We now extend these findings using new stimulus types and a novel haptic matching task. We also introduce a computational model based on initially separate first-and second-order channels that are combined within orientation and subsequently across orientation to produce a shading signal. Contrast gain control allows the LM + AM cue to suppress responses to the LM-AM when presented in a plaid. Thus, the model sees LM -AM as flat in these circumstances. We conclude that second-order vision plays a key role in disambiguating the origin of luminance changes within an image. © ARVO.

Sun and sky:does human vision assume a mixture of point and diffuse illumination when interpreting shape-from-shading?

People readily perceive smooth luminance variations as being due to the shading produced by undulations of a 3-D surface (shape-from-shading). In doing so, the visual system must simultaneously estimate the shape of the surface and the nature of the illumination. Remarkably, shape-from-shading operates even when both these properties are unknown and neither can be estimated directly from the image. In such circumstances humans are thought to adopt a default illumination model. A widely held view is that the default illuminant is a point source located above the observer's head. However, some have argued instead that the default illuminant is a diffuse source. We now present evidence that humans may adopt a flexible illumination model that includes both diffuse and point source elements. Our model estimates a direction for the point source and then weights the contribution of this source according to a bias function. For most people the preferred illuminant direction is overhead with a strong diffuse component.

Contrast- and illumination-invariant object recognition from active sensation

It has been suggested that the deleterious effect of contrast reversal on visual recognition is unique to faces, not objects. Here we show from priming, supervised category learning, and generalization that there is no such thing as general invariance of recognition of non-face objects against contrast reversal and, likewise, changes in direction of illumination. However, when recognition varies with rendering conditions, invariance may be restored, and effects of continuous learning may be reduced, by providing prior object knowledge from active sensation. Our findings suggest that the degree of contrast invariance achieved reflects functional characteristics of object representations learned in a task-dependent fashion.

Resonant gradient force on atom interacting with laser field

The gradient force, as a function of position and velocity, is derived for a two-level atom interacting with a standing-wave laser field. Basing on optical Bloch equations, the numerical solutions for the gradient force f_(|_;n) (n = 0, 1, 2, 3, 4, ...) pointing in the direction of the transverse of the laser beam are given. It is shown the higher order gradient force plays important role at strong intensity (G = 64), the contribution of them can not be neglected.

The investigation of the sensitivity gradient of the visual field, as a function of stimulus dynamic range, in the normal and abnormal eye

The study utilized the advanced technology provided by automated perimeters to investigate the hypothesis that patients with retinitis pigmentosa behave atypically over the dynamic range and to concurrently determine the influence of extraneous factors on the format of the normal perimetric sensitivity profile. The perimetric processing of some patients with retinitis pigmentosa was considered to be abnormal in either the temporal and/or the spatial domain. The standard size III stimulus saturated the central regions and was thus ineffective in detecting early depressions in sensitivity in these areas. When stimulus size was scaled in inverse proportion to the square root of ganglion cell receptive field density (M-scaled), isosensitive profiles did not result, although cortical representation was theoretically equivalent across the visual field. It was conjectured that this was due to variations in the ganglion cell characteristics with increasing peripheral angle, most notably spatial summation. It was concluded that the development of perimetric routines incorporating stimulus sizes adjusted in proportion to the coverage factor of retinal ganglion cells would enhance the diagnostic capacity of perimetry. Good general and local correspondence was found between perimetric sensitivity and the available retinal cell counts. Intraocular light scatter arising both from simulations and media opacities depressed perimetric sensitivity. Attenuation was greater centrally for the smaller LED stimuli, whereas the reverse was true for the larger projected stimuli. Prior perimetric experience and pupil size also demonstrated eccentricity-dependent effect on sensitivity. Practice improved perimetric sensitivity for projected stimuli at eccentricities greater than or equal to 30o; particularly in the superior region. Increase in pupil size for LED stimuli enhanced sensitivity at eccentricities greater than 10o. Conversely, microfluctuation in the accommodative response during perimetric examination and the correction of peripheral refractive error had no significant influence on perimetric sensitivity.

Multispectral imaging of the ocular fundus using light emitting diode illumination

We present an imaging system based on light emitting diode (LED) illumination that produces multispectral optical images of the human ocular fundus. It uses a conventional fundus camera equipped with a high power LED light source and a highly sensitive electron-multiplying charge coupled device camera. It is able to take pictures at a series of wavelengths in rapid succession at short exposure times, thereby eliminating the image shift introduced by natural eye movements (saccades). In contrast with snapshot systems the images retain full spatial resolution. The system is not suitable for applications where the full spectral resolution is required as it uses discrete wavebands for illumination. This is not a problem in retinal imaging where the use of selected wavelengths is common. The modular nature of the light source allows new wavelengths to be introduced easily and at low cost. The use of wavelength-specific LEDs as a source is preferable to white light illumination and subsequent filtering of the remitted light as it minimizes the total light exposure of the subject. The system is controlled via a graphical user interface that enables flexible control of intensity, duration, and sequencing of sources in synchrony with the camera. Our initial experiments indicate that the system can acquire multispectral image sequences of the human retina at exposure times of 0.05 s in the range of 500-620 nm with mean signal to noise ratio of 17 dB (min 11, std 4.5), making it suitable for quantitative analysis with application to the diagnosis and screening of eye diseases such as diabetic retinopathy and age-related macular degeneration.

Computationally efficient natural gradient descent

DUE TO COPYRIGHT RESTRICTIONS ONLY AVAILABLE FOR CONSULTATION AT ASTON UNIVERSITY LIBRARY AND INFORMATION SERVICES WITH PRIOR ARRANGEMENT

Microarray-formatted clinical biomarker assay development using peptide aptamers to anterior gradient-2

Anterior gradient-2 protein was identified using proteomic technologies as a p53 inhibitor which is overexpressed in human cancers, and this protein presents a novel pro-oncogenic target with which to develop diagnostic assays for biomarker detection in clinical tissue. Combinatorial phage-peptide libraries were used to select 12 amino acid polypeptide aptamers toward anterior gradient-2 to determine whether methods can be developed to affinity purify the protein from clinical biopsies. Selecting phage aptamers through four rounds of screening on recombinant human anterior gradient-2 protein identified two classes of peptide ligand that bind to distinct epitopes on anterior gradient-2 protein in an immunoblot. Synthetic biotinylated peptide aptamers bound in an ELISA format to anterior gradient-2, and substitution mutagenesis further minimized one polypeptide aptamer to a hexapeptide core. Aptamers containing this latter consensus sequence could be used to affinity purify to homogeneity human anterior gradient-2 protein from a single clinical biopsy. The spotting of a panel of peptide aptamers onto a protein microarray matrix could be used to quantify anterior gradient-2 protein from crude clinical biopsy lysates, providing a format for quantitative screening. These data highlight the utility of peptide combinatorial libraries to acquire rapidly a high-affinity ligand that can selectively bind a target protein from a clinical biopsy and provide a technological approach for clinical biomarker assay development in an aptamer microarray format.

A variational conjugate gradient method for determining the fluid velocity of a slow viscous flow

The problem considered is that of determining the fluid velocity for linear hydrostatics Stokes flow of slow viscous fluids from measured velocity and fluid stress force on a part of the boundary of a bounded domain. A variational conjugate gradient iterative procedure is proposed based on solving a series of mixed well-posed boundary value problems for the Stokes operator and its adjoint. In order to stabilize the Cauchy problem, the iterations are ceased according to an optimal order discrepancy principle stopping criterion. Numerical results obtained using the boundary element method confirm that the procedure produces a convergent and stable numerical solution.

An investigation to determine the influence of illumination sources on colour discrimination in normal subjects and smokers

Accurate colour vision testing requires using the correct illumination. With the plethora of 'daylight' lamps available, is there a cost-effective alternative to the discontinued MacBeth Easel lamp? Smoking is a known risk factor for macula degeneration. As the macula is responsible for colour discrimination, any toxin that affects it has the potential to influence colour discrimination. Aims: To find a costeffective light source for colour vision testing. To investigate the effect of smoking on colour discrimination. To explore how deuteranomalous trichromats compare with normal trichromats. Methods: Using the Ishihara colour vision test subjects were classified into the groups: 'Normal/Control', 'Smoker/Test', and 'Case Study' (subjects who failed the screening test and did not smoke). They completed the Farnsworth Munsell 100 Hue test under each of the three light sources: Phillips EcoHalo Twist (tungsten halogen - THL), Kosnic KCF07ALU/GU10-865 (compact fluorescent- CFL), and Deal Guardian Ltd. GU103X2WA4B-60 (light-emitting diode - LED) Results: 42 subjects took part in the study: 18 in the Normal/Control group, 18 in the Smoker/Test group, and 6 in the Case Study group. For the Normal/Control group the total error scores (TESs) were significantly lower with the CFL than with the THL (p = 0.017) as it was for the Case Study group (p = 0.009). No significant differences were found between the Normal/Control group and the Smoker/Test group for each light source. Decision tree analysis found pack years to be a significant variable for TES. Discussion: All three light sources were comparable with previous studies. The CFL provided better colour discrimination than the LED despite them both being 6500 K. Deuteranomalous trichromats showed a greatest deviation than normal trichromats using the LED. Conclusions: The Kosnic KCF07ALU/GU10-865 is a cost-effective alternative for colour vision testing. Smoking appears to have an effect on colour vision, but requires further investigation.