Globally optimal parameters for on-line learning in multilayer neural networks

We present a framework for calculating globally optimal parameters, within a given time frame, for on-line learning in multilayer neural networks. We demonstrate the capability of this method by computing optimal learning rates in typical learning scenarios. A similar treatment allows one to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule as well as to compare different training methods.

Dynamics of on-line learning in radial basis function networks

On-line learning is examined for the radial basis function network, an important and practical type of neural network. The evolution of generalization error is calculated within a framework which allows the phenomena of the learning process, such as the specialization of the hidden units, to be analyzed. The distinct stages of training are elucidated, and the role of the learning rate described. The three most important stages of training, the symmetric phase, the symmetry-breaking phase, and the convergence phase, are analyzed in detail; the convergence phase analysis allows derivation of maximal and optimal learning rates. As well as finding the evolution of the mean system parameters, the variances of these parameters are derived and shown to be typically small. Finally, the analytic results are strongly confirmed by simulations.

On-line learning of unrealizable tasks

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

The theory of on-line learning: a statistical physics approach

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

First-line therapy with latanoprost 0.005% results in improved ocular circulation in newly diagnosed primary open-angle glaucoma patients: a prospective, 6-month, open-label study

Purpose To evaluate the effect of latanoprost 0.005% on the optic nerve head (ONH) and retinal circulation of newly diagnosed and previously untreated primary open-angle glaucoma (POAG) patients. Methods Twenty-two newly diagnosed and previously untreated POAG patients (mean age±SD: 68.38±11.92 years) were included in this longitudinal open-label study. Patients were treated with latanoprost 0.005% once a day. Intraocular pressure (IOP), systemic blood pressure (BP), mean ocular perfusion pressure (MOPP), and ocular perfusion parameters ‘volume’, ‘velocity’, and ‘flow’ measured at the optic nerve head (ONH) and retina by means of Heidelberg Retina Flowmeter system were evaluated during a 6-month follow-up period. Results Treatment with latanoprost 0.005% resulted in a significant decrease in IOP (P<0.0001) and increase in MOPP (P<0.0001). After correcting for changes in MOPP, the blood velocity measured at the ONH level was significantly higher after 6 months of treatment than at baseline (P=0.0310). In addition, blood volume and flow measured at the peripapillary retina level improved after 3 and 6 months of treatment (P=0.0170; P=0.0260, and P=0.0170; P=0.0240 respectively). Conclusion Previously untreated POAG patients exhibit reduced IOP, increased MOPP and improved ocular perfusion at the ONH and retina levels when treated with Latanoprost 0.005%. These effects could be beneficial for glaucoma patients suffering from ocular vascular dysregulation.

Modelling and benchmarking business processes: The supply-line example

Benchmarking techniques have evolved over the years since Xerox’s pioneering visits to Japan in the late 1970s. The focus of benchmarking has also shifted during this period. By tracing in detail the evolution of benchmarking in one specific area of business activity, supply and distribution management, as seen by the participants in that evolution, creates a picture of a movement from single function, cost-focused, competitive benchmarking, through cross-functional, cross-sectoral, value-oriented benchmarking to process benchmarking. As process efficiency and effectiveness become the primary foci of benchmarking activities, the measurement parameters used to benchmark performance converge with the factors used in business process modelling. The possibility is therefore emerging of modelling business processes and then feeding the models with actual data from benchmarking exercises. This would overcome the most common criticism of benchmarking, namely that it intrinsically lacks the ability to move beyond current best practice. In fact the combined power of modelling and benchmarking may prove to be the basic building block of informed business process re-engineering.

Product cipher negotiation with on-line evaluation for private communication over computer networks

A method is proposed to offer privacy in computer communications, using symmetric product block ciphers. The security protocol involved a cipher negotiation stage, in which two communicating parties select privately a cipher from a public cipher space. The cipher negotiation process includes an on-line cipher evaluation stage, in which the cryptographic strength of the proposed cipher is estimated. The cryptographic strength of the ciphers is measured by confusion and diffusion. A method is proposed to describe quantitatively these two properties. For the calculation of confusion and diffusion a number of parameters are defined, such as the confusion and diffusion matrices and the marginal diffusion. These parameters involve computationally intensive calculations that are performed off-line, before any communication takes place. Once they are calculated, they are used to obtain estimation equations, which are used for on-line, fast evaluation of the confusion and diffusion of the negotiated cipher. A technique proposed in this thesis describes how to calculate the parameters and how to use the results for fast estimation of confusion and diffusion for any cipher instance within the defined cipher space.

A study of the growth and differentiation of the human adenocarcinoma of the colon cell line HT-29

The HT-29 human colon adenocarcinoma cell line, like many epithelial cells, displays an undifferentiated phenotype when cultured on plastic substrata. Biochemical markers of differentiation, such as brush border associated enzymes and carcinoembryonic antigen were expressed at very low levels. The differentiation-inducing effects of the culture of HT-29 cells on collagen type I gels were evaluated, and were assessed by morphological appearance, brush border associated enzyme activities and the secretion of CEA. The effect that this more physiological environment had on their chemosensitivity to a panel of chemotherapeutic agents was determined, so as to indicate whether this system could be used to improve the selectivity of screening for novel anticancer agents. Initial studies were performed on HT-29 cells derived from cells seeded directly from plastic substrata onto the collagen gels (designated Non-PPC gels). Their time of exposure to the collagen was limited to the time course of a single experiment and the results suggested that a longer, more permanent exposure might produce a more pronounced differentiation. HT-29 cells were then passaged continuously on collagen gels for a minimum of 10 passages prior to experimentation (designated PPC gels). The same parameters were measured, and compared to those for the cells grown on plastic and on the non-passaged collagen gels (Non-PPC) from the original studies. Permanently passaged cells displayed a similar degree of morphological differentiation as the non-passaged cells, with both culture conditions resulting in a more pronounced differentiation than that achieved by culture on plastic. It was noted that the morphological differentiation observed was very heterogeneous, a situation also seen in xenografted tumours in vivo. The activity of alkaline phosphatase and the production of CEA was higher in the cells passaged on collagen (PPC) than the cells cultured on non-passaged collagen gel (Non-PPC) and plastic. The biochemical determination of aminopeptidase activity showed that collagen gel culture enhanced the activity in both non-passaged and passaged HT-29 cells above that of the cells cultured on plastic. However, immunocytochemical localization of aminopeptidase and sucrase-isomaltase of samples of cells grown on the various substrata for 7, 14, 21 and 28 days showed a reduction in both enzymes in the cells grown on collagen gels when compared to cells grown on plastic. The reason for the discrepancy between the two assays for aminopeptidase is at this stage unexplained. Although, there was evidence to suggest that the culture of HT-29 cells on collagen gels was capable of inducing morphological and biochemical markers of enterocytic differentiation, there were no differences in the chemosensitivity of the different cell groups to a panel of anticancer agents. Preliminary studies suggested that the ability of the cells to polarize by their culture on porous filter chambers without any exogenous ECM was sufficient to enhance HT-29 differentiation and the onset of differentiation was probably correlated with the production of ECM by the cells themselves.

Study of the operating regime for all-optical passive 2R regeneration of dispersion-managed RZ data at 40 Gb/s using in-line NOLMs

In this letter, we numerically demonstrate that the use of inline nonlinear optical loop mirrors in strongly dispersion-managed transmission systems dominated by pulse distortion and amplitude noise can achieve all-optical passive 2R regeneration of a 40-Gb/s return-to-zero data stream. We define the tolerance limits of this result to the parameters of the input pulses.

Auto-solitons guided by in-line nonlinear optical loop mirrors in dispersion-managed communication systems

Summary form only given. Both dispersion management and the use of a nonlinear optical loop mirror (NOLM) as a saturable absorber can improve the performance of a soliton-based communication system. Dispersion management gives the benefits of low average dispersion while allowing pulses with higher powers to propagate, which helps to suppress Gordon-Haus timing jitter without sacrificing the signal-to-noise ratio. The NOLM suppresses the buildup of amplifier spontaneous emission noise and background dispersive radiation which, if allowed to interact with the soliton, can lead to its breakup. We examine optical pulse propagation in dispersion-managed (DM) transmission system with periodically inserted in-line NOLMs. To describe basic features of the signal transmission in such lines, we develop a simple theory based on a variational approach involving Gaussian trial functions. It, has already been proved that the variational method is an extremely effective tool for description of DM solitons. In the work we manage to include in the variational description the point action of the NOLM on pulse parameters, assuming that the Gaussian pulse shape is inherently preserved by propagation through the NOLM. The obtained results are verified by direct numerical simulations

Auto-solitons guided by in-line nonlinear optical loop mirrors in dispersion-managed communication systems

Summary form only given. Both dispersion management and the use of a nonlinear optical loop mirror (NOLM) as a saturable absorber can improve the performance of a soliton-based communication system. Dispersion management gives the benefits of low average dispersion while allowing pulses with higher powers to propagate, which helps to suppress Gordon-Haus timing jitter without sacrificing the signal-to-noise ratio. The NOLM suppresses the buildup of amplifier spontaneous emission noise and background dispersive radiation which, if allowed to interact with the soliton, can lead to its breakup. We examine optical pulse propagation in dispersion-managed (DM) transmission system with periodically inserted in-line NOLMs. To describe basic features of the signal transmission in such lines, we develop a simple theory based on a variational approach involving Gaussian trial functions. It, has already been proved that the variational method is an extremely effective tool for description of DM solitons. In the work we manage to include in the variational description the point action of the NOLM on pulse parameters, assuming that the Gaussian pulse shape is inherently preserved by propagation through the NOLM. The obtained results are verified by direct numerical simulations

Regression analysis of ranked segment parameters for optic nerve head classification:a pilot study

Purpose: To determine whether curve-fitting analysis of the ranked segment distributions of topographic optic nerve head (ONH) parameters, derived using the Heidelberg Retina Tomograph (HRT), provide a more effective statistical descriptor to differentiate the normal from the glaucomatous ONH. Methods: The sample comprised of 22 normal control subjects (mean age 66.9 years; S.D. 7.8) and 22 glaucoma patients (mean age 72.1 years; S.D. 6.9) confirmed by reproducible visual field defects on the Humphrey Field Analyser. Three 10°-images of the ONH were obtained using the HRT. The mean topography image was determined and the HRT software was used to calculate the rim volume, rim area to disc area ratio, normalised rim area to disc area ratio and retinal nerve fibre cross-sectional area for each patient at 10°-sectoral intervals. The values were ranked in descending order, and each ranked-segment curve of ordered values was fitted using the least squares method. Results: There was no difference in disc area between the groups. The group mean cup-disc area ratio was significantly lower in the normal group (0.204 ± 0.16) compared with the glaucoma group (0.533 ± 0.083) (p < 0.001). The visual field indices, mean deviation and corrected pattern S.D., were significantly greater (p < 0.001) in the glaucoma group (-9.09 dB ± 3.3 and 7.91 ± 3.4, respectively) compared with the normal group (-0.15 dB ± 0.9 and 0.95 dB ± 0.8, respectively). Univariate linear regression provided the best overall fit to the ranked segment data. The equation parameters of the regression line manually applied to the normalised rim area-disc area and the rim area-disc area ratio data, correctly classified 100% of normal subjects and glaucoma patients. In this study sample, the regression analysis of ranked segment parameters method was more effective than conventional ranked segment analysis, in which glaucoma patients were misclassified in approximately 50% of cases. Further investigation in larger samples will enable the calculation of confidence intervals for normality. These reference standards will then need to be investigated for an independent sample to fully validate the technique. Conclusions: Using a curve-fitting approach to fit ranked segment curves retains information relating to the topographic nature of neural loss. Such methodology appears to overcome some of the deficiencies of conventional ranked segment analysis, and subject to validation in larger scale studies, may potentially be of clinical utility for detecting and monitoring glaucomatous damage. © 2007 The College of Optometrists.

Optical regeneration at 40 Gb/s in dispersion-managed transmission lines with in-line synchronous modulators

Numerical optimization is performed of the 40-Gb/s dispersion-managed (DM) soliton transmission system with in-line synchronous intensity modulation. Stability of DM soliton transmission results from a combined action of dispersion, nonlinearity, in-line filtering, and modulation through effective periodic bandwidth management of carrier pulses. Therefore, analysis of the multiparametric problem is typically required. A two-stage time-saving numerical optimization procedure is applied. At the first step, the regions of the stable carrier propagation are determined using theoretical models available for DM solitons, and system parameters are optimized. At the second stage, full numerical simulations are undertaken in order to verify the tolerance of optimal transmission regimes. An approach developed demonstrates feasibility of error-free transmission over 20 000 km in a transmission line composed of standard fiber and dispersion compensation fiber at 40 Gb/s.