Globally optimal parameters for on-line learning in multilayer neural networks

We present a framework for calculating globally optimal parameters, within a given time frame, for on-line learning in multilayer neural networks. We demonstrate the capability of this method by computing optimal learning rates in typical learning scenarios. A similar treatment allows one to determine the relevance of related training algorithms based on modifications to the basic gradient descent rule as well as to compare different training methods.

Globally optimal on-line learning rules for multi-layer neural networks

We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This rule maximizes the total reduction in generalization error over the whole learning process. A simple example demonstrates that the locally optimal rule, which maximizes the rate of decrease in generalization error, may perform poorly in comparison.

On-line learning in radial basis functions networks

An analytic investigation of the average case learning and generalization properties of Radial Basis Function Networks (RBFs) is presented, utilising on-line gradient descent as the learning rule. The analytic method employed allows both the calculation of generalization error and the examination of the internal dynamics of the network. The generalization error and internal dynamics are then used to examine the role of the learning rate and the specialization of the hidden units, which gives insight into decreasing the time required for training. The realizable and over-realizable cases are studied in detail; the phase of learning in which the hidden units are unspecialized (symmetric phase) and the phase in which asymptotic convergence occurs are analyzed, and their typical properties found. Finally, simulations are performed which strongly confirm the analytic results.

Incorporating curvature information into on-line learning

We analyse the dynamics of a number of second order on-line learning algorithms training multi-layer neural networks, using the methods of statistical mechanics. We first consider on-line Newton's method, which is known to provide optimal asymptotic performance. We determine the asymptotic generalization error decay for a soft committee machine, which is shown to compare favourably with the result for standard gradient descent. Matrix momentum provides a practical approximation to this method by allowing an efficient inversion of the Hessian. We consider an idealized matrix momentum algorithm which requires access to the Hessian and find close correspondence with the dynamics of on-line Newton's method. In practice, the Hessian will not be known on-line and we therefore consider matrix momentum using a single example approximation to the Hessian. In this case good asymptotic performance may still be achieved, but the algorithm is now sensitive to parameter choice because of noise in the Hessian estimate. On-line Newton's method is not appropriate during the transient learning phase, since a suboptimal unstable fixed point of the gradient descent dynamics becomes stable for this algorithm. A principled alternative is to use Amari's natural gradient learning algorithm and we show how this method provides a significant reduction in learning time when compared to gradient descent, while retaining the asymptotic performance of on-line Newton's method.

Globally optimal on-line learning rules

We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This work complements previous results on locally optimal rules, where only the rate of change in generalization error was considered. We maximize the total reduction in generalization error over the whole learning process and show how the resulting rule can significantly outperform the locally optimal rule.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

The development of a modified dissolution method suitable for investigating powder mixtures

A novel dissolution method was developed, suitable for powder mixtures, based on the USP basket apparatus. The baskets were modified such that the powder mixtures were retained within the baskets and not dispersed, a potential difficulty that may arise when using conventional USP basket and paddle apparatus. The advantages of this method were that the components of the mixtures were maintained in close proximity, maximizing any drug:excipient interaction and leading to more linear dissolution profiles. Two weakly acidic model drugs, ibuprofen and acetaminophen, and a selection of pharmaceutical excipients, including potential dissolution-enhancing alkalizing agents, were chosen for investigation. Dissolution profiles were obtained for simple physical mixtures. The f1 fit factor values, calculated using pure drug as the reference material, demonstrated a trend in line with expectations, with several dissolution enhancers apparent for both drugs. Also, the dissolution rates were linear over substantial parts of the profiles. For both drugs, a rank order comparison between the f1 fit factor and calculated dissolution rate, obtained from the linear section of the dissolution profile, demonstrated a correlation using a significance level of P=0.05. The method was proven to be suitable for discriminating between the effects of excipients on the dissolution of the model drugs. The method design produced dissolution profiles where the dissolution rate was linear for a substantial time, allowing determination of the dissolution rate without mathematical transformation of the data. This method may be suitable as a preliminary excipient-screening tool in the drug formulation development process.

Feasibility studies of an on-line identification technique

This thesis is concerned with the study of a non-sequential identification technique, so that it may be applied to the identification of process plant mathematical models from process measurements with the greatest degree of accuracy and reliability. In order to study the accuracy of the technique under differing conditions, simple mathematical models were set up on a parallel hybrid. computer and these models identified from input/output measurements by a small on-line digital computer. Initially, the simulated models were identified on-line. However, this method of operation was found not suitable for a thorough study of the technique due to equipment limitations. Further analysis was carried out in a large off-line computer using data generated by the small on-line computer. Hence identification was not strictly on-line. Results of the work have shovm that the identification technique may be successfully applied in practice. An optimum sampling period is suggested, together with noise level limitations for maximum accuracy. A description of a double-effect evaporator is included in this thesis. It is proposed that the next stage in the work will be the identification of a mathematical model of this evaporator using the teclmique described.

Q parameter estimation using numerical simulations for linear and nonlinear transmission systems

We compare the Q parameter obtained from scalar, semi-analytical and full vector models for realistic transmission systems. One set of systems is operated in the linear regime, while another is using solitons at high peak power. We report in detail on the different results obtained for the same system using different models. Polarisation mode dispersion is also taken into account and a novel method to average Q parameters over several independent simulation runs is described. © 2006 Elsevier B.V. All rights reserved.

10-Gb/s transmission over 100 Mm of standard fiber using 2R regeneration in an optical loop mirror

The transmission of a 10-Gb/s data stream was demonstrated experimentally over a practically unlimited distance in a standard single-mode fiber system using nonlinear optical loop mirrors as simple in-line 2R regenerators. Error-free propagation over 100 000 km has been achieved with terrestrial amplifier spacing. © 2004 IEEE.

Efficient optimisation of per-channel pre-compensation in WDM 20-Gbit/s RZ-DPSK transmission in non-slope matched submarine links

We propose a computationally efficient method to the per-channel dispersion optimisation applied to 50 GHz-spaced N × 20-Gbit/s wavelength division multiplexing return-to-zero differential phase shift keying transmission in non-zero dispersion-shifted fibre based submarine systems. Crown Copyright © 2010.

Evaluating the capacity of long haul soliton transmission

We review recent developments in the use of optical solitons for communication systems spanning transoceanic distances. The implementation of "soliton control" to alleviate the detrimental impact of effects such as amplifier noise is shown to be critical for obtaining advantages over competing technologies. The potential performance of two control strategies, namely straight line filtering and synchronous phase modulation, is examined in detail. Design diagrams are used to determine the maximum permissible amplifier spacing, which is a key determinant of system economics. To focus the enquiry, two example system spans are taken, representing transatlantic and transpacific distances. It is concluded that straight line filtering provides very little improvement over a basic design without control. However synchronous phase modulation, which may be implemented using a handful of actively driven components, provides very substantial benefits. These may be used either to extend the overall bit-rate-distance product of the system or to increase the amplifier spacing at more moderate capacities.

Unequal OTDM channels in a 2R regenerated, dispersion managed transmission system

The dynamics of switching and transmission of an optical signal comprising individual OTDM channels of unequal amplitudes in a dispersion-managed link with in-line non-linear fibre loop mirrors is investigated for the first time.

Mathematical and numerical modelling of dispersion-managed solitons, autosolitons and self-similar optical pulses

This thesis presents theoretical investigation of three topics concerned with nonlinear optical pulse propagation in optical fibres. The techniques used are mathematical analysis and numerical modelling. Firstly, dispersion-managed (DM) solitons in fibre lines employing a weak dispersion map are analysed by means of a perturbation approach. In the case of small dispersion map strengths the average pulse dynamics is described by a perturbation approach (NLS) equation. Applying a perturbation theory, based on the Inverse Scattering Transform method, an analytic expression for the envelope of the DM soliton is derived. This expression correctly predicts the power enhancement arising from the dispersion management.Secondly, autosoliton transmission in DM fibre systems with periodical in-line deployment of nonlinear optical loop mirrors (NOLMs) is investigated. The use of in-line NOLMs is addressed as a general technique for all-optical passive 2R regeneration of return-to-zero data in high speed transmission system with strong dispersion management. By system optimisation, the feasibility of ultra-long single-channel and wavelength-division multiplexed data transmission at bit-rates ³ 40 Gbit s-1 in standard fibre-based systems is demonstrated. The tolerance limits of the results are defined.Thirdly, solutions of the NLS equation with gain and normal dispersion, that describes optical pulse propagation in an amplifying medium, are examined. A self-similar parabolic solution in the energy-containing core of the pulse is matched through Painlevé functions to the linear low-amplitude tails. The analysis provides a full description of the features of high-power pulses generated in an amplifying medium.