Adaptive back-propagation in on-line learning of multilayer networks

An adaptive back-propagation algorithm is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, both numerical studies and a rigorous analysis show that the adaptive back-propagation method results in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

On-line learning with adaptive back-propagation in two-layer networks

An adaptive back-propagation algorithm parameterized by an inverse temperature 1/T is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, we analyse these learning algorithms in both the symmetric and the convergence phase for finite learning rates in the case of uncorrelated teachers of similar but arbitrary length T. These analyses show that adaptive back-propagation results generally in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

A constructive learning algorithm based on back-propagation

There are been a resurgence of interest in the neural networks field in recent years, provoked in part by the discovery of the properties of multi-layer networks. This interest has in turn raised questions about the possibility of making neural network behaviour more adaptive by automating some of the processes involved. Prior to these particular questions, the process of determining the parameters and network architecture required to solve a given problem had been a time consuming activity. A number of researchers have attempted to address these issues by automating these processes, concentrating in particular on the dynamic selection of an appropriate network architecture.The work presented here specifically explores the area of automatic architecture selection; it focuses upon the design and implementation of a dynamic algorithm based on the Back-Propagation learning algorithm. The algorithm constructs a single hidden layer as the learning process proceeds using individual pattern error as the basis of unit insertion. This algorithm is applied to several problems of differing type and complexity and is found to produce near minimal architectures that are shown to have a high level of generalisation ability.

Compensation of intra-channel nonlinear fibre impairments using simplified digital back-propagation algorithm

We investigate a digital back-propagation simplification method to enable computationally-efficient digital nonlinearity compensation for a coherently-detected 112 Gb/s polarization multiplexed quadrature phase shifted keying transmission over a 1,600 km link (20x80km) with no inline compensation. Through numerical simulation, we report up to 80% reduction in required back-propagation steps to perform nonlinear compensation, in comparison to the standard back-propagation algorithm. This method takes into account the correlation between adjacent symbols at a given instant using a weighted-average approach, and optimization of the position of nonlinear compensator stage to enable practical digital back-propagation.

Impact of longitudinal power budget in coherent transmission systems employing digital back-propagation

We report the impact of longitudinal signal power profile on the transmission performance of coherently-detected 112 Gb/s m-ary polarization multiplexed quadrature amplitude modulation system after compensation of deterministic nonlinear fibre impairments. Performance improvements up to 0.6 dB (Q(eff)) are reported for a non-uniform transmission link power profile. Further investigation reveals that the evolution of the transmission performance with power profile management is fully consistent with the parametric amplification of the amplified spontaneous emission by the signal through four-wave mixing. In particular, for a non-dispersion managed system, a single-step increment of 4 dB in the amplifier gain, with respect to a uniform gain profile, at similar to 2/3(rd) of the total reach considerably improves the transmission performance for all the formats studied. In contrary a negative-step profile, emulating a failure (gain decrease or loss increase), significantly degrades the bit-error rate.

Polarization multiplexed 16QAM transmission employing modified digital back-propagation

We experimentally demonstrate performance enhancements enabled by weighted digital back propagation method for 28 Gbaud PM-16QAM transmission systems, over a 250 km ultra-large area fibre, using only one back-propagation step for the entire link, enabling up to 3 dB improvement in power tolerance with respect to linear compensation only. We observe that this is roughly the same improvement that can be obtained with the conventional, computationally heavy, non-weighted digital back propagation compensation with one step per span. As a further benchmark, we analyze performance improvement as a function of number of steps, and show that the performance improvement saturates at approximately 20 steps per span, at which a 5 dB improvement in power tolerance is obtained with respect to linear compensation only. Furthermore, we show that coarse-step self-phase modulation compensation is inefficient in wavelength division multiplexed transmission.

Impact of signal-ASE four-wave mixing on the effectiveness of digital back-propagation in 112 Gb/s PM-QPSK systems

Limitations in the performance of coherent transmission systems employing digital back-propagation due to four-wave mixing impairments are reported for the first time. A significant performance constraint is identified, originating from four-wave mixing between signals and amplified spontaneous emission noise which induces a linear increase in the standard deviation of the received field with signal power, and linear dependence on transmission distance.

Digital back-propagation for spectrally efficient WDM 112 Gbit/s PM m-ary QAM transmission

We report the performance of coherently-detected nine-channel WDM transmission over high dispersion fibers, using polarization multiplexed m-ary quadrature amplitude modulation (m = 4, 16, 64, 256) at 112 Gbit/s. Compensation of fiber nonlinearities via digital back-propagation enables up to 10 dB improvement in maximum transmittable power and similar to 8 dB Q(eff) improvement which translates to a nine-fold enhancement in transmission reach for PM-256QAM, where the largest improvements are associated with higher-order modulation formats. We further demonstrate that even under strong nonlinear distortion the transmission reach only reduces by a factor of similar to 2.5 for a 2 unit increase in capacity (log(2)m) when full band DBP is employed, in proportion to the required back-to-back OSNR.

Nonlinearity compensation via spectral inversion and digital back-propagation:a practical approach

We report performance enhancements enabled by pre-dispersed spectral inversion equivalent to that of ideal back-propagation, with further x2 increase in reach from multi-channel compensation, with spectral inversion employed upto 400km (from mid-link) with <1dB penalties. © 2012 OSA.

Impact of longitudinal power budget in coherent transmission systems employing digital back-propagation

We report the impact of longitudinal signal power profile on the transmission performance of coherently-detected 112 Gb/s m-ary polarization multiplexed quadrature amplitude modulation system after compensation of deterministic nonlinear fibre impairments. Performance improvements up to 0.6 dB (Q(eff)) are reported for a non-uniform transmission link power profile. Further investigation reveals that the evolution of the transmission performance with power profile management is fully consistent with the parametric amplification of the amplified spontaneous emission by the signal through four-wave mixing. In particular, for a non-dispersion managed system, a single-step increment of 4 dB in the amplifier gain, with respect to a uniform gain profile, at similar to 2/3(rd) of the total reach considerably improves the transmission performance for all the formats studied. In contrary a negative-step profile, emulating a failure (gain decrease or loss increase), significantly degrades the bit-error rate.

Polarization multiplexed 16QAM transmission employing modified digital back-propagation

We experimentally demonstrate performance enhancements enabled by weighted digital back propagation method for 28 Gbaud PM-16QAM transmission systems, over a 250 km ultra-large area fibre, using only one back-propagation step for the entire link, enabling up to 3 dB improvement in power tolerance with respect to linear compensation only. We observe that this is roughly the same improvement that can be obtained with the conventional, computationally heavy, non-weighted digital back propagation compensation with one step per span. As a further benchmark, we analyze performance improvement as a function of number of steps, and show that the performance improvement saturates at approximately 20 steps per span, at which a 5 dB improvement in power tolerance is obtained with respect to linear compensation only. Furthermore, we show that coarse-step self-phase modulation compensation is inefficient in wavelength division multiplexed transmission.

Unrepeatered Nyquist PDM-16QAM transmission over 364 km using Raman amplification and multi-channel digital back-propagation

Transmission of a net 467-Gb/s PDM-16QAM Nyquist-spaced superchannel is reported with an intra-superchannel net spectral efficiency (SE) of 6.6 (b/s)/Hz, over 364-km SMF-28 ULL ultra-low loss optical fiber, enabled by bi-directional second-order Raman amplification and digital nonlinearity compensation. Multi-channel digital back-propagation (MC-DBP) was applied to compensate for nonlinear interference; an improvement of 2 dB in Q2 factor was achieved when 70-GHz DBP bandwidth was applied, allowing an increase in span length of 37 km.

Curvature-driven smoothing: a learning algorithm for feed-forward networks

The performance of feed-forward neural networks in real applications can be often be improved significantly if use is made of a-priori information. For interpolation problems this prior knowledge frequently includes smoothness requirements on the network mapping, and can be imposed by the addition to the error function of suitable regularization terms. The new error function, however, now depends on the derivatives of the network mapping, and so the standard back-propagation algorithm cannot be applied. In this paper, we derive a computationally efficient learning algorithm, for a feed-forward network of arbitrary topology, which can be used to minimize the new error function. Networks having a single hidden layer, for which the learning algorithm simplifies, are treated as a special case.

Description and training of neural network dynamics

Attractor properties of a popular discrete-time neural network model are illustrated through numerical simulations. The most complex dynamics is found to occur within particular ranges of parameters controlling the symmetry and magnitude of the weight matrix. A small network model is observed to produce fixed points, limit cycles, mode-locking, the Ruelle-Takens route to chaos, and the period-doubling route to chaos. Training algorithms for tuning this dynamical behaviour are discussed. Training can be an easy or difficult task, depending whether the problem requires the use of temporal information distributed over long time intervals. Such problems require training algorithms which can handle hidden nodes. The most prominent of these algorithms, back propagation through time, solves the temporal credit assignment problem in a way which can work only if the relevant information is distributed locally in time. The Moving Targets algorithm works for the more general case, but is computationally intensive, and prone to local minima.

The 'moving targets' training algorithm

A simple method for training the dynamical behavior of a neural network is derived. It is applicable to any training problem in discrete-time networks with arbitrary feedback. The algorithm resembles back-propagation in that an error function is minimized using a gradient-based method, but the optimization is carried out in the hidden part of state space either instead of, or in addition to weight space. Computational results are presented for some simple dynamical training problems, one of which requires response to a signal 100 time steps in the past.