Incorporating curvature information into on-line learning

We analyse the dynamics of a number of second order on-line learning algorithms training multi-layer neural networks, using the methods of statistical mechanics. We first consider on-line Newton's method, which is known to provide optimal asymptotic performance. We determine the asymptotic generalization error decay for a soft committee machine, which is shown to compare favourably with the result for standard gradient descent. Matrix momentum provides a practical approximation to this method by allowing an efficient inversion of the Hessian. We consider an idealized matrix momentum algorithm which requires access to the Hessian and find close correspondence with the dynamics of on-line Newton's method. In practice, the Hessian will not be known on-line and we therefore consider matrix momentum using a single example approximation to the Hessian. In this case good asymptotic performance may still be achieved, but the algorithm is now sensitive to parameter choice because of noise in the Hessian estimate. On-line Newton's method is not appropriate during the transient learning phase, since a suboptimal unstable fixed point of the gradient descent dynamics becomes stable for this algorithm. A principled alternative is to use Amari's natural gradient learning algorithm and we show how this method provides a significant reduction in learning time when compared to gradient descent, while retaining the asymptotic performance of on-line Newton's method.

Micro-fabrication of advanced photonic devices by means of direct point-by-point femtosecond inscription in silica

We present recent results on experimental micro-fabrication and numerical modeling of advanced photonic devices by means of direct writing by femtosecond laser. Transverse inscription geometry was routinely used to inscribe and modify photonic devices based on waveguiding structures. Typically, standard commercially available fibers were used as a template with a pre-fabricated waveguide. Using a direct, point-by-point inscription by infrared femtosecond laser, a range of fiber-based photonic devices was fabricated including Fiber Bragg Gratings (FBG) and Long Period Gratings (LPG). Waveguides with a core of a couple of microns, periodic structures, and couplers have been also fabricated in planar geometry using the same method.

Micro-fabrication of advanced photonic devices by means of direct point-by-point femtosecond inscription in silica

We present recent results on experimental micro-fabrication and numerical modeling of advanced photonic devices by means of direct writing by femtosecond laser. Transverse inscription geometry was routinely used to inscribe and modify photonic devices based on waveguiding structures. Typically, standard commercially available fibers were used as a template with a pre-fabricated waveguide. Using a direct, point-by-point inscription by infrared femtosecond laser, a range of fiber-based photonic devices was fabricated including Fiber Bragg Gratings (FBG) and Long Period Gratings (LPG). Waveguides with a core of a couple of microns, periodic structures, and couplers have been also fabricated in planar geometry using the same method.