On-line learning in radial basis functions networks

An analytic investigation of the average case learning and generalization properties of Radial Basis Function Networks (RBFs) is presented, utilising on-line gradient descent as the learning rule. The analytic method employed allows both the calculation of generalization error and the examination of the internal dynamics of the network. The generalization error and internal dynamics are then used to examine the role of the learning rate and the specialization of the hidden units, which gives insight into decreasing the time required for training. The realizable and over-realizable cases are studied in detail; the phase of learning in which the hidden units are unspecialized (symmetric phase) and the phase in which asymptotic convergence occurs are analyzed, and their typical properties found. Finally, simulations are performed which strongly confirm the analytic results.

Incorporating curvature information into on-line learning

We analyse the dynamics of a number of second order on-line learning algorithms training multi-layer neural networks, using the methods of statistical mechanics. We first consider on-line Newton's method, which is known to provide optimal asymptotic performance. We determine the asymptotic generalization error decay for a soft committee machine, which is shown to compare favourably with the result for standard gradient descent. Matrix momentum provides a practical approximation to this method by allowing an efficient inversion of the Hessian. We consider an idealized matrix momentum algorithm which requires access to the Hessian and find close correspondence with the dynamics of on-line Newton's method. In practice, the Hessian will not be known on-line and we therefore consider matrix momentum using a single example approximation to the Hessian. In this case good asymptotic performance may still be achieved, but the algorithm is now sensitive to parameter choice because of noise in the Hessian estimate. On-line Newton's method is not appropriate during the transient learning phase, since a suboptimal unstable fixed point of the gradient descent dynamics becomes stable for this algorithm. A principled alternative is to use Amari's natural gradient learning algorithm and we show how this method provides a significant reduction in learning time when compared to gradient descent, while retaining the asymptotic performance of on-line Newton's method.

Globally optimal on-line learning rules

We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This work complements previous results on locally optimal rules, where only the rate of change in generalization error was considered. We maximize the total reduction in generalization error over the whole learning process and show how the resulting rule can significantly outperform the locally optimal rule.

On-line learning of unrealizable tasks

The dynamics of on-line learning is investigated for structurally unrealizable tasks in the context of two-layer neural networks with an arbitrary number of hidden neurons. Within a statistical mechanics framework, a closed set of differential equations describing the learning dynamics can be derived, for the general case of unrealizable isotropic tasks. In the asymptotic regime one can solve the dynamics analytically in the limit of large number of hidden neurons, providing an analytical expression for the residual generalization error, the optimal and critical asymptotic training parameters, and the corresponding prefactor of the generalization error decay.

On-line learning in neural networks

On-line learning is one of the most powerful and commonly used techniques for training large layered networks and has been used successfully in many real-world applications. Traditional analytical methods have been recently complemented by ones from statistical physics and Bayesian statistics. This powerful combination of analytical methods provides more insight and deeper understanding of existing algorithms and leads to novel and principled proposals for their improvement. This book presents a coherent picture of the state-of-the-art in the theoretical analysis of on-line learning. An introduction relates the subject to other developments in neural networks and explains the overall picture. Surveys by leading experts in the field combine new and established material and enable non-experts to learn more about the techniques and methods used. This book, the first in the area, provides a comprehensive view of the subject and will be welcomed by mathematicians, scientists and engineers, whether in industry or academia.

The theory of on-line learning: a statistical physics approach

In this paper we review recent theoretical approaches for analysing the dynamics of on-line learning in multilayer neural networks using methods adopted from statistical physics. The analysis is based on monitoring a set of macroscopic variables from which the generalisation error can be calculated. A closed set of dynamical equations for the macroscopic variables is derived analytically and solved numerically. The theoretical framework is then employed for defining optimal learning parameters and for analysing the incorporation of second order information into the learning process using natural gradient descent and matrix-momentum based methods. We will also briefly explain an extension of the original framework for analysing the case where training examples are sampled with repetition.

A Bayesian approach to on-line learning

Online learning is discussed from the viewpoint of Bayesian statistical inference. By replacing the true posterior distribution with a simpler parametric distribution, one can define an online algorithm by a repetition of two steps: An update of the approximate posterior, when a new example arrives, and an optimal projection into the parametric family. Choosing this family to be Gaussian, we show that the algorithm achieves asymptotic efficiency. An application to learning in single layer neural networks is given.

On-line learning in a changing environment

DUE TO COPYRIGHT RESTRICTIONS ONLY AVAILABLE FOR CONSULTATION AT ASTON UNIVERSITY LIBRARY AND INFORMATION SERVICES WITH PRIOR ARRANGEMENT

Complex Adaptive On-line Learning System “Theory of Automatic Control"

The controlled from distance teaching (DT) in the system of technical education has a row of features: complication of informative content, necessity of development of simulation models and trainers for conducting of practical and laboratory employments, conducting of knowledge diagnostics on the basis of mathematical-based algorithms, organization of execution collective projects of the applied setting. For development of the process of teaching bases of fundamental discipline control system Theory of automatic control (TAC) the combined approach of optimum combination of existent programmatic instruments of support was chosen DT and own developments. The system DT TAC included: controlled from distance course (DC) of TAC, site of virtual laboratory practical works in LAB.TAC and students knowledge remote diagnostic system d-tester.

Effectiveness of on-line resources to enhance student learning and engagement

BACKGROUND For engineering graduates to be work-ready with marketable skills they must not only be well-versed with engineering science and its applications, but also able to adapt to using commercial software that is widely used in engineering practice. Hydrological/hydraulic modelling is one aspect of engineering practice which demands the ability to apply fundamentals into design and construction using software. The user manuals for such software are usually tailored for the experienced engineer but not for undergraduates who typically are novices to concepts of modelling and software tools. As the focus of a course such as Advanced Water Engineering is on the wider aspects of engineering application of hydrological and hydraulic concepts, it is ineffective for the lecturers to direct the students to user manuals as students have neither the time nor the desire to sift through numerous pages in a manual. An alternative and efficient way to demonstrate the use of the software is enabling students to develop a model to simulate real-world scenario using the tools of the software and directing them to make informed decisions based on outcomes. PURPOSE Past experience of the lecturer showed that the resources available for the students left a knowledge gap leading to numerous student queries outside contact hours. The purpose of this study is to assess how effective purpose-built video resources can be in supplementing the traditional learning resources to enhance student learning. APPROACH Short-length animated video clips comprising guided step-by-step instructions were prepared using screen capture software to capture screen activity and later edited to focus on specific features using pop-up annotations; Vocal narration was purposely excluded to avoid disturbances due to noise and allow different learning paces of individual students. The video clips were made available to the students alongside the traditional resources/approaches such as in-class demonstrations, guideline notes, and tips for efficient and error-free procedural descriptions. The number of queries the lecturer received from the student cohort outside the lecture times was recorded. An anonymous survey to assess the usefulness and adequacy of the courseware was conducted. OUTCOMES While a significant decline in the number of student queries was noted, an overwhelming majority of the survey respondents confirmed the usefulness of the purpose-developed courseware. CONCLUSIONS/RECOMMENDATIONS/SUMMARY The survey and lecturer’s experience indicated that animated demonstration video clips illustrating the various steps involved in developing hydrologic and hydraulic models and simulating design scenarios is an effective supplement for traditional learning resources. Among the many advantages of the custom-made video clips as a learning resource are that they (1) highlight the aspects that are important to undergraduate learning but not available in the software manuals as the latter are designed for more mature users/learners; (2) provide short, to-the point communication in a step-by-step manner; (3) allow students flexibility to self-learn at their own pace; (4) enhance student learning; and (5) enable time savings for the lecturer in the long term by avoiding queries of a repetitive nature. It is expected that these newly developed resources will be improved to incorporate students’ suggestions before being offered to future cohorts of students. The concept can also be expanded to other relevant courses where animated demonstrations of key modelling steps are beneficial to student learning.

A formação continuada de professores do ensino superior para a atuação docente on-line: desafios e possibilidades

Pós-graduação em Educação - FCT

Enhancing learning in dance technique through on-line-mediated reflective practice

‘Practice makes perfect’ expresses the common misconception that repetitive practice without appropriate feed-back will deliver improvement in tasks being practised. This paper explores the implementation of a student-driven feed-back mechanism and shows how functional and aesthetic understanding can be progressively enhanced through reflective practice. More efficient practice of clearly understood tasks will enhance dance training outcomes. We were looking for ways to improve teaching efficiency, effectiveness of the students’ practice in the studio and application of safe dance practices. We devised a web-based on-line format, ‘Performing Reflective Practice’, designed to augment and refine studio practice. Only perfect practice makes perfect!