Learning with noise and regularizers in multilayer neural networks

We study the effect of two types of noise, data noise and model noise, in an on-line gradient-descent learning scenario for general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labeled by a two-layer teacher network with an arbitrary number of hidden units. Data is then corrupted by Gaussian noise affecting either the output or the model itself. We examine the effect of both types of noise on the evolution of order parameters and the generalization error in various phases of the learning process.

Training with noise is equivalent to Tikhonov regularization

It is well known that the addition of noise to the input data of a neural network during training can, in some circumstances, lead to significant improvements in generalization performance. Previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function. However, the regularization term, which involves second derivatives of the error function, is not bounded below, and so can lead to difficulties if used directly in a learning algorithm based on error minimization. In this paper we show that, for the purposes of network training, the regularization term can be reduced to a positive definite form which involves only first derivatives of the network mapping. For a sum-of-squares error function, the regularization term belongs to the class of generalized Tikhonov regularizers. Direct minimization of the regularized error function provides a practical alternative to training with noise.

Learning with regularizers in multilayer neural networks

We study the effect of regularization in an on-line gradient-descent learning scenario for a general two-layer student network with an arbitrary number of hidden units. Training examples are randomly drawn input vectors labelled by a two-layer teacher network with an arbitrary number of hidden units which may be corrupted by Gaussian output noise. We examine the effect of weight decay regularization on the dynamical evolution of the order parameters and generalization error in various phases of the learning process, in both noiseless and noisy scenarios.

On-line learning with adaptive back-propagation in two-layer networks

An adaptive back-propagation algorithm parameterized by an inverse temperature 1/T is studied and compared with gradient descent (standard back-propagation) for on-line learning in two-layer neural networks with an arbitrary number of hidden units. Within a statistical mechanics framework, we analyse these learning algorithms in both the symmetric and the convergence phase for finite learning rates in the case of uncorrelated teachers of similar but arbitrary length T. These analyses show that adaptive back-propagation results generally in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.

Dynamics of learning with restricted training sets

The dynamics of supervised learning in layered neural networks were studied in the regime where the size of the training set is proportional to the number of inputs. The evolution of macroscopic observables, including the two relevant performance measures can be predicted by using the dynamical replica theory. Three approximation schemes aimed at eliminating the need to solve a functional saddle-point equation at each time step have been derived.

Visual data mining: integrating machine learning with information visualization

Today, the data available to tackle many scientific challenges is vast in quantity and diverse in nature. The exploration of heterogeneous information spaces requires suitable mining algorithms as well as effective visual interfaces. Most existing systems concentrate either on mining algorithms or on visualization techniques. Though visual methods developed in information visualization have been helpful, for improved understanding of a complex large high-dimensional dataset, there is a need for an effective projection of such a dataset onto a lower-dimension (2D or 3D) manifold. This paper introduces a flexible visual data mining framework which combines advanced projection algorithms developed in the machine learning domain and visual techniques developed in the information visualization domain. The framework follows Shneiderman’s mantra to provide an effective user interface. The advantage of such an interface is that the user is directly involved in the data mining process. We integrate principled projection methods, such as Generative Topographic Mapping (GTM) and Hierarchical GTM (HGTM), with powerful visual techniques, such as magnification factors, directional curvatures, parallel coordinates, billboarding, and user interaction facilities, to provide an integrated visual data mining framework. Results on a real life high-dimensional dataset from the chemoinformatics domain are also reported and discussed. Projection results of GTM are analytically compared with the projection results from other traditional projection methods, and it is also shown that the HGTM algorithm provides additional value for large datasets. The computational complexity of these algorithms is discussed to demonstrate their suitability for the visual data mining framework.

Computing with noise: phase transitions in Boolean formulas

Computing circuits composed of noisy logical gates and their ability to represent arbitrary Boolean functions with a given level of error are investigated within a statistical mechanics setting. Existing bounds on their performance are straightforwardly retrieved, generalized, and identified as the corresponding typical-case phase transitions. Results on error rates, function depth, and sensitivity, and their dependence on the gate-type and noise model used are also obtained.

Learning with other health professions in the United Kingdom MPharm degree:Multidisciplinary and placement education

This paper reviews the approach to multidisciplinary and placement education in UK schools of pharmacy. The methodology involved triangulation of course documentation, staff interviews and a final year student survey. Staff members were supportive of multidisciplinary learning. The advantages were development of a wider appreciation of the students? future professional role and better understanding of the roles of other professional groups. The barriers were logistics (student numbers; multiple sites; different timetables), the achievement of balanced numbers between disciplines and engagement of students from all participating disciplines. Placement education was offered by all schools, predominantly in hospital settings. Key problems were funding and the lack of staff resources. Currently, multidisciplinary learning within the UK for pharmacy students is inadequate and is coupled with relatively low levels of placement education. In order for things to change, there should be a review of funding and support from government and the private sector employers.

Protein-protein interactions classification from text via local learning with class priors

Text classification is essential for narrowing down the number of documents relevant to a particular topic for further pursual, especially when searching through large biomedical databases. Protein-protein interactions are an example of such a topic with databases being devoted specifically to them. This paper proposed a semi-supervised learning algorithm via local learning with class priors (LL-CP) for biomedical text classification where unlabeled data points are classified in a vector space based on their proximity to labeled nodes. The algorithm has been evaluated on a corpus of biomedical documents to identify abstracts containing information about protein-protein interactions with promising results. Experimental results show that LL-CP outperforms the traditional semisupervised learning algorithms such as SVMand it also performs better than local learning without incorporating class priors.

Learning with a strategic management simulation game:a case study

The use of simulation games as a pedagogic method is well established though its effective use is context-driven. This study adds to the increasing growing body of empirical evidence of the effectiveness of simulation games but more importantly emphasises why by explaining the instructional design implemented reflecting best practices. This multi-method study finds evidence that student learning was enhanced through the use of simulation games, reflected in the two key themes; simulation games as a catalyst for learning and simulation games as a vehicle for learning. In so doing the research provides one of the few empirically based studies that support simulation games in enhancing learning and, more importantly, contextualizes the enhancement in terms of the instructional design of the curriculum. This research should prove valuable for those with an academic interest in the use of simulation games and management educators who use, or are considering its use. Further, the findings contribute to the academic debate concerning the effective implementation of simulation game-based training in business and management education.

Semisupervised learning of hierarchical latent trait models for data visualization

Recently, we have developed the hierarchical Generative Topographic Mapping (HGTM), an interactive method for visualization of large high-dimensional real-valued data sets. In this paper, we propose a more general visualization system by extending HGTM in three ways, which allows the user to visualize a wider range of data sets and better support the model development process. 1) We integrate HGTM with noise models from the exponential family of distributions. The basic building block is the Latent Trait Model (LTM). This enables us to visualize data of inherently discrete nature, e.g., collections of documents, in a hierarchical manner. 2) We give the user a choice of initializing the child plots of the current plot in either interactive, or automatic mode. In the interactive mode, the user selects "regions of interest," whereas in the automatic mode, an unsupervised minimum message length (MML)-inspired construction of a mixture of LTMs is employed. The unsupervised construction is particularly useful when high-level plots are covered with dense clusters of highly overlapping data projections, making it difficult to use the interactive mode. Such a situation often arises when visualizing large data sets. 3) We derive general formulas for magnification factors in latent trait models. Magnification factors are a useful tool to improve our understanding of the visualization plots, since they can highlight the boundaries between data clusters. We illustrate our approach on a toy example and evaluate it on three more complex real data sets. © 2005 IEEE.

Lifelong learning and environmental sustainability

The current policy focus on lifelong learning ensures a gendered and class-based skills-driven agenda, with lifelong learners expected to become neo-liberal subjects rather than empowered members of communities. What complexities and challenges arise from attempts to align lifelong learning with social justice? What are the costs of a focus on learning which rests on economic imperatives? Lifelong learning is at the forefront of the educational arena, both nationally and internationally, although what it means is highly contestable. In recent times, lifelong learning has increasingly come to mean vocational education and training within a globalised knowledge economy.

Municipal benchmarking:organisational learning and network performance in the public sector

The main purpose of this dissertation is to assess the relation between municipal benchmarking and organisational learning with a specific emphasis on benchlearning and performance within municipalities and between groups of municipalities in the building and housing sector in the Netherlands. The first and main conclusion is that this relation exists, but that the relative success of different approaches to dimensions of change and organisational learning are a key explanatory factor for differences in the success of benchlearning. Seven other important conclusions could be derived from the empirical research. First, a combination of interpretative approaches at the group level with a mixture of hierarchical and network strategies, positively influences benchlearning. Second, interaction among professionals at the inter-organisational level strengthens benchlearning. Third, stimulating supporting factors can be seen as a more important strategy to strengthen benchlearning than pulling down barriers. Fourth, in order to facilitate benchlearning, intrinsic motivation and communication skills matter, and are supported by a high level of cooperation (i.e., team work), a flat organisational structure and interactions between individuals. Fifth, benchlearning is facilitated by a strategy that is based on a balanced use of episodic (emergent) and systemic (deliberate) forms of power. Sixth, high levels of benchlearning will be facilitated by an analyser or prospector strategic stance. Prospectors and analysers reach a different learning outcome than defenders and reactors. Whereas analysers and prospectors are willing to change policies when it is perceived as necessary, the strategic stances of defenders and reactors result in narrow process improvements (i.e., single-loop learning). Seventh, performance improvement is influenced by functional perceptions towards performance, and these perceptions ultimately influence the elements adopted. This research shows that efforts aimed at benchlearning and ultimately improved service delivery, should be directed to a multi-level and multi-dimensional approach addressing the context, content and process of dimensions of change and organisational learning.

Practice based learning approaches in collaborative design and engineering education:a case study investigation into the benefits of a crossdisciplinary practice based learning strategy

This study explores the ongoing pedagogical development of a number of undergraduate design and engineering programmes in the United Kingdom. Observations and data have been collected over several cohorts to bring a valuable perspective to the approaches piloted across two similar university departments while trialling a number of innovative learning strategies. In addition to the concurrent institutional studies the work explores curriculum design that applies the principles of Co-Design, multidisciplinary and trans disciplinary learning, with both engineering and product design students working alongside each other through a practical problem solving learning approach known as the CDIO learning initiative (Conceive, Design Implement and Operate) [1]. The study builds on previous work presented at the 2010 EPDE conference: The Effect of Personality on the Design Team: Lessons from Industry for Design Education [2]. The subsequent work presented in this paper applies the findings to mixed design and engineering team based learning, building on the insight gained through a number of industrial process case studies carried out in current design practice. Developments in delivery also aligning the CDIO principles of learning through doing into a practice based, collaborative learning experience and include elements of the TRIZ creative problem solving technique [3]. The paper will outline case studies involving a number of mixed engineering and design student projects that highlight the CDIO principles, combined with an external industrial design brief. It will compare and contrast the learning experience with that of a KTP derived student project, to examine an industry based model for student projects. In addition key areas of best practice will be presented, and student work from each mode will be discussed at the conference.

Regularization and complexity control in feed-forward networks

In this paper we consider four alternative approaches to complexity control in feed-forward networks based respectively on architecture selection, regularization, early stopping, and training with noise. We show that there are close similarities between these approaches and we argue that, for most practical applications, the technique of regularization should be the method of choice.