Studies on the generalisation of Gaussian processes and Bayesian neural networks

The assessment of the reliability of systems which learn from data is a key issue to investigate thoroughly before the actual application of information processing techniques to real-world problems. Over the recent years Gaussian processes and Bayesian neural networks have come to the fore and in this thesis their generalisation capabilities are analysed from theoretical and empirical perspectives. Upper and lower bounds on the learning curve of Gaussian processes are investigated in order to estimate the amount of data required to guarantee a certain level of generalisation performance. In this thesis we analyse the effects on the bounds and the learning curve induced by the smoothness of stochastic processes described by four different covariance functions. We also explain the early, linearly-decreasing behaviour of the curves and we investigate the asymptotic behaviour of the upper bounds. The effect of the noise and the characteristic lengthscale of the stochastic process on the tightness of the bounds are also discussed. The analysis is supported by several numerical simulations. The generalisation error of a Gaussian process is affected by the dimension of the input vector and may be decreased by input-variable reduction techniques. In conventional approaches to Gaussian process regression, the positive definite matrix estimating the distance between input points is often taken diagonal. In this thesis we show that a general distance matrix is able to estimate the effective dimensionality of the regression problem as well as to discover the linear transformation from the manifest variables to the hidden-feature space, with a significant reduction of the input dimension. Numerical simulations confirm the significant superiority of the general distance matrix with respect to the diagonal one.In the thesis we also present an empirical investigation of the generalisation errors of neural networks trained by two Bayesian algorithms, the Markov Chain Monte Carlo method and the evidence framework; the neural networks have been trained on the task of labelling segmented outdoor images.

Neural networks for perpetual grouping

A number of researchers have investigated the application of neural networks to visual recognition, with much of the emphasis placed on exploiting the network's ability to generalise. However, despite the benefits of such an approach it is not at all obvious how networks can be developed which are capable of recognising objects subject to changes in rotation, translation and viewpoint. In this study, we suggest that a possible solution to this problem can be found by studying aspects of visual psychology and in particular, perceptual organisation. For example, it appears that grouping together lines based upon perceptually significant features can facilitate viewpoint independent recognition. The work presented here identifies simple grouping measures based on parallelism and connectivity and shows how it is possible to train multi-layer perceptrons (MLPs) to detect and determine the perceptual significance of any group presented. In this way, it is shown how MLPs which are trained via backpropagation to perform individual grouping tasks, can be brought together into a novel, large scale network capable of determining the perceptual significance of the whole input pattern. Finally the applicability of such significance values for recognition is investigated and results indicate that both the NILP and the Kohonen Feature Map can be trained to recognise simple shapes described in terms of perceptual significances. This study has also provided an opportunity to investigate aspects of the backpropagation algorithm, particularly the ability to generalise. In this study we report the results of various generalisation tests. In applying the backpropagation algorithm to certain problems, we found that there was a deficiency in performance with the standard learning algorithm. An improvement in performance could however, be obtained when suitable modifications were made to the algorithm. The modifications and consequent results are reported here.

A constructive learning algorithm based on back-propagation

There are been a resurgence of interest in the neural networks field in recent years, provoked in part by the discovery of the properties of multi-layer networks. This interest has in turn raised questions about the possibility of making neural network behaviour more adaptive by automating some of the processes involved. Prior to these particular questions, the process of determining the parameters and network architecture required to solve a given problem had been a time consuming activity. A number of researchers have attempted to address these issues by automating these processes, concentrating in particular on the dynamic selection of an appropriate network architecture.The work presented here specifically explores the area of automatic architecture selection; it focuses upon the design and implementation of a dynamic algorithm based on the Back-Propagation learning algorithm. The algorithm constructs a single hidden layer as the learning process proceeds using individual pattern error as the basis of unit insertion. This algorithm is applied to several problems of differing type and complexity and is found to produce near minimal architectures that are shown to have a high level of generalisation ability.

Finding structures in information networks using the affinity network

This thesis proposes a novel graphical model for inference called the Affinity Network,which displays the closeness between pairs of variables and is an alternative to Bayesian Networks and Dependency Networks. The Affinity Network shares some similarities with Bayesian Networks and Dependency Networks but avoids their heuristic and stochastic graph construction algorithms by using a message passing scheme. A comparison with the above two instances of graphical models is given for sparse discrete and continuous medical data and data taken from the UCI machine learning repository. The experimental study reveals that the Affinity Network graphs tend to be more accurate on the basis of an exhaustive search with the small datasets. Moreover, the graph construction algorithm is faster than the other two methods with huge datasets. The Affinity Network is also applied to data produced by a synchronised system. A detailed analysis and numerical investigation into this dynamical system is provided and it is shown that the Affinity Network can be used to characterise its emergent behaviour even in the presence of noise.

Learning in ultrametric committee machines

The problem of learning by examples in ultrametric committee machines (UCMs) is studied within the framework of statistical mechanics. Using the replica formalism we calculate the average generalization error in UCMs with L hidden layers and for a large enough number of units. In most of the regimes studied we find that the generalization error, as a function of the number of examples presented, develops a discontinuous drop at a critical value of the load parameter. We also find that when L>1 a number of teacher networks with the same number of hidden layers and different overlaps induce learning processes with the same critical points.

Knowledge mediation and overlapping in interfirm networks

Purpose – The literature on interfirm networks devotes scant attention to the ways collaborating firms combine and integrate the knowledge they share and to the subsequent learning outcomes. This study aims to investigate how motorsport companies use network ties to share and recombine knowledge and the learning that occurs both at the organizational and dyadic network levels. Design/methodology/approach – The paper adopts a qualitative and inductive approach with the aim of developing theory from an in-depth examination of the dyadic ties between motorsport companies and the way they share and recombine knowledge. Findings – The research shows that motorsport companies having substantial competences at managing knowledge flows do so by getting advantage of bridging ties. While bridging ties allow motorsport companies to reach distant and diverse sources of knowledge, their strengthening and the formation of relational capital facilitate the mediation and overlapping of that knowledge. Research limitations/implications – The analysis rests on a qualitative account in a single industry and does not take into account different types of inter-firm networks (e.g. alliances; constellations; consortia etc.) and governance structures. Cross-industry analyses may provide a more fine-grained picture of the practices used to recombine knowledge and the ideal composition of inter-firm ties. Practical implications – This study provides some interesting implications for scholars and managers concerned with the management of innovation activities at the interfirm level. From a managerial point of view, the recognition of the different roles played by network spanning connections is particularly salient and raises issues concerning the effective design and management of interfirm ties. Originality/value – Although much of the literature emphasizes the role of bridging ties in connecting to diverse pools of knowledge, this paper goes one step further and investigates in more depth how firms gather and combine distant and heterogeneous sources of knowledge through the use of strengthened bridging ties and a micro-context conducive to high quality relationships.

Learning to share knowledge in the Italian motorsport industry

The paper outlines a perspective on learning how to share knowledge in the context of inter-firm networks and highlights the essential role of participation in collaborative activities. This perspective suggests that knowledge sharing is not something achieved through the simple transfer of resources, but rather is an ongoing social accomplishment in which network firms constitute and re-constitute knowledge while engaging in collaborative activities. Empirical support for this view is offered by an in-depth and multiyear study of the development of collaborative relationships between a leading racing car manufacturer and its suppliers in the Italian motorsport industry. The study shows that knowledge is generated over time through the instigation of three knowledge sharing processes: the promotion of a culture of working together, co-location and the use of resident engineers, and shared education and training.

Exploring the relationships between network change and network learning

This paper relates the concept of network learning - learning by a group of organizations as a group - to change and notions of change management. Derived initially from a review of literature on organizational learning (OL) and interorganizational networks, and secondary cases of network learning, the concept was evaluated and developed through empirical investigation of five network learning episodes in the group of organizations that comprises the English prosthetics service. We argue that the notion of network learning enables a richer understanding of developments in networks over extended periods of time than can be afforded through more established concepts of change and change management alone.

Competence requirements for managing supply in interorganizational networks

The authors address the growing call for research into the management of supply networks serving the public sector. Building on prior action research, this empirical paper focuses on the management of supply in interorganizational, health sector networks identifying the competence requirements (skills, knowledge, traits, and behavioural indicators) associated with effective team performance. Drawing on empirical data, the authors present a competence framework that aims to capture a team’s tacit understanding of strategic supply management. Competence indicators are organized into six themes: network understanding; developing network position; relationship management; learning, knowledge and knowledge management; strategy formulation; strategy implementation. Finally, the relevance of the framework to boundary spanning personnel outside the purchasing function and to other organizations is considered.

Network learning:an empirically-derived conceptual model of learning by groups of organizations

Building on a previous conceptual article, we present an empirically derived model of network learning - learning by a group of organizations as a group. Based on a qualitative, longitudinal, multiple-method empirical investigation, five episodes of network learning were identified. Treating each episode as a discrete analytic case, through cross-case comparison, a model of network learning is developed which reflects the common, critical features of the episodes. The model comprises three conceptual themes relating to learning outcomes, and three conceptual themes of learning process. Although closely related to conceptualizations that emphasize the social and political character of organizational learning, the model of network learning is derived from, and specifically for, more extensive networks in which relations among numerous actors may be arms-length or collaborative, and may be expected to change over time.

Reflecting on time, timing and timeliness in public sector supply networks

Despite recent research on time (e.g. Hedaa & Törnroos, 2001), consideration of the time dimension in data collection, analysis and interpretation in research in supply networks is, to date, still limited. Drawing on a body of literature from organization studies, and empirical findings from a six-year action research programme and a related study of network learning, we reflect on time, timing and timeliness in interorganizational networks. The empirical setting is supply networks in the English health sector wherein we identify and elaborate various issues of time, within the case and in terms of research process. Our analysis is wide-ranging and multi-level, from the global (e.g. identifying the notion of life cycles) to the particular (e.g. different cycle times in supply, such as daily for deliveries and yearly for contracts). We discuss the ‘speeding up’ of inter-organizational ‘e’ time and tensions with other time demands. In closing the paper, we relate our conclusions to the future conduct of the research programme and supply research more generally, and to the practice of managing supply (in) networks.

Evolution, recurrency and kernels in learning to model inflation

This paper provides the most fully comprehensive evidence to date on whether or not monetary aggregates are valuable for forecasting US inflation in the early to mid 2000s. We explore a wide range of different definitions of money, including different methods of aggregation and different collections of included monetary assets. We use non-linear, artificial intelligence techniques, namely, recurrent neural networks, evolution strategies and kernel methods in our forecasting experiment. In the experiment, these three methodologies compete to find the best fitting US inflation forecasting models and are then compared to forecasts from a naive random walk model. The best models were non-linear autoregressive models based on kernel methods. Our findings do not provide much support for the usefulness of monetary aggregates in forecasting inflation. There is evidence in the literature that evolutionary methods can be used to evolve kernels hence our future work should combine the evolutionary and kernel methods to get the benefits of both.

A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning

To solve multi-objective problems, multiple reward signals are often scalarized into a single value and further processed using established single-objective problem solving techniques. While the field of multi-objective optimization has made many advances in applying scalarization techniques to obtain good solution trade-offs, the utility of applying these techniques in the multi-objective multi-agent learning domain has not yet been thoroughly investigated. Agents learn the value of their decisions by linearly scalarizing their reward signals at the local level, while acceptable system wide behaviour results. However, the non-linear relationship between weighting parameters of the scalarization function and the learned policy makes the discovery of system wide trade-offs time consuming. Our first contribution is a thorough analysis of well known scalarization schemes within the multi-objective multi-agent reinforcement learning setup. The analysed approaches intelligently explore the weight-space in order to find a wider range of system trade-offs. In our second contribution, we propose a novel adaptive weight algorithm which interacts with the underlying local multi-objective solvers and allows for a better coverage of the Pareto front. Our third contribution is the experimental validation of our approach by learning bi-objective policies in self-organising smart camera networks. We note that our algorithm (i) explores the objective space faster on many problem instances, (ii) obtained solutions that exhibit a larger hypervolume, while (iii) acquiring a greater spread in the objective space.

Improved adaptivity and robustness in decentralised multi-camera networks

In this paper we present increased adaptivity and robustness in distributed object tracking by multi-camera networks using a socio-economic mechanism for learning the vision graph. To build-up the vision graph autonomously within a distributed smart-camera network, we use an ant-colony inspired mechanism, which exchanges responsibility for tracking objects using Vickrey auctions. Employing the learnt vision graph allows the system to optimise its communication continuously. Since distributed smart camera networks are prone to uncertainties in individual cameras, such as failures or changes in extrinsic parameters, the vision graph should be sufficiently robust and adaptable during runtime to enable seamless tracking and optimised communication. To better reflect real smart-camera platforms and networks, we consider that communication and handover are not instantaneous, and that cameras may be added, removed or their properties changed during runtime. Using our dynamic socio-economic approach, the network is able to continue tracking objects well, despite all these uncertainties, and in some cases even with improved performance. This demonstrates the adaptivity and robustness of our approach.