Computing with infinite networks

For neural networks with a wide class of weight-priors, it can be shown that in the limit of an infinite number of hidden units the prior over functions tends to a Gaussian process. In this paper analytic forms are derived for the covariance function of the Gaussian processes corresponding to networks with sigmoidal and Gaussian hidden units. This allows predictions to be made efficiently using networks with an infinite number of hidden units, and shows that, somewhat paradoxically, it may be easier to compute with infinite networks than finite ones.

Regression with input-dependent noise: A Bayesian treatment

In most treatments of the regression problem it is assumed that the distribution of target data can be described by a deterministic function of the inputs, together with additive Gaussian noise having constant variance. The use of maximum likelihood to train such models then corresponds to the minimization of a sum-of-squares error function. In many applications a more realistic model would allow the noise variance itself to depend on the input variables. However, the use of maximum likelihood to train such models would give highly biased results. In this paper we show how a Bayesian treatment can allow for an input-dependent variance while overcoming the bias of maximum likelihood.

Globally optimal on-line learning rules

We present a method for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework. This work complements previous results on locally optimal rules, where only the rate of change in generalization error was considered. We maximize the total reduction in generalization error over the whole learning process and show how the resulting rule can significantly outperform the locally optimal rule.

Adding constrained discontinuities to Gaussian process models of wind fields

Gaussian Processes provide good prior models for spatial data, but can be too smooth. In many physical situations there are discontinuities along bounding surfaces, for example fronts in near-surface wind fields. We describe a modelling method for such a constrained discontinuity and demonstrate how to infer the model parameters in wind fields with MCMC sampling.

The belief in TAP

We show the similarity between belief propagation and TAP, for decoding corrupted messages encoded by Sourlas's method. The latter is a special case of the Gallager error- correcting code, where the code word comprises products of K bits selected randomly from the original message. We examine the efficacy of solutions obtained by the two methods for various values of K and show that solutions for K>=3 may be sensitive to the choice of initial conditions in the case of unbiased patterns. Good approximations are obtained generally for K=2 and for biased patterns in the case of K>=3, especially when Nishimori's temperature is being used.

Sparse representation for Gaussian process models

We develop an approach for a sparse representation for Gaussian Process (GP) models in order to overcome the limitations of GPs caused by large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the model. Experimental results on toy examples and large real-world datasets indicate the efficiency of the approach.

General bounds on Bayes errors for regression with Gaussian processes

Based on a simple convexity lemma, we develop bounds for different types of Bayesian prediction errors for regression with Gaussian processes. The basic bounds are formulated for a fixed training set. Simpler expressions are obtained for sampling from an input distribution which equals the weight function of the covariance kernel, yielding asymptotically tight results. The results are compared with numerical experiments.

Mean field methods for classification with Gaussian processes

We discuss the Application of TAP mean field methods known from Statistical Mechanics of disordered systems to Bayesian classification with Gaussian processes. In contrast to previous applications, no knowledge about the distribution of inputs is needed. Simulation results for the Sonar data set are given.

Error-correcting codes on a Bethe-like lattice

We analyse Gallager codes by employing a simple mean-field approximation that distorts the model geometry and preserves important interactions between sites. The method naturally recovers the probability propagation decoding algorithm as a minimization of a proper free-energy. We find a thermodynamical phase transition that coincides with information theoretical upper-bounds and explain the practical code performance in terms of the free-energy landscape.

A variational approach to learning curves

We combine the replica approach from statistical physics with a variational approach to analyze learning curves analytically. We apply the method to Gaussian process regression. As a main result we derive approximative relations between empirical error measures, the generalization error and the posterior variance.

Message passing for task redistribution on sparse graphs

The problem of resource allocation in sparse graphs with real variables is studied using methods of statistical physics. An efficient distributed algorithm is devised on the basis of insight gained from the analysis and is examined using numerical simulations, showing excellent performance and full agreement with the theoretical results.

Artefactual structure from least squares multidimensional scaling

We consider the problem of illusory or artefactual structure from the visualisation of high-dimensional structureless data. In particular we examine the role of the distance metric in the use of topographic mappings based on the statistical field of multidimensional scaling. We show that the use of a squared Euclidean metric (i.e. the SSTRESs measure) gives rise to an annular structure when the input data is drawn from a high-dimensional isotropic distribution, and we provide a theoretical justification for this observation.

The design and piloting of an eBusiness design studio

This paper describes the establishment and early use of an eBusiness Design Studio in Aston Business School at Aston University which is located in Birmingham in the UK. It was originally conceived as an R&D aid as much as a teaching resource. However the bulk of its use to date has been in the support of Masters level modules (courses) and this description will focus on that application in the first instance. We have less experience of its use in an R&D context but we will make some preliminary comments on this in a later section.

The temporal binding deficit hypothesis of autism

Frith has argued that people with autism show “weak central coherence,” an unusual bias toward piecemeal rather than configurational processing and a reduction in the normal tendency to process information in context. However, the precise cognitive and neurological mechanisms underlying weak central coherence are still unknown. We propose the hypothesis that the features of autism associated with weak central coherence result from a reduction in the integration of specialized local neural networks in the brain caused by a deficit in temporal binding. The visuoperceptual anomalies associated with weak central coherence may be attributed to a reduction in synchronization of high-frequency gamma activity between local networks processing local features. The failure to utilize context in language processing in autism can be explained in similar terms. Temporal binding deficits could also contribute to executive dysfunction in autism and to some of the deficits in socialization and communication.

Reading skills are related to global, but not local, acoustic pattern perception

Although reading ability has been related to the processing of simple pitch features such as isolated transitions or continuous modulation spoken language also contains complex patterns of pitch changes that are important for establishing stress location and for segmenting the speech stream. These aspects of spoken language processing depend critically on pitch pattern (global structure) rather than on absolute pitch values (local structure). Here we show that the detection of global structure, and not local structure, is predictive of performance on measures of phonological skill and reading ability, which supports a critical importance of pitch contour processing in the acquisition of literacy.