Prediction with Gaussian processes: from linear regression to linear prediction and beyond

The main aim of this paper is to provide a tutorial on regression with Gaussian processes. We start from Bayesian linear regression, and show how by a change of viewpoint one can see this method as a Gaussian process predictor based on priors over functions, rather than on priors over parameters. This leads in to a more general discussion of Gaussian processes in section 4. Section 5 deals with further issues, including hierarchical modelling and the setting of the parameters that control the Gaussian process, the covariance functions for neural network models and the use of Gaussian processes in classification problems.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

Gaussian processes:iterative sparse approximations

In recent years there has been an increased interest in applying non-parametric methods to real-world problems. Significant research has been devoted to Gaussian processes (GPs) due to their increased flexibility when compared with parametric models. These methods use Bayesian learning, which generally leads to analytically intractable posteriors. This thesis proposes a two-step solution to construct a probabilistic approximation to the posterior. In the first step we adapt the Bayesian online learning to GPs: the final approximation to the posterior is the result of propagating the first and second moments of intermediate posteriors obtained by combining a new example with the previous approximation. The propagation of em functional forms is solved by showing the existence of a parametrisation to posterior moments that uses combinations of the kernel function at the training points, transforming the Bayesian online learning of functions into a parametric formulation. The drawback is the prohibitive quadratic scaling of the number of parameters with the size of the data, making the method inapplicable to large datasets. The second step solves the problem of the exploding parameter size and makes GPs applicable to arbitrarily large datasets. The approximation is based on a measure of distance between two GPs, the KL-divergence between GPs. This second approximation is with a constrained GP in which only a small subset of the whole training dataset is used to represent the GP. This subset is called the em Basis Vector, or BV set and the resulting GP is a sparse approximation to the true posterior. As this sparsity is based on the KL-minimisation, it is probabilistic and independent of the way the posterior approximation from the first step is obtained. We combine the sparse approximation with an extension to the Bayesian online algorithm that allows multiple iterations for each input and thus approximating a batch solution. The resulting sparse learning algorithm is a generic one: for different problems we only change the likelihood. The algorithm is applied to a variety of problems and we examine its performance both on more classical regression and classification tasks and to the data-assimilation and a simple density estimation problems.

Improved robust control of nonlinear stochastic systems using uncertain models

We introduce a technique for quantifying and then exploiting uncertainty in nonlinear stochastic control systems. The approach is suboptimal though robust and relies upon the approximation of the forward and inverse plant models by neural networks, which also estimate the intrinsic uncertainty. Sampling from the resulting Gaussian distributions of the inversion based neurocontroller allows us to introduce a control law which is demonstrably more robust than traditional adaptive controllers.

Sparse on-line Gaussian processes

We develop an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets. The method is based on a combination of a Bayesian online algorithm together with a sequential construction of a relevant subsample of the data which fully specifies the prediction of the GP model. By using an appealing parametrisation and projection techniques that use the RKHS norm, recursions for the effective parameters and a sparse Gaussian approximation of the posterior process are obtained. This allows both for a propagation of predictions as well as of Bayesian error measures. The significance and robustness of our approach is demonstrated on a variety of experiments.

Subthalamic nucleus neurones in slices from 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine-lesioned mice show irregular, dopamine-reversible firing pattern changes, but without synchronous activity

The loss of dopamine in idiopathic or animal models of Parkinson's disease induces synchronized low-frequency oscillatory burst-firing in subthalamic nucleus neurones. We sought to establish whether these firing patterns observed in vivo were preserved in slices taken from dopamine-depleted animals, thus establishing a role for the isolated subthalamic-globus pallidus complex in generating the pathological activity. Mice treated with 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP) showed significant reductions of over 90% in levels of dopamine as measured in striatum by high pressure liquid chromatography. Likewise, significant reductions in tyrosine hydroxylase immunostaining within the striatum (>90%) and tyrosine hydroxylase positive cell numbers (65%) in substantia nigra were observed. Compared with slices from intact mice, neurones in slices from MPTP-lesioned mice fired significantly more slowly (mean rate of 4.2 Hz, cf. 7.2 Hz in control) and more irregularly (mean coefficient of variation of inter-spike interval of 94.4%, cf. 37.9% in control). Application of ionotropic glutamate receptor antagonists 6-cyano-7-nitroquinoxaline-2,3-dione (CNQX) and 2-amino-5-phosphonopentanoic acid (AP5) and the GABAA receptor antagonist picrotoxin caused no change in firing pattern. Bath application of dopamine significantly increased cell firing rate and regularized the pattern of activity in cells from slices from both MPTP-treated and control animals. Although the absolute change was more modest in control slices, the maximum dopamine effect in the two groups was comparable. Indeed, when taking into account the basal firing rate, no differences in the sensitivity to dopamine were observed between these two cohorts. Furthermore, pairs of subthalamic nucleus cells showed no correlated activity in slices from either control (21 pairs) or MPTP-treated animals (20 pairs). These results indicate that the isolated but interconnected subthalamic-globus pallidus network is not itself sufficient to generate the aberrant firing patterns in dopamine-depleted animals. More likely, inputs from other regions, such as the cortex, are needed to generate pathological oscillatory activity. © 2006 IBRO.

Techniques to improve forecasting models: applications to energy demand and price

This thesis is a study of three techniques to improve performance of some standard fore-casting models, application to the energy demand and prices. We focus on forecasting demand and price one-day ahead. First, the wavelet transform was used as a pre-processing procedure with two approaches: multicomponent-forecasts and direct-forecasts. We have empirically compared these approaches and found that the former consistently outperformed the latter. Second, adaptive models were introduced to continuously update model parameters in the testing period by combining ?lters with standard forecasting methods. Among these adaptive models, the adaptive LR-GARCH model was proposed for the fi?rst time in the thesis. Third, with regard to noise distributions of the dependent variables in the forecasting models, we used either Gaussian or Student-t distributions. This thesis proposed a novel algorithm to infer parameters of Student-t noise models. The method is an extension of earlier work for models that are linear in parameters to the non-linear multilayer perceptron. Therefore, the proposed method broadens the range of models that can use a Student-t noise distribution. Because these techniques cannot stand alone, they must be combined with prediction models to improve their performance. We combined these techniques with some standard forecasting models: multilayer perceptron, radial basis functions, linear regression, and linear regression with GARCH. These techniques and forecasting models were applied to two datasets from the UK energy markets: daily electricity demand (which is stationary) and gas forward prices (non-stationary). The results showed that these techniques provided good improvement to prediction performance.

Managing uncertainty in complex stochastic models: design and emulation of a Rabies model

In this paper we present a novel method for emulating a stochastic, or random output, computer model and show its application to a complex rabies model. The method is evaluated both in terms of accuracy and computational efficiency on synthetic data and the rabies model. We address the issue of experimental design and provide empirical evidence on the effectiveness of utilizing replicate model evaluations compared to a space-filling design. We employ the Mahalanobis error measure to validate the heteroscedastic Gaussian process based emulator predictions for both the mean and (co)variance. The emulator allows efficient screening to identify important model inputs and better understanding of the complex behaviour of the rabies model.

Drug action on anxiety models with special reference to serotonergic mechanisms

A study has been made of drugs acting at 5-HT receptors on animal models of anxiety. An elevated X-maze was used as a model of anxiety for rats and the actions of various ligands for the 5-HT receptor, and its subtypes, were examined in this model. 5-HT agonists, with varying affinities for the 5-HT receptor subtypes, were demonstrated to have anxiogenic-like activity. The 5-HT2 receptor antagonists ritanserin and ketanserin exhibited an anxiolytic-like profile. The new putatuve anxiolytics ipsapirone and buspirone, which are believed to be selective for 5-HT1 receptors, were also examined. The former had an anxiolytic profile whilst the latter was without effect. Antagonism studies showed the anxiogenic response to 8-hydroxy-2-(Di-n-propylamino)tetralin (8-OH-DPAT) to be antagonised by ipsapirone, pindolol, alprenolol and para-chlorophenylalanine, but not by diazepam, ritanserin, metoprolol, ICI118,551 or buspirone. To confirm some of the results obtained in the elevated X-maze the Social Interaction Test of anxiety was used. Results in this test mirrored the effects seen with the 5-HT agonists, ipsapirone and pindolol, whilst the 5-HT2 receptor antagonists were without effect. Studies using operant conflict models of anxiety produced marginal and varying results which appear to be in agreement with recent criticisms of such models. Finally, lesions of the dorsal raphe nucleus (DRN) were performed in order to investigate the mechanisms involved in the production of the anxiogenic response to 8-OH-DPAT. Overall the results lend support to the involvement of 5-HT, and more precisely 5-HT1, receptors in the manifestation of anxiety in such animal models.

Effects of some 5-hydroxytryptamine and related ligands in anxiety models

Drugs acting at 5-HT receptors were evaluated on three animal models of anxiety. On the elevated X-maze test the majority of 5-HT1 agonists were found to be anxiogenic. However, ipsapirone was anxiolytic and buspirone and gepirone were inactive. The 5-HT2 agonist DOI and the 5-HT2 antagonist ritanserin were anxiolytic while ICI 169,369, a 5-HT2 antagonist was inactive. All 5-HT3 antagonists tested were inactive in this test, while the indirect serotomimetics zimeldine and fenfluramine were anxiogenic. Neither beta-adrenoceptor agonists nor antagonists had reproducible effects on anxiety in this model. Combined beta-1/beta-2 adrenoceptor antagonists reversed the anxiogenic effects of 8-OH-DPAT while selective beta-1 or beta-2 antagonists did not. On the social interaction model the 5-HT1 agonists 8-OH-DPAT, RU 24969 and 5-MeODMT were anxiogenic and ipsapirone was anxiolytic. The 5-HT2 agonist DOI and the beta-adrenoceptor- and 5-HT- antagonist pindolol were anxiolytic, while the 5-HT2 and 5-HT3 antagonists were inactive. In the marble burying test, the 5-HT upake inhibitors zimeldine, fluvoxamine, indalpine and citalopram, the 5-HT1B/5-HT1C agonists mCPP and TFMPP and the 5-HT2/5-HT1C agonist DOI reduced marble burying without affecting locomotor activity. 5-HT1A agonists and the 5-HT2 and 5-HT3 antagonists were without effect. Lesions of the dorsal raphe nucleus reversed the anxiogenic effects of 8-OH-DPAT in the X-maze model. The implication of these results for the understanding of the pharmacology of 5-HT in anxiety is discussed.

Mach bands and multiscale models of spatial vision:the role of first, second, and third derivative operators in encoding bars and edges

Ernst Mach observed that light or dark bands could be seen at abrupt changes of luminance gradient in the absence of peaks or troughs in luminance. Many models of feature detection share the idea that bars, lines, and Mach bands are found at peaks and troughs in the output of even-symmetric spatial filters. Our experiments assessed the appearance of Mach bands (position and width) and the probability of seeing them on a novel set of generalized Gaussian edges. Mach band probability was mainly determined by the shape of the luminance profile and increased with the sharpness of its corners, controlled by a single parameter (n). Doubling or halving the size of the images had no significant effect. Variations in contrast (20%-80%) and duration (50-300 ms) had relatively minor effects. These results rule out the idea that Mach bands depend simply on the amplitude of the second derivative, but a multiscale model, based on Gaussian-smoothed first- and second-derivative filtering, can account accurately for the probability and perceived spatial layout of the bands. A key idea is that Mach band visibility depends on the ratio of second- to first-derivative responses at peaks in the second-derivative scale-space map. This ratio is approximately scale-invariant and increases with the sharpness of the corners of the luminance ramp, as observed. The edges of Mach bands pose a surprisingly difficult challenge for models of edge detection, but a nonlinear third-derivative operation is shown to predict the locations of Mach band edges strikingly well. Mach bands thus shed new light on the role of multiscale filtering systems in feature coding. © 2012 ARVO.

Synchronized oscillations at α and θ frequencies in the lateral geniculate nucleus

In relaxed wakefulness, the EEG exhibits robust rhythms in the alpha band (8-13 Hz), which decelerate to theta (approximately 2-7 Hz) frequencies during early sleep. In animal models, these rhythms occur coherently with synchronized activity in the thalamus. However, the mechanisms of this thalamic activity are unknown. Here we show that, in slices of the lateral geniculate nucleus maintained in vitro, activation of the metabotropic glutamate receptor (mGluR) mGluR1a induces synchronized oscillations at alpha and theta frequencies that share similarities with thalamic alpha and theta rhythms recorded in vivo. These in vitro oscillations are driven by an unusual form of burst firing that is present in a subset of thalamocortical neurons and are synchronized by gap junctions. We propose that mGluR1a-induced oscillations are a potential mechanism whereby the thalamus promotes EEG alpha and theta rhythms in the intact brain.

Wind power forecasts using Gaussian processes and numerical weather prediction

Since wind at the earth's surface has an intrinsically complex and stochastic nature, accurate wind power forecasts are necessary for the safe and economic use of wind energy. In this paper, we investigated a combination of numeric and probabilistic models: a Gaussian process (GP) combined with a numerical weather prediction (NWP) model was applied to wind-power forecasting up to one day ahead. First, the wind-speed data from NWP was corrected by a GP, then, as there is always a defined limit on power generated in a wind turbine due to the turbine controlling strategy, wind power forecasts were realized by modeling the relationship between the corrected wind speed and power output using a censored GP. To validate the proposed approach, three real-world datasets were used for model training and testing. The empirical results were compared with several classical wind forecast models, and based on the mean absolute error (MAE), the proposed model provides around 9% to 14% improvement in forecasting accuracy compared to an artificial neural network (ANN) model, and nearly 17% improvement on a third dataset which is from a newly-built wind farm for which there is a limited amount of training data. © 2013 IEEE.

Short-term wind power forecasting using Gaussian processes

Since wind has an intrinsically complex and stochastic nature, accurate wind power forecasts are necessary for the safety and economics of wind energy utilization. In this paper, we investigate a combination of numeric and probabilistic models: one-day-ahead wind power forecasts were made with Gaussian Processes (GPs) applied to the outputs of a Numerical Weather Prediction (NWP) model. Firstly the wind speed data from NWP was corrected by a GP. Then, as there is always a defined limit on power generated in a wind turbine due the turbine controlling strategy, a Censored GP was used to model the relationship between the corrected wind speed and power output. To validate the proposed approach, two real world datasets were used for model construction and testing. The simulation results were compared with the persistence method and Artificial Neural Networks (ANNs); the proposed model achieves about 11% improvement in forecasting accuracy (Mean Absolute Error) compared to the ANN model on one dataset, and nearly 5% improvement on another.

Projected sequential Gaussian processes:a C++ tool for interpolation of large datasets with heterogeneous noise

Heterogeneous datasets arise naturally in most applications due to the use of a variety of sensors and measuring platforms. Such datasets can be heterogeneous in terms of the error characteristics and sensor models. Treating such data is most naturally accomplished using a Bayesian or model-based geostatistical approach; however, such methods generally scale rather badly with the size of dataset, and require computationally expensive Monte Carlo based inference. Recently within the machine learning and spatial statistics communities many papers have explored the potential of reduced rank representations of the covariance matrix, often referred to as projected or fixed rank approaches. In such methods the covariance function of the posterior process is represented by a reduced rank approximation which is chosen such that there is minimal information loss. In this paper a sequential Bayesian framework for inference in such projected processes is presented. The observations are considered one at a time which avoids the need for high dimensional integrals typically required in a Bayesian approach. A C++ library, gptk, which is part of the INTAMAP web service, is introduced which implements projected, sequential estimation and adds several novel features. In particular the library includes the ability to use a generic observation operator, or sensor model, to permit data fusion. It is also possible to cope with a range of observation error characteristics, including non-Gaussian observation errors. Inference for the covariance parameters is explored, including the impact of the projected process approximation on likelihood profiles. We illustrate the projected sequential method in application to synthetic and real datasets. Limitations and extensions are discussed. © 2010 Elsevier Ltd.