Bayesian classification with Gaussian processes

We consider the problem of assigning an input vector <span class='mathrm'>bfx</span> to one of <span class='mathrm'>m</span> classes by predicting <span class='mathrm'>P(c|bfx)</span> for <span class='mathrm'>c = 1, ldots, m</span>. For a two-class problem, the probability of class 1 given <span class='mathrm'>bfx</span> is estimated by <span class='mathrm'>s(y(bfx))</span>, where <span class='mathrm'>s(y) = 1/(1 + e^-y)</span>. A Gaussian process prior is placed on <span class='mathrm'>y(bfx)</span>, and is combined with the training data to obtain predictions for new <span class='mathrm'>bfx</span> points. We provide a Bayesian treatment, integrating over uncertainty in <span class='mathrm'>y</span> and in the parameters that control the Gaussian process prior; the necessary integration over <span class='mathrm'>y</span> is carried out using Laplace's approximation. The method is generalized to multi-class problems <span class='mathrm'>(m >2)</span> using the softmax function. We demonstrate the effectiveness of the method on a number of datasets.