Information geometric measurements of generalisation

Neural networks can be regarded as statistical models, and can be analysed in a Bayesian framework. Generalisation is measured by the performance on independent test data drawn from the same distribution as the training data. Such performance can be quantified by the posterior average of the information divergence between the true and the model distributions. Averaging over the Bayesian posterior guarantees internal coherence; Using information divergence guarantees invariance with respect to representation. The theory generalises the least mean squares theory for linear Gaussian models to general problems of statistical estimation. The main results are: (1)~the ideal optimal estimate is always given by average over the posterior; (2)~the optimal estimate within a computational model is given by the projection of the ideal estimate to the model. This incidentally shows some currently popular methods dealing with hyperpriors are in general unnecessary and misleading. The extension of information divergence to positive normalisable measures reveals a remarkable relation between the dlt dual affine geometry of statistical manifolds and the geometry of the dual pair of Banach spaces Ld and Ldd. It therefore offers conceptual simplification to information geometry. The general conclusion on the issue of evaluating neural network learning rules and other statistical inference methods is that such evaluations are only meaningful under three assumptions: The prior P(p), describing the environment of all the problems; the divergence Dd, specifying the requirement of the task; and the model Q, specifying available computing resources.

Measurements of generalisation based on information geometry

Neural networks are statistical models and learning rules are estimators. In this paper a theory for measuring generalisation is developed by combining Bayesian decision theory with information geometry. The performance of an estimator is measured by the information divergence between the true distribution and the estimate, averaged over the Bayesian posterior. This unifies the majority of error measures currently in use. The optimal estimators also reveal some intricate interrelationships among information geometry, Banach spaces and sufficient statistics.

Some Applications of Simons’ Inequality

We survey several applications of Simons’ inequality and state related open problems. We show that if a Banach space X has a strongly sub-differentiable norm, then every bounded weakly closed subset of X is an intersection of finite union of balls.

Asplund Functions and Projectional Resolutions of the Identity

*Supported by the Grants AV ˇCR 101-97-02, 101-90-03, GA ˇCR 201-98-1449, and by the Grant of the Faculty of Civil Engineering of the Czech Technical University No. 2003.

The Space of Differences of Convex Functions on [0, 1]

∗Participant in Workshop in Linear Analysis and Probability, Texas A & M University, College Station, Texas, 2000. Research partially supported by the Edmund Landau Center for Research in Mathematical Analysis and related areas, sponsored by Minerva Foundation (Germany).

Uniformly Gâteaux Differentiable Norms in Spaces with Unconditional Basis

*Supported in part by GAˇ CR 201-98-1449 and AV 101 9003. This paper is based on a part of the author’s MSc thesis written under the supervison of Professor V. Zizler.

Paracompact Spaces and Radon Spaces

We prove that if E is a subset of a Banach space whose density is of measure zero and such that (E, weak) is a paracompact space, then (E, weak) is a Radon space of type (F ) under very general conditions.

An Example Concerning Valdivia Compact Spaces

∗ Supported by Research grants GAUK 190/96 and GAUK 1/1998

A Remark On S.M. Bates' Theorem

In his paper [1], Bates investigates the existence of nonlinear, but highly smooth, surjective operators between various classes of Banach spaces. Modifying his basic method, he obtains the following striking results.

Topological Dichotomy and Unconditional Convergence

In this paper, we give a criterion for unconditional convergence with respect to some summability methods, dealing with the topological size of the set of choices of sign providing convergence. We obtain similar results for boundedness. In particular, quasi-sure unconditional convergence implies unconditional convergence.

Boolean Rings that are Baire Spaces

∗ The present article was originally submitted for the second volume of Murcia Seminar on Functional Analysis (1989). Unfortunately it has been not possible to continue with Murcia Seminar publication anymore. For historical reasons the present vesion correspond with the original one.

Compactness in the First Baire Class and Baire-1 Operators

For a polish space M and a Banach space E let B1 (M, E) be the space of first Baire class functions from M to E, endowed with the pointwise weak topology. We study the compact subsets of B1 (M, E) and show that the fundamental results proved by Rosenthal, Bourgain, Fremlin, Talagrand and Godefroy, in case E = R, also hold true in the general case. For instance: a subset of B1 (M, E) is compact iff it is sequentially (resp. countably) compact, the convex hull of a compact bounded subset of B1 (M, E) is relatively compact, etc. We also show that our class includes Gulko compact. In the second part of the paper we examine under which conditions a bounded linear operator T : X ∗ → Y so that T |BX ∗ : (BX ∗ , w∗ ) → Y is a Baire-1 function, is a pointwise limit of a sequence (Tn ) of operators with T |BX ∗ : (BX ∗ , w∗ ) → (Y, · ) continuous for all n ∈ N. Our results in this case are connected with classical results of Choquet, Odell and Rosenthal.

Greedy Approximation with Regard to Bases and General Minimal Systems

*This research was supported by the National Science Foundation Grant DMS 0200187 and by ONR Grant N00014-96-1-1003

Isomorphism Problems for the Baire Function Spaces of Topological Spaces

Let a compact Hausdorff space X contain a non-empty perfect subset. If α < β and β is a countable ordinal, then the Banach space Bα (X) of all bounded real-valued functions of Baire class α on X is a proper subspace of the Banach space Bβ (X). In this paper it is shown that: 1. Bα (X) has a representation as C(bα X), where bα X is a compactification of the space P X – the underlying set of X in the Baire topology generated by the Gδ -sets in X. 2. If 1 ≤ α < β ≤ Ω, where Ω is the first uncountable ordinal number, then Bα (X) is uncomplemented as a closed subspace of Bβ (X). These assertions for X = [0, 1] were proved by W. G. Bade [4] and in the case when X contains an uncountable compact metrizable space – by F.K.Dashiell [9]. Our argumentation is one non-metrizable modification of both Bade’s and Dashiell’s methods.

Three Spaces Problem for Lyapunov Theorem on Vector Measure

It is proved that a Banach space X has the Lyapunov property if its subspace Y and the quotient space X/Y have it.