Learning from Geometry

Subspaces and manifolds are two powerful models for high dimensional signals. Subspaces model linear correlation and are a good fit to signals generated by physical systems, such as frontal images of human faces and multiple sources impinging at an antenna array. Manifolds model sources that are not linearly correlated, but where signals are determined by a small number of parameters. Examples are images of human faces under different poses or expressions, and handwritten digits with varying styles. However, there will always be some degree of model mismatch between the subspace or manifold model and the true statistics of the source. This dissertation exploits subspace and manifold models as prior information in various signal processing and machine learning tasks.

A near-low-rank Gaussian mixture model measures proximity to a union of linear or affine subspaces. This simple model can effectively capture the signal distribution when each class is near a subspace. This dissertation studies how the pairwise geometry between these subspaces affects classification performance. When model mismatch is vanishingly small, the probability of misclassification is determined by the product of the sines of the principal angles between subspaces. When the model mismatch is more significant, the probability of misclassification is determined by the sum of the squares of the sines of the principal angles. Reliability of classification is derived in terms of the distribution of signal energy across principal vectors. Larger principal angles lead to smaller classification error, motivating a linear transform that optimizes principal angles. This linear transformation, termed TRAIT, also preserves some specific features in each class, being complementary to a recently developed Low Rank Transform (LRT). Moreover, when the model mismatch is more significant, TRAIT shows superior performance compared to LRT.

The manifold model enforces a constraint on the freedom of data variation. Learning features that are robust to data variation is very important, especially when the size of the training set is small. A learning machine with large numbers of parameters, e.g., deep neural network, can well describe a very complicated data distribution. However, it is also more likely to be sensitive to small perturbations of the data, and to suffer from suffer from degraded performance when generalizing to unseen (test) data.

From the perspective of complexity of function classes, such a learning machine has a huge capacity (complexity), which tends to overfit. The manifold model provides us with a way of regularizing the learning machine, so as to reduce the generalization error, therefore mitigate overfiting. Two different overfiting-preventing approaches are proposed, one from the perspective of data variation, the other from capacity/complexity control. In the first approach, the learning machine is encouraged to make decisions that vary smoothly for data points in local neighborhoods on the manifold. In the second approach, a graph adjacency matrix is derived for the manifold, and the learned features are encouraged to be aligned with the principal components of this adjacency matrix. Experimental results on benchmark datasets are demonstrated, showing an obvious advantage of the proposed approaches when the training set is small.

Stochastic optimization makes it possible to track a slowly varying subspace underlying streaming data. By approximating local neighborhoods using affine subspaces, a slowly varying manifold can be efficiently tracked as well, even with corrupted and noisy data. The more the local neighborhoods, the better the approximation, but the higher the computational complexity. A multiscale approximation scheme is proposed, where the local approximating subspaces are organized in a tree structure. Splitting and merging of the tree nodes then allows efficient control of the number of neighbourhoods. Deviation (of each datum) from the learned model is estimated, yielding a series of statistics for anomaly detection. This framework extends the classical {\em changepoint detection} technique, which only works for one dimensional signals. Simulations and experiments highlight the robustness and efficacy of the proposed approach in detecting an abrupt change in an otherwise slowly varying low-dimensional manifold.

Degeneracy of velocity constraints in rigid body systems

The equations governing the dynamics of rigid body systems with velocity constraints are singular at degenerate configurations in the constraint distribution. In this report, we describe the causes of singularities in the constraint distribution of interconnected rigid body systems with smooth configuration manifolds. A convention of defining primary velocity constraints in terms of orthogonal complements of one-dimensional subspaces is introduced. Using this convention, linear maps are defined and used to describe the space of allowable velocities of a rigid body. Through the definition of these maps, we present a condition for non-degeneracy of velocity constraints in terms of the one dimensional subspaces defining the primary velocity constraints. A method for defining the constraint subspace and distribution in terms of linear maps is presented. Using these maps, the constraint distribution is shown to be singular at configuration where there is an increase in its dimension.

Orbital and strongly orbital spaces

We say that a (countably dimensional) topological vector space X is orbital if there is T∈L(X) and a vector x∈X such that X is the linear span of the orbit {Tnx:n=0,1,…}. We say that X is strongly orbital if, additionally, x can be chosen to be a hypercyclic vector for T. Of course, X can be orbital only if the algebraic dimension of X is finite or infinite countable. We characterize orbital and strongly orbital metrizable locally convex spaces. We also show that every countably dimensional metrizable locally convex space X does not have the invariant subset property. That is, there is T∈L(X) such that every non-zero x∈X is a hypercyclic vector for T. Finally, assuming the Continuum Hypothesis, we construct a complete strongly orbital locally convex space.

As a byproduct of our constructions, we determine the number of isomorphism classes in the set of dense countably dimensional subspaces of any given separable infinite dimensional Fréchet space X. For instance, in X=ℓ2×ω, there are exactly 3 pairwise non-isomorphic (as topological vector spaces) dense countably dimensional subspaces.

The tetrahexahedric angular Calogero model

The spherical reduction of the rational Calogero model (of type A n−1 and after removing the center of mass) is considered as a maximally superintegrable quantum system, which describes a particle on the (n−2)-sphere subject to a very particular potential. We present a detailed analysis of the simplest non-separable case, n=4, whose potential is singular at the edges of a spherical tetrahexahedron. A complete set of independent conserved charges and of Hamiltonian intertwiners is constructed, and their algebra is elucidated. They arise from the ring of polynomials in Dunkl-deformed angular momenta, by classifying the subspaces invariant and antiinvariant under all Weyl reflections, respectively.

Hilbert Space Valued Signal Plus Noise Models: Analysis of Structural Breaks under High Dimensionality and Temporal Dependence

This thesis is concerned with change point analysis for time series, i.e. with detection of structural breaks in time-ordered, random data. This long-standing research field regained popularity over the last few years and is still undergoing, as statistical analysis in general, a transformation to high-dimensional problems. We focus on the fundamental »change in the mean« problem and provide extensions of the classical non-parametric Darling-Erdős-type cumulative sum (CUSUM) testing and estimation theory within highdimensional Hilbert space settings. In the first part we contribute to (long run) principal component based testing methods for Hilbert space valued time series under a rather broad (abrupt, epidemic, gradual, multiple) change setting and under dependence. For the dependence structure we consider either traditional m-dependence assumptions or more recently developed m-approximability conditions which cover, e.g., MA, AR and ARCH models. We derive Gumbel and Brownian bridge type approximations of the distribution of the test statistic under the null hypothesis of no change and consistency conditions under the alternative. A new formulation of the test statistic using projections on subspaces allows us to simplify the standard proof techniques and to weaken common assumptions on the covariance structure. Furthermore, we propose to adjust the principal components by an implicit estimation of a (possible) change direction. This approach adds flexibility to projection based methods, weakens typical technical conditions and provides better consistency properties under the alternative. In the second part we contribute to estimation methods for common changes in the means of panels of Hilbert space valued time series. We analyze weighted CUSUM estimates within a recently proposed »high-dimensional low sample size (HDLSS)« framework, where the sample size is fixed but the number of panels increases. We derive sharp conditions on »pointwise asymptotic accuracy« or »uniform asymptotic accuracy« of those estimates in terms of the weighting function. Particularly, we prove that a covariance-based correction of Darling-Erdős-type CUSUM estimates is required to guarantee uniform asymptotic accuracy under moderate dependence conditions within panels and that these conditions are fulfilled, e.g., by any MA(1) time series. As a counterexample we show that for AR(1) time series, close to the non-stationary case, the dependence is too strong and uniform asymptotic accuracy cannot be ensured. Finally, we conduct simulations to demonstrate that our results are practically applicable and that our methodological suggestions are advantageous.