Learning from Geometry

Subspaces and manifolds are two powerful models for high dimensional signals. Subspaces model linear correlation and are a good fit to signals generated by physical systems, such as frontal images of human faces and multiple sources impinging at an antenna array. Manifolds model sources that are not linearly correlated, but where signals are determined by a small number of parameters. Examples are images of human faces under different poses or expressions, and handwritten digits with varying styles. However, there will always be some degree of model mismatch between the subspace or manifold model and the true statistics of the source. This dissertation exploits subspace and manifold models as prior information in various signal processing and machine learning tasks.

A near-low-rank Gaussian mixture model measures proximity to a union of linear or affine subspaces. This simple model can effectively capture the signal distribution when each class is near a subspace. This dissertation studies how the pairwise geometry between these subspaces affects classification performance. When model mismatch is vanishingly small, the probability of misclassification is determined by the product of the sines of the principal angles between subspaces. When the model mismatch is more significant, the probability of misclassification is determined by the sum of the squares of the sines of the principal angles. Reliability of classification is derived in terms of the distribution of signal energy across principal vectors. Larger principal angles lead to smaller classification error, motivating a linear transform that optimizes principal angles. This linear transformation, termed TRAIT, also preserves some specific features in each class, being complementary to a recently developed Low Rank Transform (LRT). Moreover, when the model mismatch is more significant, TRAIT shows superior performance compared to LRT.

The manifold model enforces a constraint on the freedom of data variation. Learning features that are robust to data variation is very important, especially when the size of the training set is small. A learning machine with large numbers of parameters, e.g., deep neural network, can well describe a very complicated data distribution. However, it is also more likely to be sensitive to small perturbations of the data, and to suffer from suffer from degraded performance when generalizing to unseen (test) data.

From the perspective of complexity of function classes, such a learning machine has a huge capacity (complexity), which tends to overfit. The manifold model provides us with a way of regularizing the learning machine, so as to reduce the generalization error, therefore mitigate overfiting. Two different overfiting-preventing approaches are proposed, one from the perspective of data variation, the other from capacity/complexity control. In the first approach, the learning machine is encouraged to make decisions that vary smoothly for data points in local neighborhoods on the manifold. In the second approach, a graph adjacency matrix is derived for the manifold, and the learned features are encouraged to be aligned with the principal components of this adjacency matrix. Experimental results on benchmark datasets are demonstrated, showing an obvious advantage of the proposed approaches when the training set is small.

Stochastic optimization makes it possible to track a slowly varying subspace underlying streaming data. By approximating local neighborhoods using affine subspaces, a slowly varying manifold can be efficiently tracked as well, even with corrupted and noisy data. The more the local neighborhoods, the better the approximation, but the higher the computational complexity. A multiscale approximation scheme is proposed, where the local approximating subspaces are organized in a tree structure. Splitting and merging of the tree nodes then allows efficient control of the number of neighbourhoods. Deviation (of each datum) from the learned model is estimated, yielding a series of statistics for anomaly detection. This framework extends the classical {\em changepoint detection} technique, which only works for one dimensional signals. Simulations and experiments highlight the robustness and efficacy of the proposed approach in detecting an abrupt change in an otherwise slowly varying low-dimensional manifold.

Degeneracy of velocity constraints in rigid body systems

The equations governing the dynamics of rigid body systems with velocity constraints are singular at degenerate configurations in the constraint distribution. In this report, we describe the causes of singularities in the constraint distribution of interconnected rigid body systems with smooth configuration manifolds. A convention of defining primary velocity constraints in terms of orthogonal complements of one-dimensional subspaces is introduced. Using this convention, linear maps are defined and used to describe the space of allowable velocities of a rigid body. Through the definition of these maps, we present a condition for non-degeneracy of velocity constraints in terms of the one dimensional subspaces defining the primary velocity constraints. A method for defining the constraint subspace and distribution in terms of linear maps is presented. Using these maps, the constraint distribution is shown to be singular at configuration where there is an increase in its dimension.

Face recognition using a unified 3D morphable model

We address the problem of 3D-assisted 2D face recognition in scenarios when the input image is subject to degradations or exhibits intra-personal variations not captured by the 3D model. The proposed solution involves a novel approach to learn a subspace spanned by perturbations caused by the missing modes of variation and image degradations, using 3D face data reconstructed from 2D images rather than 3D capture. This is accomplished by modelling the difference in the texture map of the 3D aligned input and reference images. A training set of these texture maps then defines a perturbation space which can be represented using PCA bases. Assuming that the image perturbation subspace is orthogonal to the 3D face model space, then these additive components can be recovered from an unseen input image, resulting in an improved fit of the 3D face model. The linearity of the model leads to efficient fitting. Experiments show that our method achieves very competitive face recognition performance on Multi-PIE and AR databases. We also present baseline face recognition results on a new data set exhibiting combined pose and illumination variations as well as occlusion.

Data-Driven Dynamic Modeling of Coupled Thermal and Electric Outputs of Microturbines

Microturbines are among the most successfully commercialized distributed energy resources, especially when they are used for combined heat and power generation. However, the interrelated thermal and electrical system dynamic behaviors have not been fully investigated. This is technically challenging due to the complex thermo-fluid-mechanical energy conversion processes which introduce multiple time-scale dynamics and strong nonlinearity into the analysis. To tackle this problem, this paper proposes a simplified model which can predict the coupled thermal and electric output dynamics of microturbines. Considering the time-scale difference of various dynamic processes occuring within microturbines, the electromechanical subsystem is treated as a fast quasi-linear process while the thermo-mechanical subsystem is treated as a slow process with high nonlinearity. A three-stage subspace identification method is utilized to capture the dominant dynamics and predict the electric power output. For the thermo-mechanical process, a radial basis function model trained by the particle swarm optimization method is employed to handle the strong nonlinear characteristics. Experimental tests on a Capstone C30 microturbine show that the proposed modeling method can well capture the system dynamics and produce a good prediction of the coupled thermal and electric outputs in various operating modes.

Data-Driven Dynamic Prediction of Interrelated Heat and Electric Outputs of Microturbines

Microturbines are among the most successfully commercialized distributed energy resources, especially when they are used for combined heat and power generation. However, the interrelated thermal and electrical system dynamic behaviors have not been fully investigated. This is technically challenging due to the complex thermo-fluid-mechanical energy conversion processes which introduce multiple time-scale dynamics and strong nonlinearity into the analysis. To tackle this problem, this paper proposes a simplified model which can predict the coupled thermal and electric output dynamics of microturbines. Considering the time-scale difference of various dynamic processes occuring within microturbines, the electromechanical subsystem is treated as a fast quasi-linear process while the thermo-mechanical subsystem is treated as a slow process with high nonlinearity. A three-stage subspace identification method is utilized to capture the dominant dynamics and predict the electric power output. For the thermo-mechanical process, a radial basis function model trained by the particle swarm optimization method is employed to handle the strong nonlinear characteristics. Experimental tests on a Capstone C30 microturbine show that the proposed modeling method can well capture the system dynamics and produce a good prediction of the coupled thermal and electric outputs in various operating modes.

Short-term wind power forecast based on cluster analysis and artificial neural networks

[EN]In this paper an architecture for an estimator of short-term wind farm power is proposed. The estimator is made up of a Linear Machine classifier and a set of k Multilayer Perceptrons, training each one for a specific subspace of the input space. The splitting of the input dataset into the k clusters is done using a k-means technique, obtaining the equivalent Linear Machine classifier from the cluster centroids...

New developments on VCA unmixing algorithm

Hyperspectral sensors are being developed for remote sensing applications. These sensors produce huge data volumes which require faster processing and analysis tools. Vertex component analysis (VCA) has become a very useful tool to unmix hyperspectral data. It has been successfully used to determine endmembers and unmix large hyperspectral data sets without the use of any a priori knowledge of the constituent spectra. Compared with other geometric-based approaches VCA is an efficient method from the computational point of view. In this paper we introduce new developments for VCA: 1) a new signal subspace identification method (HySime) is applied to infer the signal subspace where the data set live. This step also infers the number of endmembers present in the data set; 2) after the projection of the data set onto the signal subspace, the algorithm iteratively projects the data set onto several directions orthogonal to the subspace spanned by the endmembers already determined. The new endmember signature corresponds to these extreme of the projections. The capability of VCA to unmix large hyperspectral scenes (real or simulated), with low computational complexity, is also illustrated.

Some results on orbifold quotients and related objects

The quotient of a finite-dimensional Euclidean space by a finite linear group inherits different structures from the initial space, e.g. a topology, a metric and a piecewise linear structure. The question when such a quotient is a manifold leads to the study of finite groups generated by reflections and rotations, i.e. by orthogonal transformations whose fixed point subspace has codimension one or two. We classify such groups and thereby complete earlier results by M. A. Mikhaîlova from the 70s and 80s. Moreover, we show that a finite group is generated by reflections and) rotations if and only if the corresponding quotient is a Lipschitz-, or equivalently, a piecewise linear manifold (with boundary). For the proof of this statement we show in addition that each piecewise linear manifold of dimension up to four on which a finite group acts by piecewise linear homeomorphisms admits a compatible smooth structure with respect to which the group acts smoothly. This solves a challenge by Thurston and confirms a conjecture by Kwasik and Lee. In the topological category a counterexample to the above mentioned characterization is given by the binary icosahedral group. We show that this is the only counterexample up to products. In particular, we answer the question by Davis of when the underlying space of an orbifold is a topological manifold. As a corollary of our results we generalize a fixed point theorem by Steinberg on unitary reflection groups to finite groups generated by reflections and rotations. As an application thereof we answer a question by Petrunin on quotients of spheres.

Operatory kompozycji i miary Carlesona w teorii przestrzeni Hardy’ego-Orlicza na obszarach

Wydział Matematyki i Informatyki: Zakład Teorii Interpolacji i Aproksymacji

Strictly singular operators in pairs of L (p) space

Let E and F be Banach spaces. A linear operator from E to F is said to be strictly singular if, for any subspace Q aS, E, the restriction of A to Q is not an isomorphism. A compactness criterion for any strictly singular operator from L (p) to L (q) is found. There exists a strictly singular but not superstrictly singular operator on L (p) , provided that p not equal 2.

Rota's universal operators and invariant subspaces in Hilbert spaces

A Hilbert space operator is called universal (in the sense of Rota) if every operator on the Hilbert space is similar to a multiple of the restriction of the universal operator to one of its invariant subspaces. We exhibit an analytic Toeplitz operator whose adjoint is universal in the sense of Rota and commutes with a quasi-nilpotent injective compact operator with dense range. In particular, this new universal operator invites an approach to the Invariant Subspace Problem that uses properties of operators that commute with the universal operator.

Decomposizione CUR e applicazioni

Nell’ambito dell’analisi dati è spesso necessario fare uso di grandi matrici per descrivere un dataset. In molti casi, per poterlo analizzare, risulta utile approssimare la matrice dei dati come prodotto di altre matrici. A tale fine, la Decomposizione in Valori Singolari (SVD) è una decomposizione ampiamente utilizzata, in quanto consente di ottenere la migliore approssimazione di un certo rango della matrice. Nonostante le proprietà di ottimalità, le matrici che si ottengono da tale decomposizione non risultano particolarmente significative in funzione dei dati e risulta perciò difficile utilizzarle direttamente per dedurre informazioni sul dataset. Per superare questo limite, abbiamo introdotto la Decomposizione CUR, un particolare tipo di decomposizione di rango basso, in cui la matrice di partenza viene espressa in funzione solo di alcune righe e colonne della matrice stessa. Essendo costruite a partire da alcuni dati effettivi della matrice, individuati attraverso la tecnica del Subspace Sampling, le matrici ottenute da questa decomposizione consentono di poter fare considerazioni sull’intero dataset. In questa tesi viene descritto l’algoritmo per la costruzione delle matrici di tale decomposizione e vengono messe in evidenza le garanzie sull’errore relativo dell’approssimazione ottenuta. Infine, viene presentata l’applicazione della Decomposizione CUR ad un dataset reale contenente i risultati delle votazioni dei giudici della Corte Suprema degli Stati Uniti e vengono analizzati e discussi i risultati ottenuti.

Implementazione di reti neurali per missioni di telerilevamento satellitare da immagini iperspettrali

L’utilizzo di reti neurali, applicate a immagini iperspettrali, direttamente a bordo di un satellite, permetterebbe una stima tempestiva ed aggiornata di alcuni parametri del suolo, necessari per ottimizzare il processo di fertilizzazione in agricoltura. Questo elaborato confronta due modelli derivati dalle reti EfficientNet-Lite0 ed EdgeNeXt per la stima del valore di pH del terreno e delle concentrazioni di Potassio (K), Pentossido di Fosforo (P2O5) e Magnesio (Mg) da immagini iperspettrali raffiguranti campi agricoli. Sono stati inoltre testati due metodi di riduzione delle bande: l’Analisi delle Componenti Principali (PCA) e un algoritmo di selezione basato sull’Orthogonal Subspace Projection (OSP). Lo scopo è ridurre le dimensioni delle immagini al fine di limitare le risorse necessarie all’inferenza delle reti, pur preservandone l’accuratezza. L’esecuzione in tempo reale (23.6 fps) della migliore soluzione ottenuta sul sistema embedded Dev Board Mini ne dimostra l’applicabilità a bordo di nanosatelliti.