Linear dimensionality reduction: Survey, insights, and generalizations

© 2015 John P. Cunningham and Zoubin Ghahramani. Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

Multi-dimensional conditional moment closure modelling applied to a heavy-duty common-rail diesel engine

A multi-dimensional combustion code implementing the Conditional Moment Closure turbulent combustion model interfaced with a well-established RANS two- phase flow field solver has been employed to study a broad range of operating conditions for a heavy duty direct-injection common-rail Diesel engine. These conditions include different loads (25%, 50%, 75% and full load) and engine speeds (1250 and 1830 RPM) and, with respect to the fuel path, different injection timings and rail pressures. A total of nine cases have been simulated. Excellent agreement with experimental data has been found for the pressure traces and the heat release rates, without adjusting any model constants. The chemical mechanism used contains a detailed NOx sub-mechanism. The predicted emissions agree reasonably well with the experimental data considering the range of operating points and given no adjustments of any rate constants have been employed. In an effort to identify CPU cost reduction potential, various dimensionality reduction strategies have been assessed. Furthermore, the sensitivity of the predictions with respect to resolution in particular relating to the CMC grid has been investigated. Overall, the results suggest that the presented modelling strategy has considerable predictive capability concerning Diesel engine combustion without requiring model constant calibration based on experimental data. This is true particularly for the heat release rates predictions and, to a lesser extent, for NOx emissions where further progress is still necessary. © 2009 SAE International.

On the relation of slow feature analysis and Laplacian eigenmaps

The past decade has seen a rise of interest in Laplacian eigenmaps (LEMs) for nonlinear dimensionality reduction. LEMs have been used in spectral clustering, in semisupervised learning, and for providing efficient state representations for reinforcement learning. Here, we show that LEMs are closely related to slow feature analysis (SFA), a biologically inspired, unsupervised learning algorithm originally designed for learning invariant visual representations. We show that SFA can be interpreted as a function approximation of LEMs, where the topological neighborhoods required for LEMs are implicitly defined by the temporal structure of the data. Based on this relation, we propose a generalization of SFA to arbitrary neighborhood relations and demonstrate its applicability for spectral clustering. Finally, we review previous work with the goal of providing a unifying view on SFA and LEMs. © 2011 Massachusetts Institute of Technology.

Geometric optimization methods for independent component analysis applied on gene expression data

DNA microarrays provide a huge amount of data and require therefore dimensionality reduction methods to extract meaningful biological information. Independent Component Analysis (ICA) was proposed by several authors as an interesting means. Unfortunately, experimental data are usually of poor quality- because of noise, outliers and lack of samples. Robustness to these hurdles will thus be a key feature for an ICA algorithm. This paper identifies a robust contrast function and proposes a new ICA algorithm. © 2007 IEEE.

Manopt, a matlab toolbox for optimization on manifolds

Optimization on manifolds is a rapidly developing branch of nonlinear optimization. Its focus is on problems where the smooth geometry of the search space can be leveraged to design effcient numerical algorithms. In particular, optimization on manifolds is well-suited to deal with rank and orthogonality constraints. Such structured constraints appear pervasively in machine learning applications, including low-rank matrix completion, sensor network localization, camera network registration, independent component analysis, metric learning, dimensionality reduction and so on. The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms. By dealing internally with most of the differential geometry, the package aims particularly at lowering the entrance barrier. © 2014 Nicolas Boumal.

Robust noise reduction for speech and audio signals

Statistical model-based methods are presented for the reconstruction of autocorrelated signals in impulsive plus continuous noise environments. Signals are modelled as autoregressive and noise sources as discrete and continuous mixtures of Gaussians, allowing for robustness in highly impulsive and non-Gaussian environments. Markov Chain Monte Carlo methods are used for reconstruction of the corrupted waveforms within a Bayesian probabilistic framework and results are presented for contaminated voice and audio signals.

Stress reduction and bond stability during thermal annealing of tetrahedral amorphous carbon

A comprehensive study of the stress release and structural changes caused by postdeposition thermal annealing of tetrahedral amorphous carbon (ta-C) on Si has been carried out. Complete stress relief occurs at 600-700°C and is accompanied by minimal structural modifications, as indicated by electron energy loss spectroscopy, Raman spectroscopy, and optical gap measurements. Further annealing in vacuum converts sp3 sites to sp2 with a drastic change occurring after 1100°C. The field emitting behavior is substantially retained up to the complete stress relief, confirming that ta-C is a robust emitting material. © 1999 American Institute of Physics.