Linear dimensionality reduction: Survey, insights, and generalizations

© 2015 John P. Cunningham and Zoubin Ghahramani. Linear dimensionality reduction methods are a cornerstone of analyzing high dimensional data, due to their simple geometric interpretations and typically attractive computational properties. These methods capture many data features of interest, such as covariance, dynamical structure, correlation between data sets, input-output relationships, and margin between data classes. Methods have been developed with a variety of names and motivations in many fields, and perhaps as a result the connections between all these methods have not been highlighted. Here we survey methods from this disparate literature as optimization programs over matrix manifolds. We discuss principal component analysis, factor analysis, linear multidimensional scaling, Fisher's linear discriminant analysis, canonical correlations analysis, maximum autocorrelation factors, slow feature analysis, sufficient dimensionality reduction, undercomplete independent component analysis, linear regression, distance metric learning, and more. This optimization framework gives insight to some rarely discussed shortcomings of well-known methods, such as the suboptimality of certain eigenvector solutions. Modern techniques for optimization over matrix manifolds enable a generic linear dimensionality reduction solver, which accepts as input data and an objective to be optimized, and returns, as output, an optimal low-dimensional projection of the data. This simple optimization framework further allows straightforward generalizations and novel variants of classical methods, which we demonstrate here by creating an orthogonal-projection canonical correlations analysis. More broadly, this survey and generic solver suggest that linear dimensionality reduction can move toward becoming a blackbox, objective-agnostic numerical technology.

On the relation of slow feature analysis and Laplacian eigenmaps

The past decade has seen a rise of interest in Laplacian eigenmaps (LEMs) for nonlinear dimensionality reduction. LEMs have been used in spectral clustering, in semisupervised learning, and for providing efficient state representations for reinforcement learning. Here, we show that LEMs are closely related to slow feature analysis (SFA), a biologically inspired, unsupervised learning algorithm originally designed for learning invariant visual representations. We show that SFA can be interpreted as a function approximation of LEMs, where the topological neighborhoods required for LEMs are implicitly defined by the temporal structure of the data. Based on this relation, we propose a generalization of SFA to arbitrary neighborhood relations and demonstrate its applicability for spectral clustering. Finally, we review previous work with the goal of providing a unifying view on SFA and LEMs. © 2011 Massachusetts Institute of Technology.

Manopt, a matlab toolbox for optimization on manifolds

Optimization on manifolds is a rapidly developing branch of nonlinear optimization. Its focus is on problems where the smooth geometry of the search space can be leveraged to design effcient numerical algorithms. In particular, optimization on manifolds is well-suited to deal with rank and orthogonality constraints. Such structured constraints appear pervasively in machine learning applications, including low-rank matrix completion, sensor network localization, camera network registration, independent component analysis, metric learning, dimensionality reduction and so on. The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms. By dealing internally with most of the differential geometry, the package aims particularly at lowering the entrance barrier. © 2014 Nicolas Boumal.

Multi-dimensional conditional moment closure modelling applied to a heavy-duty common-rail diesel engine

A multi-dimensional combustion code implementing the Conditional Moment Closure turbulent combustion model interfaced with a well-established RANS two- phase flow field solver has been employed to study a broad range of operating conditions for a heavy duty direct-injection common-rail Diesel engine. These conditions include different loads (25%, 50%, 75% and full load) and engine speeds (1250 and 1830 RPM) and, with respect to the fuel path, different injection timings and rail pressures. A total of nine cases have been simulated. Excellent agreement with experimental data has been found for the pressure traces and the heat release rates, without adjusting any model constants. The chemical mechanism used contains a detailed NOx sub-mechanism. The predicted emissions agree reasonably well with the experimental data considering the range of operating points and given no adjustments of any rate constants have been employed. In an effort to identify CPU cost reduction potential, various dimensionality reduction strategies have been assessed. Furthermore, the sensitivity of the predictions with respect to resolution in particular relating to the CMC grid has been investigated. Overall, the results suggest that the presented modelling strategy has considerable predictive capability concerning Diesel engine combustion without requiring model constant calibration based on experimental data. This is true particularly for the heat release rates predictions and, to a lesser extent, for NOx emissions where further progress is still necessary. © 2009 SAE International.

Geometric optimization methods for independent component analysis applied on gene expression data

DNA microarrays provide a huge amount of data and require therefore dimensionality reduction methods to extract meaningful biological information. Independent Component Analysis (ICA) was proposed by several authors as an interesting means. Unfortunately, experimental data are usually of poor quality- because of noise, outliers and lack of samples. Robustness to these hurdles will thus be a key feature for an ICA algorithm. This paper identifies a robust contrast function and proposes a new ICA algorithm. © 2007 IEEE.

Adaptive modulation induced WDM impairment reduction in optical OFDM PONs

The transmission performance of multi-channel adaptively modulated optical OFDM (AMOOFDM) signals is numerically investigated, for the first time, in optical amplification- and chromatic dispersion compensation-free, intensity-modulation and direct-detection systems incorporating directly modulated DFB lasers (DMLs). It is shown that adaptive modulation not only reduces significantly the nonlinear WDM impairments induced by the effects of cross-phase modulation and four-wave mixing, but also compensates effectively for the DML-induced frequency chirp effect. In comparison with identical modulation, adaptive modulation improves the maximum achievable signal transmission capacity of a central channel by a factor of 1.3 and 3.6 for 40km and 80km SMFs, respectively, with corresponding dynamic input optical power ranges being extended by approximately 5dB. In addition, adaptive modulation also enables cross-channel complementary modulation format mapping, leading to an improved transmission capacity of the entire WDM system. Copyright © 2010 The authors.

Modelling and control of nonlinear systems using Gaussian processes with partial model information

Gaussian processes are gaining increasing popularity among the control community, in particular for the modelling of discrete time state space systems. However, it has not been clear how to incorporate model information, in the form of known state relationships, when using a Gaussian process as a predictive model. An obvious example of known prior information is position and velocity related states. Incorporation of such information would be beneficial both computationally and for faster dynamics learning. This paper introduces a method of achieving this, yielding faster dynamics learning and a reduction in computational effort from O(Dn2) to O((D - F)n2) in the prediction stage for a system with D states, F known state relationships and n observations. The effectiveness of the method is demonstrated through its inclusion in the PILCO learning algorithm with application to the swing-up and balance of a torque-limited pendulum and the balancing of a robotic unicycle in simulation. © 2012 IEEE.