Learning Nonlinear Visual Processing from Natural Images

The paradigm of computational vision hypothesizes that any visual function -- such as the recognition of your grandparent -- can be replicated by computational processing of the visual input. What are these computations that the brain performs? What should or could they be? Working on the latter question, this dissertation takes the statistical approach, where the suitable computations are attempted to be learned from the natural visual data itself. In particular, we empirically study the computational processing that emerges from the statistical properties of the visual world and the constraints and objectives specified for the learning process. This thesis consists of an introduction and 7 peer-reviewed publications, where the purpose of the introduction is to illustrate the area of study to a reader who is not familiar with computational vision research. In the scope of the introduction, we will briefly overview the primary challenges to visual processing, as well as recall some of the current opinions on visual processing in the early visual systems of animals. Next, we describe the methodology we have used in our research, and discuss the presented results. We have included some additional remarks, speculations and conclusions to this discussion that were not featured in the original publications. We present the following results in the publications of this thesis. First, we empirically demonstrate that luminance and contrast are strongly dependent in natural images, contradicting previous theories suggesting that luminance and contrast were processed separately in natural systems due to their independence in the visual data. Second, we show that simple cell -like receptive fields of the primary visual cortex can be learned in the nonlinear contrast domain by maximization of independence. Further, we provide first-time reports of the emergence of conjunctive (corner-detecting) and subtractive (opponent orientation) processing due to nonlinear projection pursuit with simple objective functions related to sparseness and response energy optimization. Then, we show that attempting to extract independent components of nonlinear histogram statistics of a biologically plausible representation leads to projection directions that appear to differentiate between visual contexts. Such processing might be applicable for priming, \ie the selection and tuning of later visual processing. We continue by showing that a different kind of thresholded low-frequency priming can be learned and used to make object detection faster with little loss in accuracy. Finally, we show that in a computational object detection setting, nonlinearly gain-controlled visual features of medium complexity can be acquired sequentially as images are encountered and discarded. We present two online algorithms to perform this feature selection, and propose the idea that for artificial systems, some processing mechanisms could be selectable from the environment without optimizing the mechanisms themselves. In summary, this thesis explores learning visual processing on several levels. The learning can be understood as interplay of input data, model structures, learning objectives, and estimation algorithms. The presented work adds to the growing body of evidence showing that statistical methods can be used to acquire intuitively meaningful visual processing mechanisms. The work also presents some predictions and ideas regarding biological visual processing.

An exponential stability criterion for certain nonlinear systems

Improved sufficient conditions are derived for the exponential stability of a nonlinear time varying feedback system having a time invariant blockG in the forward path and a nonlinear time varying gain ϕ(.)k(t) in the feedback path. φ(.) being an odd monotone nondecreasing function. The resulting bound on is less restrictive than earlier criteria.

Nonlinear mode coupling of surface acoustic waves on an isotropic solid

The nonlinear mode coupling between two co-directional quasi-harmonic Rayleigh surface waves on an isotropic solid is analysed using the method of multiple scales. This procedure yields a system of six semi-linear hyperbolic partial differential equations with the same principal part governing the slow variations in the (complex) amplitudes of the two fundamental, the two second harmonic and the two combination frequency waves at the second stage of the perturbation expansion. A numerical solution of these equations for excitation by monochromatic signals at two arbitrary frequencies, indicates that there is a continuous transfer of energy back and forth among the fundamental, second harmonic and combination frequency waves due to mode coupling. The mode coupling tends to be more pronounced as the frequencies of the interacting waves approach each other.

Nonlinear surface acoustic waves on an isotropic solid

A systematic derivation of the approximate coupled amplitude equations governing the propagation of a quasi-monochromatic Rayleigh surface wave on an isotropic solid is presented, starting from the non-linear governing differential equations and the non-linear free-surface boundary conditions, using the method of mulitple scales. An explicit solution of these equations for a signalling problem is obtained in terms of hyperbolic functions. In the case of monochromatic excitation, it is shown that the second harmonic amplitude grows initially at the expense of the fundamental and that the amplitudes of the fundamental and second harmonic remain bounded for all time.

Nonlinear saturation of ion-acoustic instability

It is proposed that the wave mediated indirect wave-particle interaction may be responsible for nonlinear saturation of current driven low frequency ion-acoustic turbulence. This process decreases the growth rate and increases the damping rate of the wave. Comparison has been made with some experiments.

L_{2}- stability of a class of nonstationary feedback systems with dynamical nonlinear subsystems

A class of feedback systems, consisting of dynamical non-linear subsystems which arise in many diverse control applications, is analyzed for L2-stability. It is shown that, although a transformation of these systems to the familiar Lur'e configuration does not seem to be possible, a one-to-one correspondence may be effected between the stability properties of these and the Lur'e systems. Interesting stability criteria are developed by exploiting this characteristic.

Nonlinear Control Design for an Air-breathing Engine with State Estimation

A nonlinear control design approach is presented in this paper for a challenging application problem of ensuring robust performance of an air-breathing engine operating at supersonic speed. The primary objective of control design is to ensure that the engine produces the required thrust that tracks the commanded thrust as closely as possible by appropriate regulation of the fuel flow rate. However, since the engine operates in the supersonic range, an important secondary objective is to ensure an optimal location of the shock in the intake for maximum pressure recovery with a sufficient margin. This is manipulated by varying the throat area of the nozzle. The nonlinear dynamic inversion technique has been successfully used to achieve both of the above objectives. In this problem, since the process is faster than the actuators, independent control designs have also been carried out for the actuators as well to assure the satisfactory performance of the system. Moreover, an extended Kalman Filter based state estimation design has been carried out both to filter out the process and sensor noises as well as to make the control design operate based on output feedback. Promising simulation results indicate that the proposed control design approach is quite successful in obtaining robust performance of the air-breathing system.

An exponential stability criterion for certain nonlinear systems

Improved sufficient conditions are derived for the exponential stability of a nonlinear time varying feedback system having a time invariant blockG in the forward path and a nonlinear time varying gain ϕ(.)k(t) in the feedback path. φ(.) being an odd monotone nondecreasing function. The resulting bound on $$\left( {{{\frac{{dk}}{{dt}}} \mathord{\left/ {\vphantom {{\frac{{dk}}{{dt}}} k}} \right. \kern-\nulldelimiterspace} k}} \right)$$ is less restrictive than earlier criteria.

On the stability of a nonlinear system with periodic coefficients

A quasi-geometric stability criterion for feedback systems with a linear time invariant forward block and a periodically time varying nonlinear gain in the feedback loop is developed.

Nonlinear saturation of ion-acoustic instability

It is proposed that the wave mediated indirect wave-particle interaction may be responsible for nonlinear saturation of current driven low frequency ion-acoustic turbulence. This process decreases the growth rate and increases the damping rate of the wave. Comparison has been made with some experiments.

A semi-analytical particle filter for identification of nonlinear oscillators

Particle filters find important applications in the problems of state and parameter estimations of dynamical systems of engineering interest. Since a typical filtering algorithm involves Monte Carlo simulations of the process equations, sample variance of the estimator is inversely proportional to the number of particles. The sample variance may be reduced if one uses a Rao-Blackwell marginalization of states and performs analytical computations as much as possible. In this work, we propose a semi-analytical particle filter, requiring no Rao-Blackwell marginalization, for state and parameter estimations of nonlinear dynamical systems with additively Gaussian process/observation noises. Through local linearizations of the nonlinear drift fields in the process/observation equations via explicit Ito-Taylor expansions, the given nonlinear system is transformed into an ensemble of locally linearized systems. Using the most recent observation, conditionally Gaussian posterior density functions of the linearized systems are analytically obtained through the Kalman filter. This information is further exploited within the particle filter algorithm for obtaining samples from the optimal posterior density of the states. The potential of the method in state/parameter estimations is demonstrated through numerical illustrations for a few nonlinear oscillators. The proposed filter is found to yield estimates with reduced sample variance and improved accuracy vis-a-vis results from a form of sequential importance sampling filter.

Nonlinear secondary whirl of an overhung rotor

Gravity critical speeds of rotors have hitherto been studied using linear analysis, and ascribed to rotor stiffness asymmetry. Here, we study an idealized asymmetric nonlinear overhung rotor model of Crandall and Brosens, spinning close to its gravity critical speed.Nonlinearities arise from finite displacements, and the rotor's staticlateral deflection under gravity is taken as small. Assuming small asymmetry and damping, slow modulations of whirl amplitudes are studied using the method of multiple scales. Inertia asymmetry appears only at second order. More interestingly, even without stiffness asymmetry, the gravity-induced resonance survives through geometric nonlinearities. The gravity resonant forcing does not influence the resonant mode at leading order, unlike the typical resonant oscillations. Nevertheless,the usual phenomena of resonances, namely saddle-node bifurcations, jump phenomena and hysteresis, are all observed. An unanticipated periodic solution branch is found. In the three-dimensional space oftwo modal coefficients and a detuning parameter, the full set of periodic solutions is found to be an imperfect version of three mutually intersecting curves: a straight line,a parabola and an ellipse.

R1SVM: A randomised nonlinear approach to large-scale anomaly detection

The problem of unsupervised anomaly detection arises in a wide variety of practical applications. While one-class support vector machines have demonstrated their effectiveness as an anomaly detection technique, their ability to model large datasets is limited due to their memory and time complexity for training. To address this issue for supervised learning of kernel machines, there has been growing interest in random projection methods as an alternative to the computationally expensive problems of kernel matrix construction and sup-port vector optimisation. In this paper we leverage the theory of nonlinear random projections and propose the Randomised One-class SVM (R1SVM), which is an efficient and scalable anomaly detection technique that can be trained on large-scale datasets. Our empirical analysis on several real-life and synthetic datasets shows that our randomised 1SVM algorithm achieves comparable or better accuracy to deep auto encoder and traditional kernelised approaches for anomaly detection, while being approximately 100 times faster in training and testing.