Notes on PCA, Regularization, Sparsity and Support Vector Machines

We derive a new representation for a function as a linear combination of local correlation kernels at optimal sparse locations and discuss its relation to PCA, regularization, sparsity principles and Support Vector Machines. We first review previous results for the approximation of a function from discrete data (Girosi, 1998) in the context of Vapnik"s feature space and dual representation (Vapnik, 1995). We apply them to show 1) that a standard regularization functional with a stabilizer defined in terms of the correlation function induces a regression function in the span of the feature space of classical Principal Components and 2) that there exist a dual representations of the regression function in terms of a regularization network with a kernel equal to a generalized correlation function. We then describe the main observation of the paper: the dual representation in terms of the correlation function can be sparsified using the Support Vector Machines (Vapnik, 1982) technique and this operation is equivalent to sparsify a large dictionary of basis functions adapted to the task, using a variation of Basis Pursuit De-Noising (Chen, Donoho and Saunders, 1995; see also related work by Donahue and Geiger, 1994; Olshausen and Field, 1995; Lewicki and Sejnowski, 1998). In addition to extending the close relations between regularization, Support Vector Machines and sparsity, our work also illuminates and formalizes the LFA concept of Penev and Atick (1996). We discuss the relation between our results, which are about regression, and the different problem of pattern classification.

An Equivalence Between Sparse Approximation and Support Vector Machines

In the first part of this paper we show a similarity between the principle of Structural Risk Minimization Principle (SRM) (Vapnik, 1982) and the idea of Sparse Approximation, as defined in (Chen, Donoho and Saunders, 1995) and Olshausen and Field (1996). Then we focus on two specific (approximate) implementations of SRM and Sparse Approximation, which have been used to solve the problem of function approximation. For SRM we consider the Support Vector Machine technique proposed by V. Vapnik and his team at AT&T Bell Labs, and for Sparse Approximation we consider a modification of the Basis Pursuit De-Noising algorithm proposed by Chen, Donoho and Saunders (1995). We show that, under certain conditions, these two techniques are equivalent: they give the same solution and they require the solution of the same quadratic programming problem.

Uncertainty Propagation in Model-Based Recognition

Building robust recognition systems requires a careful understanding of the effects of error in sensed features. Error in these image features results in a region of uncertainty in the possible image location of each additional model feature. We present an accurate, analytic approximation for this uncertainty region when model poses are based on matching three image and model points, for both Gaussian and bounded error in the detection of image points, and for both scaled-orthographic and perspective projection models. This result applies to objects that are fully three- dimensional, where past results considered only two-dimensional objects. Further, we introduce a linear programming algorithm to compute the uncertainty region when poses are based on any number of initial matches. Finally, we use these results to extend, from two-dimensional to three- dimensional objects, robust implementations of alignmentt interpretation- tree search, and ransformation clustering.

Preconditioning and iterative solution of symmetric indefinite linear systems arising from interior point methods for linear programming

We study the preconditioning of symmetric indefinite linear systems of equations that arise in interior point solution of linear optimization problems. The preconditioning method that we study exploits the block structure of the augmented matrix to design a similar block structure preconditioner to improve the spectral properties of the resulting preconditioned matrix so as to improve the convergence rate of the iterative solution of the system. We also propose a two-phase algorithm that takes advantage of the spectral properties of the transformed matrix to solve for the Newton directions in the interior-point method. Numerical experiments have been performed on some LP test problems in the NETLIB suite to demonstrate the potential of the preconditioning method discussed.

Non-linear mechanical behavior of the elastomer polydimethylsiloxane (PDMS) used in the manufacture of microfluidic devices

Polydimethylsiloxane (PDMS) is the elastomer of choice to create a variety of microfluidic devices by soft lithography techniques (eg., [1], [2], [3], [4]). Accurate and reliable design, manufacture, and operation of microfluidic devices made from PDMS, require a detailed characterization of the deformation and failure behavior of the material. This paper discusses progress in a recently-initiated research project towards this goal. We have conducted large-deformation tension and compression experiments on traditional macroscale specimens, as well as microscale tension experiments on thin-film (≈ 50µm thickness) specimens of PDMS with varying ratios of monomer:curing agent (5:1, 10:1, 20:1). We find that the stress-stretch response of these materials shows significant variability, even for nominally identically prepared specimens. A non-linear, large-deformation rubber-elasticity model [5], [6] is applied to represent the behavior of PDMS. The constitutive model has been implemented in a finite-element program [7] to aid the design of microfluidic devices made from this material. As a first attempt towards the goal of estimating the non-linear material parameters for PDMS from indentation experiments, we have conducted micro-indentation experiments using a spherical indenter-tip, and carried out corresponding numerical simulations to verify how well the numerically-predicted P(load-h(depth of indentation) curves compare with the corresponding experimental measurements. The results are encouraging, and show the possibility of estimating the material parameters for PDMS from relatively simple micro-indentation experiments, and corresponding numerical simulations.

Convex linear combination processes for compositions

Aitchison and Bacon-Shone (1999) considered convex linear combinations of compositions. In other words, they investigated compositions of compositions, where the mixing composition follows a logistic Normal distribution (or a perturbation process) and the compositions being mixed follow a logistic Normal distribution. In this paper, I investigate the extension to situations where the mixing composition varies with a number of dimensions. Examples would be where the mixing proportions vary with time or distance or a combination of the two. Practical situations include a river where the mixing proportions vary along the river, or across a lake and possibly with a time trend. This is illustrated with a dataset similar to that used in the Aitchison and Bacon-Shone paper, which looked at how pollution in a loch depended on the pollution in the three rivers that feed the loch. Here, I explicitly model the variation in the linear combination across the loch, assuming that the mean of the logistic Normal distribution depends on the river flows and relative distance from the source origins

A Log-linear model of grain size influence on the geochemistry of sediments

Sediment composition is mainly controlled by the nature of the source rock(s), and chemical (weathering) and physical processes (mechanical crushing, abrasion, hydrodynamic sorting) during alteration and transport. Although the factors controlling these processes are conceptually well understood, detailed quantification of compositional changes induced by a single process are rare, as are examples where the effects of several processes can be distinguished. The present study was designed to characterize the role of mechanical crushing and sorting in the absence of chemical weathering. Twenty sediment samples were taken from Alpine glaciers that erode almost pure granitoid lithologies. For each sample, 11 grain-size fractions from granules to clay (ø grades <-1 to >9) were separated, and each fraction was analysed for its chemical composition. The presence of clear steps in the box-plots of all parts (in adequate ilr and clr scales) against ø is assumed to be explained by typical crystal size ranges for the relevant mineral phases. These scatter plots and the biplot suggest a splitting of the full grain size range into three groups: coarser than ø=4 (comparatively rich in SiO2, Na2O, K2O, Al2O3, and dominated by “felsic” minerals like quartz and feldspar), finer than ø=8 (comparatively rich in TiO2, MnO, MgO, Fe2O3, mostly related to “mafic” sheet silicates like biotite and chlorite), and intermediate grains sizes (4≤ø <8; comparatively rich in P2O5 and CaO, related to apatite, some feldspar). To further test the absence of chemical weathering, the observed compositions were regressed against three explanatory variables: a trend on grain size in ø scale, a step function for ø≥4, and another for ø≥8. The original hypothesis was that the trend could be identified with weathering effects, whereas each step function would highlight those minerals with biggest characteristic size at its lower end. Results suggest that this assumption is reasonable for the step function, but that besides weathering some other factors (different mechanical behavior of minerals) have also an important contribution to the trend. Key words: sediment, geochemistry, grain size, regression, step function

A First approximation to the realism/idealism debate about the external world

El text intenta fer una primera aproximació al debat contemporani entre realistes i anti-realistes sobre el món empíric, centrant-se en les posicions de Putnam i Nagel. El seu objectiu principal és el d'entendre les motivacions de les posicions i l'estructura actual del debat, i el d'establir les característiques que hauria de tenir qualsevol posició satisfactòria

Studies about the atomic capabilities concept for linear control systems in physical multi-agent environments

This paper shows the impact of the atomic capabilities concept to include control-oriented knowledge of linear control systems in the decisions making structure of physical agents. These agents operate in a real environment managing physical objects (e.g. their physical bodies) in coordinated tasks. This approach is presented using an introspective reasoning approach and control theory based on the specific tasks of passing a ball and executing the offside manoeuvre between physical agents in the robotic soccer testbed. Experimental results and conclusions are presented, emphasising the advantages of our approach that improve the multi-agent performance in cooperative systems

Linear algebra with applications

Lecture slides and notes for a PhD level course on linear algebra for electrical engineers and computer scientists. This course is given in in the framework of the School of Electronics and Computer Science Mathematics Training Courses https://secure.ecs.soton.ac.uk/notes/pg_maths/ (ECS password required)

MA167exam00pdf

Exam questions and solutions in PDF

MA167exam00

Exam questions and solutions in LaTex. Diagrams for the questions are all together in the support.zip file, as .eps files

MA160exam01

Exam questions and solutions in LaTex. Diagrams for the questions are all together in the support.zip file, as .eps files

MA160exam01pdf

Exam questions and solutions in PDF

Systematic Storytelling: Interactivity, Agency and Non-linear Narratives

In this lecture we go over the fundamentals of interactive game narratives. Defining what we mean by narrative, and placing games in context with other ergodic literature. We look at non-linear structures, agency, and the narrative paradox. Concluding with a set of mechanisms that games designers use to manage agency in their narrative games.