Optimal Data Distributions in Machine Learning

In the first part of the thesis we explore three fundamental questions that arise naturally when we conceive a machine learning scenario where the training and test distributions can differ. Contrary to conventional wisdom, we show that in fact mismatched training and test distribution can yield better out-of-sample performance. This optimal performance can be obtained by training with the dual distribution. This optimal training distribution depends on the test distribution set by the problem, but not on the target function that we want to learn. We show how to obtain this distribution in both discrete and continuous input spaces, as well as how to approximate it in a practical scenario. Benefits of using this distribution are exemplified in both synthetic and real data sets.

In order to apply the dual distribution in the supervised learning scenario where the training data set is fixed, it is necessary to use weights to make the sample appear as if it came from the dual distribution. We explore the negative effect that weighting a sample can have. The theoretical decomposition of the use of weights regarding its effect on the out-of-sample error is easy to understand but not actionable in practice, as the quantities involved cannot be computed. Hence, we propose the Targeted Weighting algorithm that determines if, for a given set of weights, the out-of-sample performance will improve or not in a practical setting. This is necessary as the setting assumes there are no labeled points distributed according to the test distribution, only unlabeled samples.

Finally, we propose a new class of matching algorithms that can be used to match the training set to a desired distribution, such as the dual distribution (or the test distribution). These algorithms can be applied to very large datasets, and we show how they lead to improved performance in a large real dataset such as the Netflix dataset. Their computational complexity is the main reason for their advantage over previous algorithms proposed in the covariate shift literature.

In the second part of the thesis we apply Machine Learning to the problem of behavior recognition. We develop a specific behavior classifier to study fly aggression, and we develop a system that allows analyzing behavior in videos of animals, with minimal supervision. The system, which we call CUBA (Caltech Unsupervised Behavior Analysis), allows detecting movemes, actions, and stories from time series describing the position of animals in videos. The method summarizes the data, as well as it provides biologists with a mathematical tool to test new hypotheses. Other benefits of CUBA include finding classifiers for specific behaviors without the need for annotation, as well as providing means to discriminate groups of animals, for example, according to their genetic line.

Chains of Non-regular de Branges Spaces

We consider canonical systems with singular left endpoints, and discuss the concept of a scalar spectral measure and the corresponding generalized Fourier transform associated with a canonical system with a singular left endpoint. We use the framework of de Branges’ theory of Hilbert spaces of entire functions to study the correspondence between chains of non-regular de Branges spaces, canonical systems with singular left endpoints, and spectral measures.

We find sufficient integrability conditions on a Hamiltonian H which ensure the existence of a chain of de Branges functions in the first generalized Pólya class with Hamiltonian H. This result generalizes de Branges’ Theorem 41, which showed the sufficiency of stronger integrability conditions on H for the existence of a chain in the Pólya class. We show the conditions that de Branges came up with are also necessary. In the case of Krein’s strings, namely when the Hamiltonian is diagonal, we show our proposed conditions are also necessary.

We also investigate the asymptotic conditions on chains of de Branges functions as t approaches its left endpoint. We show there is a one-to-one correspondence between chains of de Branges functions satisfying certain asymptotic conditions and chains in the Pólya class. In the case of Krein’s strings, we also establish the one-to-one correspondence between chains satisfying certain asymptotic conditions and chains in the generalized Pólya class.

Locally convex Riesz spaces and Archimedean quotient spaces

A Riesz space with a Hausdorff, locally convex topology determined by Riesz seminorms is called a locally convex Riesz space. A sequence {x_n} in a locally convex Riesz space L is said to converge locally to x ϵ L if for some topologically bounded set B and every real r ˃ 0 there exists N (r) and n ≥ N (r) implies x – x_n ϵ r^b. Local Cauchy sequences are defined analogously, and L is said to be locally complete if every local Cauchy sequence converges locally. Then L is locally complete if and only if every monotone local Cauchy sequence has a least upper bound. This is a somewhat more general form of the completeness criterion for Riesz – normed Riesz spaces given by Luxemburg and Zaanen. Locally complete, bound, locally convex Riesz spaces are barrelled. If the space is metrizable, local completeness and topological completeness are equivalent.

Two measures of the non-archimedean character of a non-archimedean Riesz space L are the smallest ideal A_o (L) such that quotient space is Archimedean and the ideal I (L) = { x ϵ L: for some 0 ≤ v ϵ L, n |x| ≤ v for n = 1, 2, …}. In general A_o (L) ᴝ I (L). If L is itself a quotient space, a necessary and sufficient condition that A_o (L) = I (L) is given. There is an example where A_o (L) ≠ I (L).

A necessary and sufficient condition that a Riesz space L have every quotient space Archimedean is that for every 0 ≤ u, v ϵ L there exist u₁ = sup (inf (n v, u): n = 1, 2, …), and real numbers m₁ and m₂ such that m₁ u₁ ≥ v₁ and m₂ v₁ ≥ u₁. If, in addition, L is Dedekind σ – complete, then L may be represented as the space of all functions which vanish off finite subsets of some non-empty set.

Multiplication in Riesz spaces

A.G. Vulih has shown how an essentially unique intrinsic multiplication can be defined in certain types of Riesz spaces (vector lattices) L. In general, the multiplication is not universally defined in L, but L can always be imbedded in a large space L^# in which multiplication is universally defined.

If ф is a normal integral in L, then ф can be extended to a normal integral on a large space L₁(ф) in L^#, and L₁(ф) may be regarded as an abstract integral space. A very general form of the Radon-Nikodym theorem can be proved in L₁(ф), and this can be used to give a relatively simple proof of a theorem of Segal giving a necessary and sufficient condition that the Radon-Nikodym theorem hold in a measure space.

In another application, the multiplication is used to give a representation of certain Riesz spaces as rings of operators on a Hilbert space.

Smooth Banach spaces and approximations

If E and F are real Banach spaces let C^p,q(E, F) O ≤ q ≤ p ≤ ∞, denote those maps from E to F which have p continuous Frechet derivatives of which the first q derivatives are bounded. A Banach space E is defined to be C^p,q smooth if C^p,q(E,R) contains a nonzero function with bounded support. This generalizes the standard C^p smoothness classification.

If an L^p space, p ≥ 1, is C^q smooth then it is also C^q,q smooth so that in particular L^p for p an even integer is C^∞,∞ smooth and L^p for p an odd integer is C^p-1,p-1 smooth. In general, however, a C^p smooth B-space need not be C^p,p smooth. C_o is shown to be a non-C^2,2 smooth B-space although it is known to be C^∞ smooth. It is proved that if E is C^p,1 smooth then C_o(E) is C^p,1 smooth and if E has an equivalent C^p norm then c_o(E) has an equivalent C^p norm.

Various consequences of C^p,q smoothness are studied. If f ϵ C^p,q(E,F), if F is C^p,q smooth and if E is non-C^p,q smooth, then the image under f of the boundary of any bounded open subset U of E is dense in the image of U. If E is separable then E is C^p,q smooth if and only if E admits C^p,q partitions of unity; E is C^p,psmooth, p ˂∞, if and only if every closed subset of E is the zero set of some C^P function.

f ϵ C^q(E,F), 0 ≤ q ≤ p ≤ ∞, is said to be C_p,q approximable on a subset U of E if for any ϵ ˃ 0 there exists a g ϵ C^p(E,F) satisfying

sup/xϵU, O≤k≤q ‖ D^k f(x) - D^k g(x) ‖ ≤ ϵ.

It is shown that if E is separable and C^p,q smooth and if f ϵ C^q(E,F) is C_p,q approximable on some neighborhood of every point of E, then F is C_p,q approximable on all of E.

In general it is unknown whether an arbitrary function in C¹(l², R) is C_2,1 approximable and an example of a function in C¹(l², R) which may not be C_2,1 approximable is given. A weak form of C_∞,q, q≥1, to functions in C^q(l², R) is proved: Let {U_α} be a locally finite cover of l² and let {T_α} be a corresponding collection of Hilbert-Schmidt operators on l². Then for any f ϵ C^q(l²,F) such that for all α

sup ‖ D^k(f(x)-g(x))[T_αh]‖ ≤ 1.

xϵU_α,‖h‖≤1, 0≤k≤q