Optimal Data Distributions in Machine Learning

In the first part of the thesis we explore three fundamental questions that arise naturally when we conceive a machine learning scenario where the training and test distributions can differ. Contrary to conventional wisdom, we show that in fact mismatched training and test distribution can yield better out-of-sample performance. This optimal performance can be obtained by training with the dual distribution. This optimal training distribution depends on the test distribution set by the problem, but not on the target function that we want to learn. We show how to obtain this distribution in both discrete and continuous input spaces, as well as how to approximate it in a practical scenario. Benefits of using this distribution are exemplified in both synthetic and real data sets.

In order to apply the dual distribution in the supervised learning scenario where the training data set is fixed, it is necessary to use weights to make the sample appear as if it came from the dual distribution. We explore the negative effect that weighting a sample can have. The theoretical decomposition of the use of weights regarding its effect on the out-of-sample error is easy to understand but not actionable in practice, as the quantities involved cannot be computed. Hence, we propose the Targeted Weighting algorithm that determines if, for a given set of weights, the out-of-sample performance will improve or not in a practical setting. This is necessary as the setting assumes there are no labeled points distributed according to the test distribution, only unlabeled samples.

Finally, we propose a new class of matching algorithms that can be used to match the training set to a desired distribution, such as the dual distribution (or the test distribution). These algorithms can be applied to very large datasets, and we show how they lead to improved performance in a large real dataset such as the Netflix dataset. Their computational complexity is the main reason for their advantage over previous algorithms proposed in the covariate shift literature.

In the second part of the thesis we apply Machine Learning to the problem of behavior recognition. We develop a specific behavior classifier to study fly aggression, and we develop a system that allows analyzing behavior in videos of animals, with minimal supervision. The system, which we call CUBA (Caltech Unsupervised Behavior Analysis), allows detecting movemes, actions, and stories from time series describing the position of animals in videos. The method summarizes the data, as well as it provides biologists with a mathematical tool to test new hypotheses. Other benefits of CUBA include finding classifiers for specific behaviors without the need for annotation, as well as providing means to discriminate groups of animals, for example, according to their genetic line.

High-energy scattering processes with cuts in the angular momentum plane

The experimental consequence of Regge cuts in the angular momentum plane are investigated. The principle tool in the study is the set of diagrams originally proposed by Amati, Fubini, and Stanghellini. Mandelstam has shown that the AFS cuts are actually cancelled on the physical sheet, but they may provide a useful guide to the properties of the real cuts. Inclusion of cuts modifies the simple Regge pole predictions for high-energy scattering data. As an example, an attempt is made to fit high energy elastic scattering data for pp, ṗp, π^±p, and K^±p, by replacing the Igi pole by terms representing the effect of a Regge cut. The data seem to be compatible with either a cut or the Igi pole.

A study of T=1, T=3/2, and T=2 states in some light nuclei using (He^3,n) reactions

The (He³, n) reactions on B¹¹, N¹⁵, O¹⁶, and O¹⁸ targets have been studied using a pulsed-beam time-of-flight spectrometer. Special emphasis was placed upon the determination of the excitation energies and properties of states with T = 1 (in Ne¹⁸), T = 3/2 (in N¹³ and F¹⁷) and T = 2 (in Ne²⁰). The identification of the T = 3/2 and T = 2 levels is based on the structure of these states as revealed by intensities and shapes of angular distributions. The reactions are interpreted in terms of double stripping theory. Angular distributions have been compared with plane and distorted wave stripping theories. Results for the four reactions are summarized below:

1) O¹⁶ (He³, n). The reaction has been studied at incident energies up to 13.5 MeV and two previously unreported levels in Ne¹⁸ were observed at E_x = 4.55 ± .015 MeV (Γ = 70 ± 30 keV) and E_x = 5.14 ± .018 MeV (Γ = 100 ± 40 keV).

2) B¹¹ (He³, n). The reaction has been studied at incident energies up to 13.5 MeV. Three T = 3/2 levels in N¹³ have been identified at E_x = 15.068 ± .008 MeV (Γ ˂ 15 keV), E_x = 18.44 ± .04, and E_x 18.98 ± .02 MeV (Γ = 40 ± 20 keV).

3) N¹⁵ (He³, n). The reaction has been studied at incident energies up to 11.88 MeV. T = 3/2 levels in F¹⁷ have been identified at E_x = 11.195 ± .007 MeV (Γ ˂ 20 keV), E_x = 12.540 ± .010 MeV (Γ ˂ 25 keV), and E_x = 13.095 ± .009 MeV (Γ ˂ 25 keV).

4) O¹⁸ (He³, n). The reaction has been studied at incident energies up to 9.0 MeV. The excitation energy of the lowest T = 2 level in Ne²⁰ has been found to be 16.730 ± .006 MeV (Γ ˂ 20 keV).

Angular distributions of the transitions leading to the above higher isospin states are well described by double stripping theory. Analog correspondences are established by comparing the present results with recent studies (t, p) and (He³, p) reactions on the same targets.

Stationary absolute distributions for chains of infinite order

Let {Ƶ_n}^∞_{n = -∞} be a stochastic process with state space S₁ = {0, 1, …, D – 1}. Such a process is called a chain of infinite order. The transitions of the chain are described by the functions

Q_i(i⁽⁰⁾) = Ƥ(Ƶ_n = i | Ƶ_{n - 1} = i ⁽⁰⁾₁, Ƶ_{n - 2} = i ⁽⁰⁾₂, …) (i ɛ S₁), where i⁽⁰⁾ = (i⁽⁰⁾₁, i⁽⁰⁾₂, …) ranges over infinite sequences from S₁. If i⁽ⁿ⁾ = (i⁽ⁿ⁾₁, i⁽ⁿ⁾₂, …) for n = 1, 2,…, then i⁽ⁿ⁾ → i⁽⁰⁾ means that for each k, i⁽ⁿ⁾_k = i⁽⁰⁾_k for all n sufficiently large.

Given functions Q_i(i⁽⁰⁾) such that

(i) 0 ≤ Q_i(i⁽⁰) ≤ ξ ˂ 1

(ii)D – 1/Ʃ/i = 0 Q_i(i⁽⁰⁾) Ξ 1

(iii) Q_i(i⁽ⁿ⁾) → Q_i(i⁽⁰⁾) whenever i⁽ⁿ⁾ → i⁽⁰⁾,

we prove the existence of a stationary chain of infinite order {Ƶ_n} whose transitions are given by

Ƥ (Ƶ_n = i | Ƶ_{n - 1}, Ƶ_{n - 2}, …) = Q_i(Ƶ_{n - 1}, Ƶ_{n - 2}, …)

With probability 1. The method also yields stationary chains {Ƶ_n} for which (iii) does not hold but whose transition probabilities are, in a sense, “locally Markovian.” These and similar results extend a paper by T.E. Harris [Pac. J. Math., 5 (1955), 707-724].

Included is a new proof of the existence and uniqueness of a stationary absolute distribution for an Nth order Markov chain in which all transitions are possible. This proof allows us to achieve our main results without the use of limit theorem techniques.

Large plane deformations of thin elastic sheets of neo-hookean material

Large plane deformations of thin elastic sheets of neo-Hookean material are considered and a method of successive substitutions is developed to solve problems within the two-dimensional theory of finite plane stress. The first approximation is determined by linear boundary value problems on two harmonic functions, and it is approached asymptotically at very large extensions in the plane of the sheet. The second and higher approximations are obtained by solving Poisson equations. The method requires modification when the membrane has a traction-free edge.

Several problems are treated involving infinite sheets under uniform biaxial stretching at infinity. First approximations are obtained when a circular or elliptic inclusion is present and when the sheet has a circular or elliptic hole, including the limiting cases of a line inclusion and a straight crack or slit. Good agreement with exact solutions is found for circularly symmetric deformations. Other examples discuss the stretching of a short wide strip, the deformation near a boundary corner which is traction-free, and the application of a concentrated load to a boundary point.