Optimal Data Distributions in Machine Learning

In the first part of the thesis we explore three fundamental questions that arise naturally when we conceive a machine learning scenario where the training and test distributions can differ. Contrary to conventional wisdom, we show that in fact mismatched training and test distribution can yield better out-of-sample performance. This optimal performance can be obtained by training with the dual distribution. This optimal training distribution depends on the test distribution set by the problem, but not on the target function that we want to learn. We show how to obtain this distribution in both discrete and continuous input spaces, as well as how to approximate it in a practical scenario. Benefits of using this distribution are exemplified in both synthetic and real data sets.

In order to apply the dual distribution in the supervised learning scenario where the training data set is fixed, it is necessary to use weights to make the sample appear as if it came from the dual distribution. We explore the negative effect that weighting a sample can have. The theoretical decomposition of the use of weights regarding its effect on the out-of-sample error is easy to understand but not actionable in practice, as the quantities involved cannot be computed. Hence, we propose the Targeted Weighting algorithm that determines if, for a given set of weights, the out-of-sample performance will improve or not in a practical setting. This is necessary as the setting assumes there are no labeled points distributed according to the test distribution, only unlabeled samples.

Finally, we propose a new class of matching algorithms that can be used to match the training set to a desired distribution, such as the dual distribution (or the test distribution). These algorithms can be applied to very large datasets, and we show how they lead to improved performance in a large real dataset such as the Netflix dataset. Their computational complexity is the main reason for their advantage over previous algorithms proposed in the covariate shift literature.

In the second part of the thesis we apply Machine Learning to the problem of behavior recognition. We develop a specific behavior classifier to study fly aggression, and we develop a system that allows analyzing behavior in videos of animals, with minimal supervision. The system, which we call CUBA (Caltech Unsupervised Behavior Analysis), allows detecting movemes, actions, and stories from time series describing the position of animals in videos. The method summarizes the data, as well as it provides biologists with a mathematical tool to test new hypotheses. Other benefits of CUBA include finding classifiers for specific behaviors without the need for annotation, as well as providing means to discriminate groups of animals, for example, according to their genetic line.

Classical and Quantum Effects in Plasmonic Metals

The field of plasmonics exploits the unique optical properties of metallic nanostructures to concentrate and manipulate light at subwavelength length scales. Metallic nanostructures get their unique properties from their ability to support surface plasmons– coherent wave-like oscillations of the free electrons at the interface between a conductive and dielectric medium. Recent advancements in the ability to fabricate metallic nanostructures with subwavelength length scales have created new possibilities in technology and research in a broad range of applications.

In the first part of this thesis, we present two investigations of the relationship between the charge state and optical state of plasmonic metal nanoparticles. Using experimental bias-dependent extinction measurements, we derive a potential- dependent dielectric function for Au nanoparticles that accounts for changes in the physical properties due to an applied bias that contribute to the optical extinction. We also present theory and experiment for the reverse effect– the manipulation of the carrier density of Au nanoparticles via controlled optical excitation. This plasmoelectric effect takes advantage of the strong resonant properties of plasmonic materials and the relationship between charge state and optical properties to eluci- date a new avenue for conversion of optical power to electrical potential.

The second topic of this thesis is the non-radiative decay of plasmons to a hot-carrier distribution, and the distribution’s subsequent relaxation. We present first-principles calculations that capture all of the significant microscopic mechanisms underlying surface plasmon decay and predict the initial excited carrier distributions so generated. We also preform ab initio calculations of the electron-temperature dependent heat capacities and electron-phonon coupling coefficients of plasmonic metals. We extend these first-principle methods to calculate the electron-temperature dependent dielectric response of hot electrons in plasmonic metals, including direct interband and phonon-assisted intraband transitions. Finally, we combine these first-principles calculations of carrier dynamics and optical response to produce a complete theoretical description of ultrafast pump-probe measurements, free of any fitting parameters that are typical in previous analyses.

Metastability and dynamics of stem cells: from direct observations to inference at the single cell level

Organismal development, homeostasis, and pathology are rooted in inherently probabilistic events. From gene expression to cellular differentiation, rates and likelihoods shape the form and function of biology. Processes ranging from growth to cancer homeostasis to reprogramming of stem cells all require transitions between distinct phenotypic states, and these occur at defined rates. Therefore, measuring the fidelity and dynamics with which such transitions occur is central to understanding natural biological phenomena and is critical for therapeutic interventions.

While these processes may produce robust population-level behaviors, decisions are made by individual cells. In certain circumstances, these minuscule computing units effectively roll dice to determine their fate. And while the 'omics' era has provided vast amounts of data on what these populations are doing en masse, the behaviors of the underlying units of these processes get washed out in averages.

Therefore, in order to understand the behavior of a sample of cells, it is critical to reveal how its underlying components, or mixture of cells in distinct states, each contribute to the overall phenotype. As such, we must first define what states exist in the population, determine what controls the stability of these states, and measure in high dimensionality the dynamics with which these cells transition between states.

To address a specific example of this general problem, we investigate the heterogeneity and dynamics of mouse embryonic stem cells (mESCs). While a number of reports have identified particular genes in ES cells that switch between 'high' and 'low' metastable expression states in culture, it remains unclear how levels of many of these regulators combine to form states in transcriptional space. Using a method called single molecule mRNA fluorescent in situ hybridization (smFISH), we quantitatively measure and fit distributions of core pluripotency regulators in single cells, identifying a wide range of variabilities between genes, but each explained by a simple model of bursty transcription. From this data, we also observed that strongly bimodal genes appear to be co-expressed, effectively limiting the occupancy of transcriptional space to two primary states across genes studied here. However, these states also appear punctuated by the conditional expression of the most highly variable genes, potentially defining smaller substates of pluripotency.

Having defined the transcriptional states, we next asked what might control their stability or persistence. Surprisingly, we found that DNA methylation, a mark normally associated with irreversible developmental progression, was itself differentially regulated between these two primary states. Furthermore, both acute or chronic inhibition of DNA methyltransferase activity led to reduced heterogeneity among the population, suggesting that metastability can be modulated by this strong epigenetic mark.

Finally, because understanding the dynamics of state transitions is fundamental to a variety of biological problems, we sought to develop a high-throughput method for the identification of cellular trajectories without the need for cell-line engineering. We achieved this by combining cell-lineage information gathered from time-lapse microscopy with endpoint smFISH for measurements of final expression states. Applying a simple mathematical framework to these lineage-tree associated expression states enables the inference of dynamic transitions. We apply our novel approach in order to infer temporal sequences of events, quantitative switching rates, and network topology among a set of ESC states.

Taken together, we identify distinct expression states in ES cells, gain fundamental insight into how a strong epigenetic modifier enforces the stability of these states, and develop and apply a new method for the identification of cellular trajectories using scalable in situ readouts of cellular state.

Extracting material response from simple mechanical tests on hardening-softening-hardening viscoplastic solids

Compliant foams are usually characterized by a wide range of desirable mechanical properties. These properties include viscoelasticity at different temperatures, energy absorption, recoverability under cyclic loading, impact resistance, and thermal, electrical, acoustic and radiation-resistance. Some foams contain nano-sized features and are used in small-scale devices. This implies that the characteristic dimensions of foams span multiple length scales, rendering modeling their mechanical properties difficult. Continuum mechanics-based models capture some salient experimental features like the linear elastic regime, followed by non-linear plateau stress regime. However, they lack mesostructural physical details. This makes them incapable of accurately predicting local peaks in stress and strain distributions, which significantly affect the deformation paths. Atomistic methods are capable of capturing the physical origins of deformation at smaller scales, but suffer from impractical computational intensity. Capturing deformation at the so-called meso-scale, which is capable of describing the phenomenon at a continuum level, but with some physical insights, requires developing new theoretical approaches.

A fundamental question that motivates the modeling of foams is ‘how to extract the intrinsic material response from simple mechanical test data, such as stress vs. strain response?’ A 3D model was developed to simulate the mechanical response of foam-type materials. The novelty of this model includes unique features such as the hardening-softening-hardening material response, strain rate-dependence, and plastically compressible solids with plastic non-normality. Suggestive links from atomistic simulations of foams were borrowed to formulate a physically informed hardening material input function. Motivated by a model that qualitatively captured the response of foam-type vertically aligned carbon nanotube (VACNT) pillars under uniaxial compression [2011,“Analysis of Uniaxial Compression of Vertically Aligned Carbon Nanotubes,” J. Mech.Phys. Solids, 59, pp. 2227–2237, Erratum 60, 1753–1756 (2012)], the property space exploration was advanced to three types of simple mechanical tests: 1) uniaxial compression, 2) uniaxial tension, and 3) nanoindentation with a conical and a flat-punch tip. The simulations attempt to explain some of the salient features in experimental data, like
1) The initial linear elastic response.
2) One or more nonlinear instabilities, yielding, and hardening.

The model-inherent relationships between the material properties and the overall stress-strain behavior were validated against the available experimental data. The material properties include the gradient in stiffness along the height, plastic and elastic compressibility, and hardening. Each of these tests was evaluated in terms of their efficiency in extracting material properties. The uniaxial simulation results proved to be a combination of structural and material influences. Out of all deformation paths, flat-punch indentation proved to be superior since it is the most sensitive in capturing the material properties.

Approximate theory for the flow through a cascade of cambered airfoils

An approximate theory for steady irrotational flow through a cascade of thin cambered airfoils is developed. Isolated thin airfoils have only slight camber is most applications, and the well known methods that replace the source and vorticity distributions of the curved camber line by similar distributions on the straight chord line are adequate. In cascades, however, the camber is usually appreciable, and significant errors are introduced if the vorticity and source distributions on the camber line are approximated by the same distribution on the chord line.

The calculation of the flow field becomes very clumsy in practice if the vorticity and source distributions are not confined to a straight line. A new method is proposed and investigated; in this method, at each point on the camber line, the vorticity and sources are assumed to be distributed along a straight line tangent to the camber line at that point, and corrections are determined to account for the deviation of the actual camber line from the tangent line. Hence, the basic calculation for the cambered airfoils is reduced to the simpler calculation of the straight line airfoils, with the equivalent straight line airfoils changing from point to point.

The results of the approximate method are compared with numerical solutions for cambers as high as 25 per cent of the chord. The leaving angles of flow are predicted quite well, even at this high value of the camber. The present method also gives the functional relationship between the exit angle and the other parameters such as airfoil shape and cascade geometry.

Total cross section measurements for ^3He(^3He, 2p)^4He at low energies

Precise measurements of the total reaction cross section for ³He(³He,2p)⁴He He have been made in the range of center-of-mass energies between 1100 keV and 80 keV. A differentially pumped gas target modified to operate with a limited quantity of the target gas was employed to minimize the uncertainties in the primary energy and energy straggle. Beam integration inside the target gas was carried out by a calorimetric device which measures the total energy spent in a heat sink rather than the total charge in a Faraday cup. Proton energy spectra have been obtained using a counter telescope consisting of a gas proportional counter and a surface barrier detector and angular distributions of these protons have been measured at seven bombarding energies. Cross section factors, S(E), have been calculated from the total cross sections and fitted to a linear function of energy over different ranges of energy. For E_cm < 500 keV

S(E_cm) = S₀ + S₁ E_cm

where S₀ = (5.0 ^+0.6_-0.4) MeV - barns and S₁ = (-1.8 ± 0.5) barns.

Sputtering of the gallium-indium eutectic alloy in the liquid phase

We have measured sputtering yields and angular distributions of sputtered atoms from both the solid and liquid phases of gallium, indium, and the gallium-indium eutectic alloy. This was done by Rutherford backscattering analysis of graphite collector foils. The solid eutectic target shows a predominance of indium crystallites on its surface which have to be sputtered away before the composition of the sputtered atoms equals the bulk target composition. The size of the crystallites depends upon the conditions under which the alloy is frozen. The sputtering of the liquid eutectic alloy by 15 keV Ar⁺ results in a ratio of indium to gallium sputtering yields which is 28 times greater than would be expected from the target stoichiometry. Furthermore, the angular distribution of gallium is much more sharply peaked about the normal to the target surface than the indium distribution. When the incident Ar⁺ energy is increased to 25 keV, the gallium distribution broadens to the same shape as the indium distribution. With the exception of the sharp gallium distribution taken from the liquid eutectic at 15 keV, all angular distributions from liquid targets fit a cos² θ function. An ion-scattering-spectroscopy analysis of the liquid eutectic alloy reveals a surface layer of almost pure indium. A thermodynamic explanation for this highly segregated layer is discussed. The liquid eutectic alloy provides us with a unique target system which allows us to estimate the fraction of sputtered material which comes from the first monolayer of the surface.

Photoproduction of charged pion pairs from hydrogen at energies up to 1500 MeV

Charged pion pair photoproduction has been investigated up to a gamma energy of 1500 MeV, using the Caltech 12-inch heavy liquid bubble chamber with a small diameter, high intensity photon beam passing through a central beam tube gaseous hydrogen target surrounded by the sensitive Freon. Scanning, analysis, and data reduction techniques have been developed to deal with the problems of two-vie stereo, hidden event origins, absence of magnetic field, and the range-energy and multiple scattering relationships that occur in the heavy materials. Roughly 5700 pictures have been scanned and analyzed, yielding 754 acceptable events. Cross section and parameter distributions are generally consistent with the results of previous experiments. A statistically insignificant “bump” was observed in the dipion mass spectrum in the region of 500 MeV, the disputed σ meson mass. This region was investigated as carefully as the limited statistics would allow; dipion angular distributions are consistent with isotropy, and there is indication that some of the events in this region might come from decay of an intermediate N*₁₁ (1425) into a proton and dipion.

Photographic materials on pp. 18, 20, 22, and 24 are essential and will not reproduce clearly on Xerox copies. Photographic copies should be ordered.