Optimal Data Distributions in Machine Learning

In the first part of the thesis we explore three fundamental questions that arise naturally when we conceive a machine learning scenario where the training and test distributions can differ. Contrary to conventional wisdom, we show that in fact mismatched training and test distribution can yield better out-of-sample performance. This optimal performance can be obtained by training with the dual distribution. This optimal training distribution depends on the test distribution set by the problem, but not on the target function that we want to learn. We show how to obtain this distribution in both discrete and continuous input spaces, as well as how to approximate it in a practical scenario. Benefits of using this distribution are exemplified in both synthetic and real data sets.

In order to apply the dual distribution in the supervised learning scenario where the training data set is fixed, it is necessary to use weights to make the sample appear as if it came from the dual distribution. We explore the negative effect that weighting a sample can have. The theoretical decomposition of the use of weights regarding its effect on the out-of-sample error is easy to understand but not actionable in practice, as the quantities involved cannot be computed. Hence, we propose the Targeted Weighting algorithm that determines if, for a given set of weights, the out-of-sample performance will improve or not in a practical setting. This is necessary as the setting assumes there are no labeled points distributed according to the test distribution, only unlabeled samples.

Finally, we propose a new class of matching algorithms that can be used to match the training set to a desired distribution, such as the dual distribution (or the test distribution). These algorithms can be applied to very large datasets, and we show how they lead to improved performance in a large real dataset such as the Netflix dataset. Their computational complexity is the main reason for their advantage over previous algorithms proposed in the covariate shift literature.

In the second part of the thesis we apply Machine Learning to the problem of behavior recognition. We develop a specific behavior classifier to study fly aggression, and we develop a system that allows analyzing behavior in videos of animals, with minimal supervision. The system, which we call CUBA (Caltech Unsupervised Behavior Analysis), allows detecting movemes, actions, and stories from time series describing the position of animals in videos. The method summarizes the data, as well as it provides biologists with a mathematical tool to test new hypotheses. Other benefits of CUBA include finding classifiers for specific behaviors without the need for annotation, as well as providing means to discriminate groups of animals, for example, according to their genetic line.

The stochastic exit problem for dynamical systems

The problem of "exit against a flow" for dynamical systems subject to small Gaussian white noise excitation is studied. Here the word "flow" refers to the behavior in phase space of the unperturbed system's state variables. "Exit against a flow" occurs if a perturbation causes the phase point to leave a phase space region within which it would normally be confined. In particular, there are two components of the problem of exit against a flow:

i) the mean exit time

ii) the phase-space distribution of exit locations.

When the noise perturbing the dynamical systems is small, the solution of each component of the problem of exit against a flow is, in general, the solution of a singularly perturbed, degenerate elliptic-parabolic boundary value problem.

Singular perturbation techniques are used to express the asymptotic solution in terms of an unknown parameter. The unknown parameter is determined using the solution of the adjoint boundary value problem.

The problem of exit against a flow for several dynamical systems of physical interest is considered, and the mean exit times and distributions of exit positions are calculated. The systems are then simulated numerically, using Monte Carlo techniques, in order to determine the validity of the asymptotic solutions.

The strategic behavior of rational novices

There is a growing amount of experimental evidence that suggests people often deviate from the predictions of game theory. Some scholars attempt to explain the observations by introducing errors into behavioral models. However, most of these modifications are situation dependent and do not generalize. A new theory, called the rational novice model, is introduced as an attempt to provide a general theory that takes account of erroneous behavior. The rational novice model is based on two central principals. The first is that people systematically make inaccurate guesses when they are evaluating their options in a game-like situation. The second is that people treat their decisions similar to a portfolio problem. As a result, non optimal actions in a game theoretic sense may be included in the rational novice strategy profile with positive weights.

The rational novice model can be divided into two parts: the behavioral model and the equilibrium concept. In a theoretical chapter, the mathematics of the behavioral model and the equilibrium concept are introduced. The existence of the equilibrium is established. In addition, the Nash equilibrium is shown to be a special case of the rational novice equilibrium. In another chapter, the rational novice model is applied to a voluntary contribution game. Numerical methods were used to obtain the solution. The model is estimated with data obtained from the Palfrey and Prisbrey experimental study of the voluntary contribution game. It is found that the rational novice model explains the data better than the Nash model. Although a formal statistical test was not used, pseudo R^2 analysis indicates that the rational novice model is better than a Probit model similar to the one used in the Palfrey and Prisbrey study.

The rational novice model is also applied to a first price sealed bid auction. Again, computing techniques were used to obtain a numerical solution. The data obtained from the Chen and Plott study were used to estimate the model. The rational novice model outperforms the CRRAM, the primary Nash model studied in the Chen and Plott study. However, the rational novice model is not the best amongst all models. A sophisticated rule-of-thumb, called the SOPAM, offers the best explanation of the data.

Mathematical study of complex networks : brain, internet, and power grid

The dissertation is concerned with the mathematical study of various network problems. First, three real-world networks are considered: (i) the human brain network (ii) communication networks, (iii) electric power networks. Although these networks perform very different tasks, they share similar mathematical foundations. The high-level goal is to analyze and/or synthesis each of these systems from a “control and optimization” point of view. After studying these three real-world networks, two abstract network problems are also explored, which are motivated by power systems. The first one is “flow optimization over a flow network” and the second one is “nonlinear optimization over a generalized weighted graph”. The results derived in this dissertation are summarized below.

Brain Networks: Neuroimaging data reveals the coordinated activity of spatially distinct brain regions, which may be represented mathematically as a network of nodes (brain regions) and links (interdependencies). To obtain the brain connectivity network, the graphs associated with the correlation matrix and the inverse covariance matrix—describing marginal and conditional dependencies between brain regions—have been proposed in the literature. A question arises as to whether any of these graphs provides useful information about the brain connectivity. Due to the electrical properties of the brain, this problem will be investigated in the context of electrical circuits. First, we consider an electric circuit model and show that the inverse covariance matrix of the node voltages reveals the topology of the circuit. Second, we study the problem of finding the topology of the circuit based on only measurement. In this case, by assuming that the circuit is hidden inside a black box and only the nodal signals are available for measurement, the aim is to find the topology of the circuit when a limited number of samples are available. For this purpose, we deploy the graphical lasso technique to estimate a sparse inverse covariance matrix. It is shown that the graphical lasso may find most of the circuit topology if the exact covariance matrix is well-conditioned. However, it may fail to work well when this matrix is ill-conditioned. To deal with ill-conditioned matrices, we propose a small modification to the graphical lasso algorithm and demonstrate its performance. Finally, the technique developed in this work will be applied to the resting-state fMRI data of a number of healthy subjects.

Communication Networks: Congestion control techniques aim to adjust the transmission rates of competing users in the Internet in such a way that the network resources are shared efficiently. Despite the progress in the analysis and synthesis of the Internet congestion control, almost all existing fluid models of congestion control assume that every link in the path of a flow observes the original source rate. To address this issue, a more accurate model is derived in this work for the behavior of the network under an arbitrary congestion controller, which takes into account of the effect of buffering (queueing) on data flows. Using this model, it is proved that the well-known Internet congestion control algorithms may no longer be stable for the common pricing schemes, unless a sufficient condition is satisfied. It is also shown that these algorithms are guaranteed to be stable if a new pricing mechanism is used.

Electrical Power Networks: Optimal power flow (OPF) has been one of the most studied problems for power systems since its introduction by Carpentier in 1962. This problem is concerned with finding an optimal operating point of a power network minimizing the total power generation cost subject to network and physical constraints. It is well known that OPF is computationally hard to solve due to the nonlinear interrelation among the optimization variables. The objective is to identify a large class of networks over which every OPF problem can be solved in polynomial time. To this end, a convex relaxation is proposed, which solves the OPF problem exactly for every radial network and every meshed network with a sufficient number of phase shifters, provided power over-delivery is allowed. The concept of “power over-delivery” is equivalent to relaxing the power balance equations to inequality constraints.

Flow Networks: In this part of the dissertation, the minimum-cost flow problem over an arbitrary flow network is considered. In this problem, each node is associated with some possibly unknown injection, each line has two unknown flows at its ends related to each other via a nonlinear function, and all injections and flows need to satisfy certain box constraints. This problem, named generalized network flow (GNF), is highly non-convex due to its nonlinear equality constraints. Under the assumption of monotonicity and convexity of the flow and cost functions, a convex relaxation is proposed, which always finds the optimal injections. A primary application of this work is in the OPF problem. The results of this work on GNF prove that the relaxation on power balance equations (i.e., load over-delivery) is not needed in practice under a very mild angle assumption.

Generalized Weighted Graphs: Motivated by power optimizations, this part aims to find a global optimization technique for a nonlinear optimization defined over a generalized weighted graph. Every edge of this type of graph is associated with a weight set corresponding to the known parameters of the optimization (e.g., the coefficients). The motivation behind this problem is to investigate how the (hidden) structure of a given real/complex valued optimization makes the problem easy to solve, and indeed the generalized weighted graph is introduced to capture the structure of an optimization. Various sufficient conditions are derived, which relate the polynomial-time solvability of different classes of optimization problems to weak properties of the generalized weighted graph such as its topology and the sign definiteness of its weight sets. As an application, it is proved that a broad class of real and complex optimizations over power networks are polynomial-time solvable due to the passivity of transmission lines and transformers.

The short-timescale behavior of glacial ice

Glaciers are often assumed to deform only at slow (i.e., glacial) rates. However, with the advent of high rate geodetic observations of ice motion, many of the intricacies of glacial deformation on hourly and daily timescales have been observed and quantified. This thesis explores two such short timescale processes: the tidal perturbation of ice stream motion and the catastrophic drainage of supraglacial meltwater lakes. Our investigation into the transmission length-scale of a tidal load represents the first study to explore the daily tidal influence on ice stream motion using three-dimensional models. Our results demonstrate both that the implicit assumptions made in the standard two-dimensional flow-line models are inherently incorrect for many ice streams, and that the anomalously large spatial extent of the tidal influence seen on the motion of some glaciers cannot be explained, as previously thought, through the elastic or viscoelastic transmission of tidal loads through the bulk of the ice stream. We then discuss how the phase delay between a tidal forcing and the ice stream’s displacement response can be used to constrain in situ viscoelastic properties of glacial ice. Lastly, for the problem of supraglacial lake drainage, we present a methodology for implementing linear viscoelasticity into an existing model for lake drainage. Our work finds that viscoelasticity is a second-order effect when trying to model the deformation of ice in response to a meltwater lake draining to a glacier’s bed. The research in this thesis demonstrates that the first-order understanding of the short-timescale behavior of naturally occurring ice is incomplete, and works towards improving our fundamental understanding of ice behavior over the range of hours to days.

Signatures of Topological Superconductors

Topological superconductors are particularly interesting in light of the active ongoing experimental efforts for realizing exotic physics such as Majorana zero modes. These systems have excitations with non-Abelian exchange statistics, which provides a path towards topological quantum information processing. Intrinsic topological superconductors are quite rare in nature. However, one can engineer topological superconductivity by inducing effective p-wave pairing in materials which can be grown in the laboratory. One possibility is to induce the proximity effect in topological insulators; another is to use hybrid structures of superconductors and semiconductors.

The proposal of interfacing s-wave superconductors with quantum spin Hall systems provides a promising route to engineered topological superconductivity. Given the exciting recent progress on the fabrication side, identifying experiments that definitively expose the topological superconducting phase (and clearly distinguish it from a trivial state) raises an increasingly important problem. With this goal in mind, we proposed a detection scheme to get an unambiguous signature of topological superconductivity, even in the presence of ordinarily detrimental effects such as thermal fluctuations and quasiparticle poisoning. We considered a Josephson junction built on top of a quantum spin Hall material. This system allows the proximity effect to turn edge states in effective topological superconductors. Such a setup is promising because experimentalists have demonstrated that supercurrents indeed flow through quantum spin Hall edges. To demonstrate the topological nature of the superconducting quantum spin Hall edges, theorists have proposed examining the periodicity of Josephson currents respect to the phase across a Josephson junction. The periodicity of tunneling currents of ground states in a topological superconductor Josephson junction is double that of a conventional Josephson junction. In practice, this modification of periodicity is extremely difficult to observe because noise sources, such as quasiparticle poisoning, wash out the signature of topological superconductors. For this reason, We propose a new, relatively simple DC measurement that can compellingly reveal topological superconductivity in such quantum spin Hall/superconductor heterostructures. More specifically, We develop a general framework for capturing the junction's current-voltage characteristics as a function of applied magnetic flux. Our analysis reveals sharp signatures of topological superconductivity in the field-dependent critical current. These signatures include the presence of multiple critical currents and a non-vanishing critical current for all magnetic field strengths as a reliable identification scheme for topological superconductivity.

This system becomes more interesting as interactions between electrons are involved. By modeling edge states as a Luttinger liquid, we find conductance provides universal signatures to distinguish between normal and topological superconductors. More specifically, we use renormalization group methods to extract universal transport characteristics of superconductor/quantum spin Hall heterostructures where the native edge states serve as a lead. Interestingly, arbitrarily weak interactions induce qualitative changes in the behavior relative to the free-fermion limit, leading to a sharp dichotomy in conductance for the trivial (narrow superconductor) and topological (wide superconductor) cases. Furthermore, we find that strong interactions can in principle induce parafermion excitations at a superconductor/quantum spin Hall junction.

As we identify the existence of topological superconductor, we can take a step further. One can use topological superconductor for realizing Majorana modes by breaking time reversal symmetry. An advantage of 2D topological insulator is that networks required for braiding Majoranas along the edge channels can be obtained by adjoining 2D topological insulator to form corner junctions. Physically cutting quantum wells for this purpose, however, presents technical challenges. For this reason, I propose a more accessible means of forming networks that rely on dynamically manipulating the location of edge states inside of a single 2D topological insulator sheet. In particular, I show that edge states can effectively be dragged into the system's interior by gating a region near the edge into a metallic regime and then removing the resulting gapless carriers via proximity-induced superconductivity. This method allows one to construct rather general quasi-1D networks along which Majorana modes can be exchanged by electrostatic means.

Apart from 2D topological insulators, Majorana fermions can also be generated in other more accessible materials such as semiconductors. Following up on a suggestion by experimentalist Charlie Marcus, I proposed a novel geometry to create Majorana fermions by placing a 2D electron gas in proximity to an interdigitated superconductor-ferromagnet structure. This architecture evades several manufacturing challenges by allowing single-side fabrication and widening the class of 2D electron gas that may be used, such as the surface states of bulk semiconductors. Furthermore, it naturally allows one to trap and manipulate Majorana fermions through the application of currents. Thus, this structure may lead to the development of a circuit that enables fully electrical manipulation of topologically-protected quantum memory. To reveal these exotic Majorana zero modes, I also proposed an interference scheme to detect Majorana fermions that is broadly applicable to any 2D topological superconductor platform.

A singular perturbation method of calculating the behavior of supercavitating hydrofoils with rounded noses

A simple, direct and accurate method to predict the pressure distribution on supercavitating hydrofoils with rounded noses is presented. The thickness of body and cavity is assumed to be small. The method adopted in the present work is that of singular perturbation theory. Far from the leading edge linearized free streamline theory is applied. Near the leading edge, however, where singularities of the linearized theory occur, a non-linear local solution is employed. The two unknown parameters which characterize this local solution are determined by a matching procedure. A uniformly valid solution is then constructed with the aid of the singular perturbation approach.

The present work is divided into two parts. In Part I isolated supercavitating hydrofoils of arbitrary profile shape with parabolic noses are investigated by the present method and its results are compared with the new computational results made with Wu and Wang's exact "functional iterative" method. The agreement is very good. In Part II this method is applied to a linear cascade of such hydrofoils with elliptic noses. A number of cases are worked out over a range of cascade parameters from which a good idea of the behavior of this type of important flow configuration is obtained.

Some of the computational aspects of Wu and Wang's functional iterative method heretofore not successfully applied to this type of problem are described in an appendix.

A novel methodology for simulating contact-line behavior in capillary-driven flows

Despite the wide swath of applications where multiphase fluid contact lines exist, there is still no consensus on an accurate and general simulation methodology. Most prior numerical work has imposed one of the many dynamic contact-angle theories at solid walls. Such approaches are inherently limited by the theory accuracy. In fact, when inertial effects are important, the contact angle may be history dependent and, thus, any single mathematical function is inappropriate. Given these limitations, the present work has two primary goals: 1) create a numerical framework that allows the contact angle to evolve naturally with appropriate contact-line physics and 2) develop equations and numerical methods such that contact-line simulations may be performed on coarse computational meshes.

Fluid flows affected by contact lines are dominated by capillary stresses and require accurate curvature calculations. The level set method was chosen to track the fluid interfaces because it is easy to calculate interface curvature accurately. Unfortunately, the level set reinitialization suffers from an ill-posed mathematical problem at contact lines: a ``blind spot'' exists. Standard techniques to handle this deficiency are shown to introduce parasitic velocity currents that artificially deform freely floating (non-prescribed) contact angles. As an alternative, a new relaxation equation reinitialization is proposed to remove these spurious velocity currents and its concept is further explored with level-set extension velocities.

To capture contact-line physics, two classical boundary conditions, the Navier-slip velocity boundary condition and a fixed contact angle, are implemented in direct numerical simulations (DNS). DNS are found to converge only if the slip length is well resolved by the computational mesh. Unfortunately, since the slip length is often very small compared to fluid structures, these simulations are not computationally feasible for large systems. To address the second goal, a new methodology is proposed which relies on the volumetric-filtered Navier-Stokes equations. Two unclosed terms, an average curvature and a viscous shear VS, are proposed to represent the missing microscale physics on a coarse mesh.

All of these components are then combined into a single framework and tested for a water droplet impacting a partially-wetting substrate. Very good agreement is found for the evolution of the contact diameter in time between the experimental measurements and the numerical simulation. Such comparison would not be possible with prior methods, since the Reynolds number Re and capillary number Ca are large. Furthermore, the experimentally approximated slip length ratio is well outside of the range currently achievable by DNS. This framework is a promising first step towards simulating complex physics in capillary-dominated flows at a reasonable computational expense.