Estimation and inference for grasping and manipulation tasks using vision and kinesthetic sensors

This thesis presents a novel framework for state estimation in the context of robotic grasping and manipulation. The overall estimation approach is based on fusing various visual cues for manipulator tracking, namely appearance and feature-based, shape-based, and silhouette-based visual cues. Similarly, a framework is developed to fuse the above visual cues, but also kinesthetic cues such as force-torque and tactile measurements, for in-hand object pose estimation. The cues are extracted from multiple sensor modalities and are fused in a variety of Kalman filters.

A hybrid estimator is developed to estimate both a continuous state (robot and object states) and discrete states, called contact modes, which specify how each finger contacts a particular object surface. A static multiple model estimator is used to compute and maintain this mode probability. The thesis also develops an estimation framework for estimating model parameters associated with object grasping. Dual and joint state-parameter estimation is explored for parameter estimation of a grasped object's mass and center of mass. Experimental results demonstrate simultaneous object localization and center of mass estimation.

Dual-arm estimation is developed for two arm robotic manipulation tasks. Two types of filters are explored; the first is an augmented filter that contains both arms in the state vector while the second runs two filters in parallel, one for each arm. These two frameworks and their performance is compared in a dual-arm task of removing a wheel from a hub.

This thesis also presents a new method for action selection involving touch. This next best touch method selects an available action for interacting with an object that will gain the most information. The algorithm employs information theory to compute an information gain metric that is based on a probabilistic belief suitable for the task. An estimation framework is used to maintain this belief over time. Kinesthetic measurements such as contact and tactile measurements are used to update the state belief after every interactive action. Simulation and experimental results are demonstrated using next best touch for object localization, specifically a door handle on a door. The next best touch theory is extended for model parameter determination. Since many objects within a particular object category share the same rough shape, principle component analysis may be used to parametrize the object mesh models. These parameters can be estimated using the action selection technique that selects the touching action which best both localizes and estimates these parameters. Simulation results are then presented involving localizing and determining a parameter of a screwdriver.

Lastly, the next best touch theory is further extended to model classes. Instead of estimating parameters, object class determination is incorporated into the information gain metric calculation. The best touching action is selected in order to best discern between the possible model classes. Simulation results are presented to validate the theory.

Genetically engineered sensors of cell signaling

Measuring electrical activity in large numbers of cells with high spatial and temporal resolution is a fundamental problem for the study of neural development and information processing. To address this problem, we have constructed FlaSh: a novel, genetically-encoded probe that can be used to measure trans-membrane voltage in single cells. We fused a modified green fluorescent protein (GFP) into a voltage-sensitive potassium channel so that voltage dependent rearrangements in the potassium channel induce changes in the fluorescence of GFP. A voltage sensor encoded into DNA has the advantage that it may be introduced into an organism non-invasively and targeted to specific developmental stages, brain regions, cell types, and sub-cellular compartments.

We also describe modifications to FlaSh that shift its color, kinetics, and dynamic range. We used multiple green fluorescent proteins to produce variants of the FlaSh sensor that generate ratiometric signal output via fluorescence resonance energy transfer (FRET). Finally, we describe initial work toward FlaSh variants that are sensitive to G-protein coupled receptor (GPCR) activation. These sensors can be used to design functional assays for receptor activation in living cells.

Optomechanical Inertial Sensors and Feedback Cooling

The optomechanical interaction is an extremely powerful tool with which to measure mechanical motion. The displacement resolution of chip-scale optomechanical systems has been measured on the order of 1⁄10th of a proton radius. So strong is this optomechanical interaction that it has recently been used to remove almost all thermal noise from a mechanical resonator and observe its quantum ground-state of motion starting from cryogenic temperatures.

In this work, chapter 1 describes the basic physics of the canonical optomechanical system, optical measurement techniques, and how the optomechanical interaction affects the coupled mechanical resonator. In chapter 2, we describe our techniques for realizing this canonical optomechanical system in a chip-scale form factor.

In chapter 3, we describe an experiment where we used radiation pressure feedback to cool a mesoscopic mechanical resonator near its quantum ground-state from room-temperature. We cooled the resonator from a room temperature phonon occupation of <n> = 6.5 million to an occupation of <n> = 66, which means the resonator is in its ground state approximately 2% of the time, while being coupled to a room-temperature thermal environment. At the time of this work, this is the closest a mesoscopic mechanical resonator has been to its ground-state of motion at room temperature, and this work begins to open the door to room-temperature quantum control of mechanical objects.

Chapter 4 begins with the realization that the displacement resolutions achieved by optomechanical systems can surpass those of conventional MEMS sensors by an order of magnitude or more. This provides the motivation to develop and calibrate an optomechanical accelerometer with a resolution of approximately 10 micro-g/rt-Hz over a bandwidth of approximately 30 kHz. In chapter 5, we improve upon the performance and practicality of this sensor by greatly increasing the test mass size, investigating and reducing low-frequency noise, and incorporating more robust optical coupling techniques and capacitive wavelength tuning. Finally, in chapter 6 we present our progress towards developing another optomechanical inertial sensor - a gyroscope.