Estimation and inference for grasping and manipulation tasks using vision and kinesthetic sensors

This thesis presents a novel framework for state estimation in the context of robotic grasping and manipulation. The overall estimation approach is based on fusing various visual cues for manipulator tracking, namely appearance and feature-based, shape-based, and silhouette-based visual cues. Similarly, a framework is developed to fuse the above visual cues, but also kinesthetic cues such as force-torque and tactile measurements, for in-hand object pose estimation. The cues are extracted from multiple sensor modalities and are fused in a variety of Kalman filters.

A hybrid estimator is developed to estimate both a continuous state (robot and object states) and discrete states, called contact modes, which specify how each finger contacts a particular object surface. A static multiple model estimator is used to compute and maintain this mode probability. The thesis also develops an estimation framework for estimating model parameters associated with object grasping. Dual and joint state-parameter estimation is explored for parameter estimation of a grasped object's mass and center of mass. Experimental results demonstrate simultaneous object localization and center of mass estimation.

Dual-arm estimation is developed for two arm robotic manipulation tasks. Two types of filters are explored; the first is an augmented filter that contains both arms in the state vector while the second runs two filters in parallel, one for each arm. These two frameworks and their performance is compared in a dual-arm task of removing a wheel from a hub.

This thesis also presents a new method for action selection involving touch. This next best touch method selects an available action for interacting with an object that will gain the most information. The algorithm employs information theory to compute an information gain metric that is based on a probabilistic belief suitable for the task. An estimation framework is used to maintain this belief over time. Kinesthetic measurements such as contact and tactile measurements are used to update the state belief after every interactive action. Simulation and experimental results are demonstrated using next best touch for object localization, specifically a door handle on a door. The next best touch theory is extended for model parameter determination. Since many objects within a particular object category share the same rough shape, principle component analysis may be used to parametrize the object mesh models. These parameters can be estimated using the action selection technique that selects the touching action which best both localizes and estimates these parameters. Simulation results are then presented involving localizing and determining a parameter of a screwdriver.

Lastly, the next best touch theory is further extended to model classes. Instead of estimating parameters, object class determination is incorporated into the information gain metric calculation. The best touching action is selected in order to best discern between the possible model classes. Simulation results are presented to validate the theory.

Responses to auditory stimuli in macaque lateral intraparietal area

The lateral intraparietal area (LIP) of macaque posterior parietal cortex participates in the sensorimotor transformations underlying visually guided eye movements. Area LIP has long been considered unresponsive to auditory stimulation. However, recent studies have shown that neurons in LIP respond to auditory stimuli during an auditory-saccade task, suggesting possible involvement of this area in auditory-to-oculomotor as well as visual-to-oculomotor processing. This dissertation describes investigations which clarify the role of area LIP in auditory-to-oculomotor processing.

Extracellular recordings were obtained from a total of 332 LIP neurons in two macaque monkeys, while the animals performed fixation and saccade tasks involving auditory and visual stimuli. No auditory activity was observed in area LIP before animals were trained to make saccades to auditory stimuli, but responses to auditory stimuli did emerge after auditory-saccade training. Auditory responses in area LIP after auditory-saccade training were significantly stronger in the context of an auditory-saccade task than in the context of a fixation task. Compared to visual responses, auditory responses were also significantly more predictive of movement-related activity in the saccade task. Moreover, while visual responses often had a fast transient component, responses to auditory stimuli in area LIP tended to be gradual in onset and relatively prolonged in duration.

Overall, the analyses demonstrate that responses to auditory stimuli in area LIP are dependent on auditory-saccade training, modulated by behavioral context, and characterized by slow-onset, sustained response profiles. These findings suggest that responses to auditory stimuli are best interpreted as supramodal (cognitive or motor) responses, rather than as modality-specific sensory responses. Auditory responses in area LIP seem to reflect the significance of auditory stimuli as potential targets for eye movements, and may differ from most visual responses in the extent to which they arc abstracted from the sensory parameters of the stimulus.

Characterization of the neural mechanisms supporting the implementation of cognitive control in human decision making

Humans are particularly adept at modifying their behavior in accordance with changing environmental demands. Through various mechanisms of cognitive control, individuals are able to tailor actions to fit complex short- and long-term goals. The research described in this thesis uses functional magnetic resonance imaging to characterize the neural correlates of cognitive control at two levels of complexity: response inhibition and self-control in intertemporal choice. First, we examined changes in neural response associated with increased experience and skill in response inhibition; successful response inhibition was associated with decreased neural response over time in the right ventrolateral prefrontal cortex, a region widely implicated in cognitive control, providing evidence for increased neural efficiency with learned automaticity. We also examined a more abstract form of cognitive control using intertemporal choice. In two experiments, we identified putative neural substrates for individual differences in temporal discounting, or the tendency to prefer immediate to delayed rewards. Using dynamic causal models, we characterized the neural circuit between ventromedial prefrontal cortex, an area involved in valuation, and dorsolateral prefrontal cortex, a region implicated in self-control in intertemporal and dietary choice, and found that connectivity from dorsolateral prefrontal cortex to ventromedial prefrontal cortex increases at the time of choice, particularly when delayed rewards are chosen. Moreover, estimates of the strength of connectivity predicted out-of-sample individual rates of temporal discounting, suggesting a neurocomputational mechanism for variation in the ability to delay gratification. Next, we interrogated the hypothesis that individual differences in temporal discounting are in part explained by the ability to imagine future reward outcomes. Using a novel paradigm, we imaged neural response during the imagining of primary rewards, and identified negative correlations between activity in regions associated the processing of both real and imagined rewards (lateral orbitofrontal cortex and ventromedial prefrontal cortex, respectively) and the individual temporal discounting parameters estimated in the previous experiment. These data suggest that individuals who are better able to represent reward outcomes neurally are less susceptible to temporal discounting. Together, these findings provide further insight into role of the prefrontal cortex in implementing cognitive control, and propose neurobiological substrates for individual variation.

Connectivity and Function of the Primate Insula

The insula is a mammalian cortical structure that has been implicated in a wide range of low- and high-level functions governing one’s sensory, emotional, and cognitive experiences. One particular role of this region is considered to be processing of olfactory stimuli. The ability to detect and evaluate odors has significant effects on an organism’s eating behavior and survival and, in case of humans, on complex decision making. Despite such importance of this function, the mechanism in which olfactory information is processed in the insula has not been thoroughly studied. Moreover, due to the structure’s close spatial relationship with the neighboring claustrum, it is not entirely clear whether the connectivity and olfactory functions attributed to the insula are truly those of the insula, rather than of the claustrum. My graduate work, consisting of two studies, seeks to help fill these gaps. In the first, the structural connectivity patterns of the insula and the claustrum in a non-human primate brain is assayed using an ultra-high-quality diffusion magnetic resonance image, and the results suggest dissociation of connectivity — and hence function — between the two structures. In the second study, a functional neuroimaging experiment investigates the insular activity during odor evaluation tasks in humans, and uncovers a potential spatial organization within the anterior portion of the insula for processing different aspects of odor characteristics.