Analog VLSI-based, neuromorphic sensorimotor systems: modeling the primate oculomotor system

Using neuromorphic analog VLSI techniques for modeling large neural systems has several advantages over software techniques. By designing massively-parallel analog circuit arrays which are ubiquitous in neural systems, analog VLSI models are extremely fast, particularly when local interactions are important in the computation. While analog VLSI circuits are not as flexible as software methods, the constraints posed by this approach are often very similar to the constraints faced by biological systems. As a result, these constraints can offer many insights into the solutions found by evolution. This dissertation describes a hardware modeling effort to mimic the primate oculomotor system which requires both fast sensory processing and fast motor control. A one-dimensional hardware model of the primate eye has been built which simulates the physical dynamics of the biological system. It is driven by analog VLSI circuits mimicking brainstem and cortical circuits that control eye movements. In this framework, a visually-triggered saccadic system is demonstrated which generates averaging saccades. In addition, an auditory localization system, based on the neural circuits of the barn owl, is used to trigger saccades to acoustic targets in parallel with visual targets. Two different types of learning are also demonstrated on the saccadic system using floating-gate technology allowing the non-volatile storage of analog parameters directly on the chip. Finally, a model of visual attention is used to select and track moving targets against textured backgrounds, driving both saccadic and smooth pursuit eye movements to maintain the image of the target in the center of the field of view. This system represents one of the few efforts in this field to integrate both neuromorphic sensory processing and motor control in a closed-loop fashion.

Essays in revealed preference theory and behavioral economics

Time, risk, and attention are all integral to economic decision making. The aim of this work is to understand those key components of decision making using a variety of approaches: providing axiomatic characterizations to investigate time discounting, generating measures of visual attention to infer consumers' intentions, and examining data from unique field settings.

Chapter 2, co-authored with Federico Echenique and Kota Saito, presents the first revealed-preference characterizations of exponentially-discounted utility model and its generalizations. My characterizations provide non-parametric revealed-preference tests. I apply the tests to data from a recent experiment, and find that the axiomatization delivers new insights on a dataset that had been analyzed by traditional parametric methods.

Chapter 3, co-authored with Min Jeong Kang and Colin Camerer, investigates whether "pre-choice" measures of visual attention improve in prediction of consumers' purchase intentions. We measure participants' visual attention using eyetracking or mousetracking while they make hypothetical as well as real purchase decisions. I find that different patterns of visual attention are associated with hypothetical and real decisions. I then demonstrate that including information on visual attention improves prediction of purchase decisions when attention is measured with mousetracking.

Chapter 4 investigates individuals' attitudes towards risk in a high-stakes environment using data from a TV game show, Jeopardy!. I first quantify players' subjective beliefs about answering questions correctly. Using those beliefs in estimation, I find that the representative player is risk averse. I then find that trailing players tend to wager more than "folk" strategies that are known among the community of contestants and fans, and this tendency is related to their confidence. I also find gender differences: male players take more risk than female players, and even more so when they are competing against two other male players.

Chapter 5, co-authored with Colin Camerer, investigates the dynamics of the favorite-longshot bias (FLB) using data on horse race betting from an online exchange that allows bettors to trade "in-play." I find that probabilistic forecasts implied by market prices before start of the races are well-calibrated, but the degree of FLB increases significantly as the events approach toward the end.

听力障碍与听力正常群体视觉注意发展及其机制的比较

One of the most important functions in the individual development is the interaction and integration of each sensory input. There exist two competing theories, i.e. the deficiency theory and the compensatory theory, regarding the origin and nature of changes in visual functions observed after auditory deprivation. The deficiency theory proposed that integrative processes are essential for normal development. In contrast, the compensatory theory stated that the loss of one sense may be met by a greater reliance upon, therefore an enhancement of the remaining senses. Given that hearing impaired children’s learning depends primarily on visual information, it is important to recognize the differences of visual attention between them and their hearing age-mates. Differences among age groups could exist in either selectivity or sustained attention. Study 1 and study 2 explored the selective and sustained attention development of hearing impaired and hearing students with average cognitive ability, aged from 7 years to college students. The analysis and discussion of the results are based on the visual attention development as well as deficiency theory and compensatory theory. According to the results of the study 1 and study 2, the spatial distribution and controlling of the visual attention between hearing impaired and hearing students were also investigated in the study 3 and study 4. The present work showed that: Firstly, both hearing impaired and hearing participants had the similar developmental trajectory of the sustained attention. The ability of children’s sustained attention appeared to improve with age, and in adolescence it reached the peak. The hearing impaired participants had the comparable sustained attention skills to the matched hearing ones. Besides, the results of the hearing impaired participants showed that they could maintain their attention and vigilance on the current task over the observation period. Secondly, group differences of visual attention development were found between hearing impaired and hearing participants. In the childhood, the visual attention developmental speed of the hearing impaired children was slower than that of the hearing ones. The selective attention skill of the hearing impaired were not comparable to the hearing ones, however, their selective skill improved with age, so in the adulthood, hearing impaired students showed the slight advantage in the selective attention skill over the hearing ones. Thirdly, hearing impaired and hearing participants showed the similar spatial distribution in the attention resources. In the low perceptual load condition, both participants were suffered great interference of the distrator at the fixation. In contrast, in the high perceptual load condition, hearing impaired adults were suffered more interference of the peripheral distractor, which suggested that they distributed more attention resources to the peripheral field when faced difficult tasks. Fourthly, both groups showed similar processing in the visual attention tasks. That is, they both searched the target with only the color feature in a parallel way, but in a serial way while processing orientation feature and the features with the combination of the color and orientation. Furthermore, the results indicated that two groups show similar ways in the attention controlling. In summary, the present study showed that visual attention development was dependent upon the integration of multimodal sensory information. Because of the interaction and integration of the input from various sensory, it has a negative impact on the intact sensory at the early stage of one sensory loss, however, it can better the functions of other intact sensory gradually with development and practice.

内源性注意与外源性注意神经机制的渐合假说

The relationship between exogenous attention and endogenous attention is still unclear. We conclude the result of previous studies and our experiments by proposing a convergence hypothesis. A lot of evidences from ERP and fMRI proved the distinction between exogenous and endogenous attention. The distinction is fit for the complicated surroundings. Some studies found that exogenous and endogenous attention activated some common brain areas but if there’s any common function in these areas was still unclear. Previous studies failed to disentangle the relationship between these two mechanisms in detail, and most of the previous studies compared these two mechanisms separately. We used the cue-target paradigm, controlled the SOA and the cue validity to separate endogenous and exogenous attention. We try to figure out the relationship between endogenous and exogenous attention. We found that, in the exogenous attention condition, the amplitude of the P1 component was larger and the amplitude of N1 was smaller than that in the invalid trials. In the endogenous attention condition, the amplitudes of the P1 and N1 components in valid trials were both larger than that in the invalid trials. The difference wave of endogenous and exogenous attentions showed that at the beginning of their process endogenous and exogenous attentions were separate and then the process was going to converge. The stimuli appear in the visual field will activate the “external trigger” in exogenous attention processing, and then the signal will be transferred to next stage, in the endogenous attention processing, when the observer want to attend, the “inner trigger” will be activated and the signal will be transfered. The difference between “external trigger” and “inner trigger” in endogenous and exogenous attention was significant, so the difference at the beginning of the process was significant. When the signal was transferred to the next processing stage, the difference between the endogenous and exogenous attention became diminished, and it was a convergence process. We used the cue-target paradigm, let the central cue and peripheral cue appear in one trial, try to investigate the relationship between endogenous and exogenous attention under different perceptual load conditions. The result from of difference wave showed that the relationship between the two attentions was consistent with the convergence hypothesis. We also found that under different perceptual load conditions, the relationship between endogenous and exogenous attention was different. The perceptual load has influence on the “external trigger” and “inner trigger”. A cue-target paradigm was used to investigate the relationship between attention, age and perceptual load, we found that the function of visual attention was decrease in the old group, they need more attentional resource when they face the issue. The aging effect has influence on the “external trigger” and “inner trigger”. The effects of age and perceptual load were much higher in exogenous attention condition.

Vision, Instruction, and Action

This thesis describes Sonja, a system which uses instructions in the course of visually-guided activity. The thesis explores an integration of research in vision, activity, and natural language pragmatics. Sonja's visual system demonstrates the use of several intermediate visual processes, particularly visual search and routines, previously proposed on psychophysical grounds. The computations Sonja performs are compatible with the constraints imposed by neuroscientifically plausible hardware. Although Sonja can operate autonomously, it can also make flexible use of instructions provided by a human advisor. The system grounds its understanding of these instructions in perception and action.

Computer vision tools for low-cost and noninvasive measurement of autism-related behaviors in infants.

The early detection of developmental disorders is key to child outcome, allowing interventions to be initiated which promote development and improve prognosis. Research on autism spectrum disorder (ASD) suggests that behavioral signs can be observed late in the first year of life. Many of these studies involve extensive frame-by-frame video observation and analysis of a child's natural behavior. Although nonintrusive, these methods are extremely time-intensive and require a high level of observer training; thus, they are burdensome for clinical and large population research purposes. This work is a first milestone in a long-term project on non-invasive early observation of children in order to aid in risk detection and research of neurodevelopmental disorders. We focus on providing low-cost computer vision tools to measure and identify ASD behavioral signs based on components of the Autism Observation Scale for Infants (AOSI). In particular, we develop algorithms to measure responses to general ASD risk assessment tasks and activities outlined by the AOSI which assess visual attention by tracking facial features. We show results, including comparisons with expert and nonexpert clinicians, which demonstrate that the proposed computer vision tools can capture critical behavioral observations and potentially augment the clinician's behavioral observations obtained from real in-clinic assessments.

Computer vision tools for the non-invasive assessment of autism-related behavioral markers

The early detection of developmental disorders is key to child outcome, allowing interventions to be initiated that promote development and improve prognosis. Research on autism spectrum disorder (ASD) suggests behavioral markers can be observed late in the first year of life. Many of these studies involved extensive frame-by-frame video observation and analysis of a child's natural behavior. Although non-intrusive, these methods are extremely time-intensive and require a high level of observer training; thus, they are impractical for clinical and large population research purposes. Diagnostic measures for ASD are available for infants but are only accurate when used by specialists experienced in early diagnosis. This work is a first milestone in a long-term multidisciplinary project that aims at helping clinicians and general practitioners accomplish this early detection/measurement task automatically. We focus on providing computer vision tools to measure and identify ASD behavioral markers based on components of the Autism Observation Scale for Infants (AOSI). In particular, we develop algorithms to measure three critical AOSI activities that assess visual attention. We augment these AOSI activities with an additional test that analyzes asymmetrical patterns in unsupported gait. The first set of algorithms involves assessing head motion by tracking facial features, while the gait analysis relies on joint foreground segmentation and 2D body pose estimation in video. We show results that provide insightful knowledge to augment the clinician's behavioral observations obtained from real in-clinic assessments.

Effect of Addiction Modeling Reinforcement Schedules on Delay Discounting

Elevated delay discounting, in which delayed rewards quickly lose value as a function of time, is associated with substance use and abuse. Currently, the direction of causation is unclear: while some research indicates that elevated delay discounting leads to future substance use, it is also possible that chronic substance use and specifically the rate of reinforcement associated with drug use, leads to elevated delay discounting. This project aims to examine the latter possibility. 47 participants completed ten 30-minute daily sessions of a visual attention task, and were reinforced at a rate intended to model drug use (fixed ratio 1) or drug abstinence (fixed ratio 10). Baseline and post-training rates of delay discounting were assessed for hypothetical $50 and $1000. Area under the curve of the indifference points as a function of delay was calculated. A greater area under the curve suggests more self-control, whereas a lower value represents more impulsiveness. Results at the monetary value of both $50 and $1000 showed increased impulsivity in relation to the control for both the FR1 and FR10 groups indicating that the two schedules may both model drug use.

The photography and the motion landscape in the urban expressway

A new kind of photographic representation, called movement-image is proposed and discussed to record the visual experience of the journey through urban highways. It consists of performing long exposure photographic shots while the track is traversed, thus registering a time-panorama which includes landscape signs and inner spaces of the ways involved. This proposal is linked to the limitations of representing these expressways, if they are understood as structures of instrumental origin, where the resulting experience comes from moving at high speed through the territory. In al almost all cases the aesthetic approach or urban integration with the city and landscape are excluded. In this sense, although such structures may be an opportunity to collect, build and colonize the urban landscape, the lack of adequate representation of the phenomenon causes a difficulty in its understanding and transformation. The options for representation using photography is assumed, knowing its own particular tradition in the use of long exposures, for the expression of the mobile, and the multiple visual attention, divided or weakened.

Brain responses and looking behavior during audiovisual speech integration in infants predict auditory speech comprehension in the second year of life

The use of visual cues during the processing of audiovisual (AV) speech is known to be less efficient in children and adults with language difficulties and difficulties are known to be more prevalent in children from low-income populations. In the present study, we followed an economically diverse group of thirty-seven infants longitudinally from 6–9 months to 14–16 months of age. We used eye-tracking to examine whether individual differences in visual attention during AV processing of speech in 6–9 month old infants, particularly when processing congruent and incongruent auditory and visual speech cues, might be indicative of their later language development. Twenty-two of these 6–9 month old infants also participated in an event-related potential (ERP) AV task within the same experimental session. Language development was then followed-up at the age of 14–16 months, using two measures of language development, the Preschool Language Scale and the Oxford Communicative Development Inventory. The results show that those infants who were less efficient in auditory speech processing at the age of 6–9 months had lower receptive language scores at 14–16 months. A correlational analysis revealed that the pattern of face scanning and ERP responses to audiovisually incongruent stimuli at 6–9 months were both significantly associated with language development at 14–16 months. These findings add to the understanding of individual differences in neural signatures of AV processing and associated looking behavior in infants.

Dynamique cérébrale et réserve cognitive en situation d’attention sélective dans le vieillissement normal et les troubles de la cognition : approche neurofonctionnelle

Réalisée en cotutelle avec l'Unité de Formation à la Recherche Lettres Arts et Sciences Humaines - Université Nice-Sophia Antipolis.

Les effets de la distance physique sur les processus attentionnels sont dépendants de la similarité distracteur-cible : étude à partir des potentiels reliés aux évènements

L’attention visuelle est un processus cognitif qui priorise le traitement de l’information visuelle d’une région particulière du champ visuel. En électroencéphalographie, la méthode des potentiels reliés aux évènements permet l’extraction de composantes associées à divers processus cognitifs. La N2pc, une composante latéralisée caractérisée par une déflexion négative entre 180 et 300 ms post-stimulus du côté controlatéral à l’hémichamp dans lequel l’attention est déployée, reflète les processus impliqués dans le déploiement de l’attention visuo-spatiale. De nombreuses études antérieures ont soulevé plusieurs facteurs pouvant moduler cette composante, provenant d’autant de processus de bas niveau que de processus de haut niveau. Cette présente étude comporte une série d’expériences qui approfondit les connaissances sur le rôle de l’attention sur le traitement et la représentation des items dans les champs récepteurs des aires extrastriées du cortex visuel. Ces études démontrent ainsi que l’attention peut effectivement éliminer l’influence d’un distracteur dissimilaire à la cible lorsque celui-ci se retrouve dans le même champ visuel que l’item auquel l’attention est attribuée. Cependant, lorsque l’item est similaire à la cible, son influence ne peut être éliminée. De plus, cette présente étude identifie le rôle des filtres précoces et tardifs de haut niveau sur la sélection attentionnelle.

Lien entre l'activité pariéto-occipitale enregistrée pendant une tâche de mémoire à court terme visuelle et les habiletés mathématiques : une étude en magnétoencéphalographie

La mémoire à court terme visuelle (MCTv) est un système qui permet le maintien temporaire de l’information visuelle en mémoire. La capacité en mémoire à court terme se définit par le nombre d’items qu’un individu peut maintenir en mémoire sur une courte période de temps et est limitée à environ quatre items. Il a été démontré que la capacité en MCTv et les habiletés mathématiques sont étroitement liées. La MCTv est utile dans beaucoup de composantes liées aux mathématiques, comme la résolution de problèmes, la visualisation mentale et l’arithmétique. En outre, la MCTv et le raisonnement mathématique font appel à des régions similaires du cerveau, notamment dans le cortex pariétal. Le sillon intrapariétal (SIP) semble être particulièrement important, autant dans la réalisation de tâches liées à la MCTv qu’aux habiletés mathématiques. Nous avons créé une tâche de MCTv que 15 participants adultes en santé ont réalisée pendant que nous enregistrions leur activité cérébrale à l’aide de la magnétoencéphalographie (MEG). Nous nous sommes intéressés principalement à la composante SPCM. Une évaluation neuropsychologique a également été administrée aux participants. Nous souhaitions tester l’hypothèse selon laquelle l’activité cérébrale aux capteurs pariéto-occipitaux pendant la tâche de MCTv en MEG sera liée à la performance en mathématiques. Les résultats indiquent que l’amplitude de l’activité pariéto-occipitale pendant la tâche de MCTv permet de prédire les habiletés mathématiques ainsi que la performance dans une tâche de raisonnement perceptif. Ces résultats permettent de confirmer le lien existant entre les habiletés mathématiques et le fonctionnement sous-jacent à la MCTv.

Étude électrophysiologique de la suppression interoculaire : implication pour l'amblyopie

L’amblyopie est un trouble développemental de la vision binoculaire. Elle est typiquement caractérisée par des atteintes de l’acuité visuelle et de la stéréoscopie. Toutefois, de plus en plus d’études indiquent la présence d’atteintes plus étendues telles que les difficultés d’attention visuelle ou de lecture. L’amblyopie est généralement expliquée par une suppression interoculaire au niveau cortical, considérée comme chronique ou permanente à l’extérieur de la période développementale. Or, un nombre croissant d’études suggèrent que des interactions binoculaires normales seraient présentes chez les amblyopes adultes. Dans une première étude, nous avons tenté d’identifier un marqueur électrophysiologique de la vision binoculaire. Nous avons enregistré des potentiels évoqués visuels chez des observateurs normaux à qui l’on a induit une dysfonction binoculaire. Les interactions binoculaires étaient caractérisées à l’aide de patrons (facilitation, moyennage et suppression) en comparant les réponses monoculaires et binoculaires. De plus, ces interactions étaient quantifiées à partir d’index d’intégration continus en soustrayant la somme des réponses monoculaires de la réponse binoculaire. Les résultats indiquaient que les patrons d’interaction n’étaient pas optimaux pour estimer les performances stéréoscopiques. Ces dernières étaient, en revanche, mieux expliquées par notre index d’intégration binoculaire. Ainsi, cette étude suggère que l’électrophysiologie est un bon prédicteur de la vision binoculaire. Dans une deuxième étude, nous avons examiné les corrélats neuronaux et comportementaux de la suppression interoculaire chez des amblyopes adultes et des observateurs normaux. Des potentiels évoqués visuels stationnaires ont été enregistrés en utilisant un paradigme de suppression par flash. La suppression était modulée par un changement de contraste du stimulus flash (10, 20, 30, ou 100%), ou le suppresseur, qui était présenté soit dans l’œil dominant ou non-dominant (ou amblyope). Sur le plan comportemental, la suppression interoculaire était observée indépendamment de l’œil stimulé par le flash chez les contrôles. Au contraire, chez les amblyopes, la suppression était asymétrique (c’est-à-dire supérieure lorsqu’elle provenait de l’œil dominant), ce qui suggérait une suppression chronique. De manière intéressante, l’œil amblyope a supprimé l’œil dominant à haut niveau de contraste. Sur le plan électrophysiologique, l’effet de suppression interoculaire observé à la région occipitale était équivalent dans chaque groupe. Toutefois, les réponses électrophysiologiques à la région frontale chez les amblyopes n’étaient pas modulées comme celles des contrôles; la suppression de l’œil amblyope était manifeste même à bas contraste. Nous résultats supportent ainsi l’existence d’interaction binoculaire fonctionnelle chez les amblyopes adultes ainsi que l’implication d’un réseau cortical étendu dans la suppression interoculaire. En somme, l’amblyopie est une condition complexe dont les atteintes corticales et les déficits fonctionnels semblent globaux. L’amblyopie ne doit plus être considérée comme limitée à une dysfonction de l’aire visuelle primaire. La suppression interoculaire semble un point central de cette problématique, mais encore beaucoup d’études seront nécessaires afin de déterminer l’ensemble des mécanismes impliqués dans celle-ci.