Faithful representation of similarities among three-dimensional shapes in human vision.

Efficient and reliable classification of visual stimuli requires that their representations reside a low-dimensional and, therefore, computationally manageable feature space. We investigated the ability of the human visual system to derive such representations from the sensory input-a highly nontrivial task, given the million or so dimensions of the visual signal at its entry point to the cortex. In a series of experiments, subjects were presented with sets of parametrically defined shapes; the points in the common high-dimensional parameter space corresponding to the individual shapes formed regular planar (two-dimensional) patterns such as a triangle, a square, etc. We then used multidimensional scaling to arrange the shapes in planar configurations, dictated by their experimentally determined perceived similarities. The resulting configurations closely resembled the original arrangements of the stimuli in the parameter space. This achievement of the human visual system was replicated by a computational model derived from a theory of object representation in the brain, according to which similarities between objects, and not the geometry of each object, need to be faithfully represented.

Selective attention to stimulus location modulates the steady-state visual evoked potential.

Steady-state visual evoked potentials (SSVEPs) were recorded from the scalp of human subjects who were cued to attend to a rapid sequence of alphanumeric characters presented to one visual half-field while ignoring a concurrent sequence of characters in the opposite half-field. These two-character sequences were each superimposed upon a small square background that was flickered at a rate of 8.6 Hz in one half-field and 12 Hz in the other half-field. The amplitude of the frequency-coded SSVEP elicited by either of the task-irrelevant flickering backgrounds was significantly enlarged when attention was focused upon the character sequence at the same location. This amplitude enhancement with attention was most prominent over occipital-temporal scalp areas of the right cerebral hemisphere regardless of the visual field of stimulation. These findings indicate that the SSVEP reflects an enhancement of neural responses to all stimuli that fall within the "spotlight" of spatial attention, whether or not the stimuli are task-relevant. Recordings of the SSVEP provide a new approach for studying the neural mechanisms and functional properties of selective attention to multi-element visual displays.

Activity of primate inferotemporal neurons related to a sought target in pair-association task.

Visual long-term memory in primates has been assessed by using the pair-association (PA) task, in which a subject retrieves and chooses the paired associate of a cue picture. Our previous studies on single neurons in the anterior inferotemporal (AIT) cortex suggested their roles in representing paired associates in the mind. To test the possibility that the delay activity of AIT neurons is related to a particular picture as a sought target, we devised the PA with color switch (PACS) task. In the PACS task, the necessity for memory retrieval and its initiation time were controlled by a color switch in the middle of the delay period. A control task, in which there is no color switch, corresponds to the conventional delayed matching-to-sample (DMS) task where the monkey chooses the same picture as a cue. We found that AIT neurons started to respond just after the color switch in the PACS task, when the cue-optimal picture's associate was presented as a cue. In contrast, they showed no response change in the DMS task. We confirmed that this effect is not due to the visual response to colors. Furthermore, when the cue-optimal picture was presented as a cue, these neurons showed suppression after the color switch in the PACS task. These results suggest that the activity of AIT neurons mediates gating mechanisms that preferentially pass information about a sought target, even when the sought target is retrieved from long-term memory.

Identification of a Drosophila G protein alpha subunit (dGq alpha-3) expressed in chemosensory cells and central neurons.

We have identified another Drosophila GTP-binding protein (G protein) alpha subunit, dGq alpha-3. Transcripts encoding dGq alpha-3 are derived from alternative splicing of the dGq alpha locus previously shown to encode two visual-system-specific transcripts [Lee, Y.-J., Dobbs, M.B., Verardi, M.L. & Hyde, D.R. (1990) Neuron 5, 889-898]. Immunolocalization studies using dGq alpha-3 isoform-specific antibodies and LacZ fusion genes show that dGq alpha-3 is expressed in chemosensory cells of the olfactory and taste structures, including a subset of olfactory and gustatory neurons, and in cells of the central nervous system, including neurons in the lamina ganglionaris. These data are consistent with a variety of roles for dGq alpha-3, including mediating a subset of olfactory and gustatory responses in Drosophila, and supports the idea that some chemosensory responses use G protein-coupled receptors and the second messenger inositol 1,4,5-trisphosphate.

Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex.

The stages of integration leading from local feature analysis to object recognition were explored in human visual cortex by using the technique of functional magnetic resonance imaging. Here we report evidence for object-related activation. Such activation was located at the lateral-posterior aspect of the occipital lobe, just abutting the posterior aspect of the motion-sensitive area MT/V5, in a region termed the lateral occipital complex (LO). LO showed preferential activation to images of objects, compared to a wide range of texture patterns. This activation was not caused by a global difference in the Fourier spatial frequency content of objects versus texture images, since object images produced enhanced LO activation compared to textures matched in power spectra but randomized in phase. The preferential activation to objects also could not be explained by different patterns of eye movements: similar levels of activation were observed when subjects fixated on the objects and when they scanned the objects with their eyes. Additional manipulations such as spatial frequency filtering and a 4-fold change in visual size did not affect LO activation. These results suggest that the enhanced responses to objects were not a manifestation of low-level visual processing. A striking demonstration that activity in LO is uniquely correlated to object detectability was produced by the "Lincoln" illusion, in which blurring of objects digitized into large blocks paradoxically increases their recognizability. Such blurring led to significant enhancement of LO activation. Despite the preferential activation to objects, LO did not seem to be involved in the final, "semantic," stages of the recognition process. Thus, objects varying widely in their recognizability (e.g., famous faces, common objects, and unfamiliar three-dimensional abstract sculptures) activated it to a similar degree. These results are thus evidence for an intermediate link in the chain of processing stages leading to object recognition in human visual cortex.

The deferred imitation task as a nonverbal measure of declarative memory.

We tested amnesic patients, patients with frontal lobe lesions, and control subjects with the deferred imitation task, a nonverbal test used to demonstrate memory abilities in human infants. On day 1, subjects were given sets of objects to obtain a baseline measure of their spontaneous performance of target actions. Then different event sequences were modeled with the object sets. On day 2, the objects were given to the subjects again, first without any instructions to imitate the sequences, and then with explicit instructions to imitate the actions exactly as they had been modeled. Control subjects and frontal lobe patients reproduced the events under both uninstructed and instructed conditions. In contrast, performance by the amnesic patients did not significantly differ from that of a second control group who had the same opportunities to handle the objects but were not shown the modeled actions. These findings suggest that deferred imitation is dependent on the brain structures essential for declarative memory that are damaged in amnesia, and they support the view that infants who imitate actions after long delays have an early capacity for long-term declarative memory.

Parameters affecting conscious versus unconscious visual discrimination with damage to the visual cortex (V1).

When the visual (striate) cortex (V1) is damaged in human subjects, cortical blindness results in the contralateral visual half field. Nevertheless, under some experimental conditions, subjects demonstrate a capacity to make visual discriminations in the blind hemifield (blindsight), even though they have no phenomenal experience of seeing. This capacity must, therefore, be mediated by parallel projections to other brain areas. It is also the case that some subjects have conscious residual vision in response to fast moving stimuli or sudden changes in light flux level presented to the blind hemifield, characterized by a contentless kind of awareness, a feeling of something happening, albeit not normal seeing. The relationship between these two modes of discrimination has never been studied systematically. We examine, in the same experiment, both the unconscious discrimination and the conscious visual awareness of moving stimuli in a subject with unilateral damage to V1. The results demonstrate an excellent capacity to discriminate motion direction and orientation in the absence of acknowledged perceptual awareness. Discrimination of the stimulus parameters for acknowledged awareness apparently follows a different functional relationship with respect to stimulus speed, displacement, and stimulus contrast. As performance in the two modes can be quantitatively matched, the findings suggest that it should be possible to image brain activity and to identify the active areas involved in the same subject performing the same discrimination task, both with and without conscious awareness, and hence to determine whether any structures contribute uniquely to conscious perception.

Visual motion perception.

The primate visual motion system performs numerous functions essential for survival in a dynamic visual world. Prominent among these functions is the ability to recover and represent the trajectories of objects in a form that facilitates behavioral responses to those movements. The first step toward this goal, which consists of detecting the displacement of retinal image features, has been studied for many years in both psychophysical and neurobiological experiments. Evidence indicates that achievement of this step is computationally straightforward and occurs at the earliest cortical stage. The second step involves the selective integration of retinal motion signals according to the object of origin. Realization of this step is computationally demanding, as the solution is formally underconstrained. It must rely--by definition--upon utilization of retinal cues that are indicative of the spatial relationships within and between objects in the visual scene. Psychophysical experiments have documented this dependence and suggested mechanisms by which it may be achieved. Neurophysiological experiments have provided evidence for a neural substrate that may underlie this selective motion signal integration. Together they paint a coherent portrait of the means by which retinal image motion gives rise to our perceptual experience of moving objects.

Role of the monkey ventrolateral prefrontal cortex in the integration of contextual information for the selection and understanding of goal-directed actions

Action selection and organization are very complex processes that need to exploit contextual information and the retrieval of previously memorized information, as well as the integration of these different types of data. On the basis of anatomical connection with premotor and parietal areas involved in action goal coding, and on the data about the literature it seems appropriate to suppose that one of the most candidate involved in the selection of neuronal pools for the selection and organization of intentional actions is the prefrontal cortex. We recorded single ventrolateral prefrontal (VLPF) neurons activity while monkeys performed simple and complex manipulative actions aimed at distinct final goals, by employing a modified and more strictly controlled version of the grasp-to-eat(a food pellet)/grasp-to-place(an object) paradigm used in previous studies on parietal (Fogassi et al., 2005) and premotor neurons (Bonini et al., 2010). With this task we have been able both to evaluate the processing and integration of distinct (visual and auditory) contextual sequentially presented information in order to select the forthcoming action to perform and to examine the possible presence of goal-related activity in this portion of cortex. Moreover, we performed an observation task to clarify the possible contribution of VLPF neurons to the understanding of others’ goal-directed actions. Simple Visuo Motor Task (sVMT). We found four main types of neurons: unimodal sensory-driven, motor-related, unimodal sensory-and-motor, and multisensory neurons. We found a substantial number of VLPF neurons showing both a motor-related discharge and a visual presentation response (sensory-and-motor neurons), with remarkable visuo-motor congruence for the preferred target. Interestingly the discharge of multisensory neurons reflected a behavioural decision independently from the sensory modality of the stimulus allowing the monkey to make it: some encoded a decision to act/refraining from acting (the majority), while others specified one among the four behavioural alternatives. Complex Visuo Motor Task (cVMT). The cVMT was similar to the sVMT, but included a further grasping motor act (grasping a lid in order to remove it, before grasping the target) and was run in two modalities: randomized and in blocks. Substantially, motor-related and sensory-and-motor neurons tested in the cVMTrandomized were activated already during the first grasping motor act, but the selectivity for one of the two graspable targets emerged only during the execution of the second grasping. In contrast, when the cVMT was run in block, almost all these neurons not only discharged during the first grasping motor act, but also displayed the same target selectivity showed in correspondence of the hand contact with the target. Observation Task (OT). A great part of the neurons active during the OT showed a firing rate modulation in correspondence with the action performed by the experimenter. Among them, we found neurons significantly activated during the observation of the experimenter’s action (action observation-related neurons) and neurons responding not only to the action observation, but also to the presented cue stimuli (sensory-and-action observation-related neurons. Among the neurons of the first set, almost the half displayed a target selectivity, with a not clear difference between the two presented targets; Concerning to the second neuronal set, sensory-and-action related neurons, we found a low target selectivity and a not strictly congruence between the selectivity exhibited in the visual response and in the action observation.

3D visual sensing of human hand for remote operation of a robotic hand

New low cost sensors and open free libraries for 3D image processing are making important advances in robot vision applications possible, such as three-dimensional object recognition, semantic mapping, navigation and localization of robots, human detection and/or gesture recognition for human-machine interaction. In this paper, a novel method for recognizing and tracking the fingers of a human hand is presented. This method is based on point clouds from range images captured by a RGBD sensor. It works in real time and it does not require visual marks, camera calibration or previous knowledge of the environment. Moreover, it works successfully even when multiple objects appear in the scene or when the ambient light is changed. Furthermore, this method was designed to develop a human interface to control domestic or industrial devices, remotely. In this paper, the method was tested by operating a robotic hand. Firstly, the human hand was recognized and the fingers were detected. Secondly, the movement of the fingers was analysed and mapped to be imitated by a robotic hand.

Control framework for dexterous manipulation using dynamic visual servoing and tactile sensors’ feedback

Tactile sensors play an important role in robotics manipulation to perform dexterous and complex tasks. This paper presents a novel control framework to perform dexterous manipulation with multi-fingered robotic hands using feedback data from tactile and visual sensors. This control framework permits the definition of new visual controllers which allow the path tracking of the object motion taking into account both the dynamics model of the robot hand and the grasping force of the fingertips under a hybrid control scheme. In addition, the proposed general method employs optimal control to obtain the desired behaviour in the joint space of the fingers based on an indicated cost function which determines how the control effort is distributed over the joints of the robotic hand. Finally, authors show experimental verifications on a real robotic manipulation system for some of the controllers derived from the control framework.

Dynamic visual servo control of a 4-axis joint tool to track image trajectories during machining complex shapes

A large part of the new generation of computer numerical control systems has adopted an architecture based on robotic systems. This architecture improves the implementation of many manufacturing processes in terms of flexibility, efficiency, accuracy and velocity. This paper presents a 4-axis robot tool based on a joint structure whose primary use is to perform complex machining shapes in some non-contact processes. A new dynamic visual controller is proposed in order to control the 4-axis joint structure, where image information is used in the control loop to guide the robot tool in the machining task. In addition, this controller eliminates the chaotic joint behavior which appears during tracking of the quasi-repetitive trajectories required in machining processes. Moreover, this robot tool can be coupled to a manipulator robot in order to form a multi-robot platform for complex manufacturing tasks. Therefore, the robot tool could perform a machining task using a piece grasped from the workspace by a manipulator robot. This manipulator robot could be guided by using visual information given by the robot tool, thereby obtaining an intelligent multi-robot platform controlled by only one camera.

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.

Towards a real-time 3D object recognition pipeline on mobile GPGPU computing platforms using low-cost RGB-D sensors

In this project, we propose the implementation of a 3D object recognition system which will be optimized to operate under demanding time constraints. The system must be robust so that objects can be recognized properly in poor light conditions and cluttered scenes with significant levels of occlusion. An important requirement must be met: the system must exhibit a reasonable performance running on a low power consumption mobile GPU computing platform (NVIDIA Jetson TK1) so that it can be integrated in mobile robotics systems, ambient intelligence or ambient assisted living applications. The acquisition system is based on the use of color and depth (RGB-D) data streams provided by low-cost 3D sensors like Microsoft Kinect or PrimeSense Carmine. The range of algorithms and applications to be implemented and integrated will be quite broad, ranging from the acquisition, outlier removal or filtering of the input data and the segmentation or characterization of regions of interest in the scene to the very object recognition and pose estimation. Furthermore, in order to validate the proposed system, we will create a 3D object dataset. It will be composed by a set of 3D models, reconstructed from common household objects, as well as a handful of test scenes in which those objects appear. The scenes will be characterized by different levels of occlusion, diverse distances from the elements to the sensor and variations on the pose of the target objects. The creation of this dataset implies the additional development of 3D data acquisition and 3D object reconstruction applications. The resulting system has many possible applications, ranging from mobile robot navigation and semantic scene labeling to human-computer interaction (HCI) systems based on visual information.

Visual perception for the 3D recognition of geometric pieces in robotic manipulation

During grasping and intelligent robotic manipulation tasks, the camera position relative to the scene changes dramatically because the robot is moving to adapt its path and correctly grasp objects. This is because the camera is mounted at the robot effector. For this reason, in this type of environment, a visual recognition system must be implemented to recognize and “automatically and autonomously” obtain the positions of objects in the scene. Furthermore, in industrial environments, all objects that are manipulated by robots are made of the same material and cannot be differentiated by features such as texture or color. In this work, first, a study and analysis of 3D recognition descriptors has been completed for application in these environments. Second, a visual recognition system designed from specific distributed client-server architecture has been proposed to be applied in the recognition process of industrial objects without these appearance features. Our system has been implemented to overcome problems of recognition when the objects can only be recognized by geometric shape and the simplicity of shapes could create ambiguity. Finally, some real tests are performed and illustrated to verify the satisfactory performance of the proposed system.