Control and Guidance of Low-Cost Robots via Gesture Perception for Monitoring Activities in the Home

This paper describes the development of a low-cost mini-robot that is controlled by visual gestures. The prototype allows a person with disabilities to perform visual inspections indoors and in domestic spaces. Such a device could be used as the operator's eyes obviating the need for him to move about. The robot is equipped with a motorised webcam that is also controlled by visual gestures. This camera is used to monitor tasks in the home using the mini-robot while the operator remains quiet and motionless. The prototype was evaluated through several experiments testing the ability to use the mini-robot’s kinematics and communication systems to make it follow certain paths. The mini-robot can be programmed with specific orders and can be tele-operated by means of 3D hand gestures to enable the operator to perform movements and monitor tasks from a distance.

Simulation Environment for Multi-Robot Cooperative 3D Target Perception

4th International Conference, SIMPAR 2014, Bergamo, Italy, October 20-23, 2014

Cine en 3D

Este proyecto pretende documentar el proceso completo de una producción audiovisual en 3D. Para ello, se hace un recorrido por cada una de sus etapas, desde los primeros pasos de la escritura hasta que la película llega a los espectadores. Como el flujo de trabajo para realizar una película es enorme, el estudio se centra exclusivamente en las peculiaridades estereoscópicas, dejando al margen los procesos habituales de una producción convencional. El proyecto está dividido en cinco grandes bloques, centrados en cada etapa de producción. El primer capítulo es una introducción al mundo tridimensional, que sienta las bases de la percepción humana para entender cómo funcionan todas las demás etapas. Además, se hace un repaso por los distintos avances producidos en el campo estereoscópico a lo largo de la historia. El segundo capítulo se centra en la pre-producción, el primer paso para llevar a cabo cualquier proyecto audiovisual, consistente en la planificación del trabajo a realizar y la organización de los distintos elementos que serán necesarios durante el rodaje, teniendo en cuenta desde el primer momento la tercera dimensión. El tercer capítulo está dedicado a la grabación de las imágenes, centrándose principalmente en las características y configuraciones de las cámaras o rigs con los que se obtienen las dos secuencias. El cuarto capítulo aborda la percepción de la película 3D, explicando las características de los variados sistemas de visualización de las imágenes grabadas, que dan a su vez diferentes sistemas de proyección estereoscópica. El quinto y último capítulo se centra en el procesado digital de las imágenes estéreo, que permite juntar el material grabado, corregirlo o perfeccionarlo, y darle la forma adecuada de película que llega al público final. La documentación escrita en español sobre todas estas fases es algo escasa, centrándose normalmente en alguna parte concreta de la cadena, por lo que el proyecto trata también de llenar ese vacío, explicándolo de manera sencilla para hacerlo accesible y factible para toda persona interesada. ABSTRACT. This project has the intention of document the complete process of 3D audiovisual production. For that reason, we make a journey through each of its stages, from the first steps of writing until the film reaches the final viewers. Since the workflow for a film is huge, the study focuses exclusively on the stereoscopic peculiarities, leaving aside the usual processes of conventional production. The Project is divided into five major sections focused on each stage of production. The first chapter is an introduction to three-dimensional world, which lays the foundation of human perception to understand how the other stages work. In addition, we review the various advances in the stereoscopic field throughout history. The second chapter focuses on the pre-production, the first step in carrying out any audiovisual project, including the design of all the works to do and the organization of the different elements that will be needed during filming, taking into account the third dimension from the first moment. The third chapter is devoted to the image recording, focusing mainly on the features and the settings of the cameras or rigs used to obtain the two sequences. The fourth chapter deals with the 3D film perception, explaining the characteristics of the various systems used for displaying the recorded images, which, in turn, give different stereoscopic projection systems. The fifth and final chapter focuses on the digital processing of stereo images, which allows collecting all the recorded material, correcting or improving it, and giving it the proper style for a film that reaches the end consumer. The documents written in Spanish about all these phases are somewhat sparse, usually focusing on a particular part of the chain, so the project also aims to fill that gap, with simple explanations in order to make it accessible and doable for anyone interested.

Does countershading contribute to visual camouflage by concealing 3D shape?

This is an abstract of an invited talk presented at AVA Animal Vision Meeting / Camocon 2015, which took place in Liverpool, 23 August 2015.

Chorégraphier le son

La version intégrale de ce mémoire est disponible uniquement pour consultation individuelle à la bibliothèque de musique de l'Université de Montréal (www.bib.umontreal.ca/MU).

Perception-guided evaluation of 3D medical visualizations

Magdeburg, Univ., Fak. für Informatik, Diss., 2015

L’impact de la stéréoscopie dans la reconnaissance, la perception et la constance de forme 3D

Les buts des recherches présentées dans cette thèse étaient d’évaluer le rôle de la stéréoscopie dans la reconnaissance de forme, dans la perception du relief et dans la constance de forme. La première étude a examiné le rôle de la stéréoscopie dans la perception des formes visuelles en utilisant une tâche de reconnaissance de formes. Les stimuli pouvaient être présentés en 2D, avec disparité normale (3D) ou avec disparité inversée. La performance de reconnaissance était meilleure avec les modes de présentation 2D et 3D qu’avec la 3D inversée. Cela indique que la stéréoscopie contribue à la reconnaissance de forme. La deuxième étude s’est intéressée à la contribution conjointe de l’ombrage et de la stéréoscopie dans la perception du relief des formes. Les stimuli étaient des images d’une forme 3D convexe synthétique présentée sous un point de vue menant à une ambigüité quant à sa convexité. L’illumination pouvait provenir du haut ou du bas et de la gauche ou de la droite, et les stimuli étaient présentés dichoptiquement avec soit de la disparité binoculaire normale, de la disparité inversée ou sans disparité entre les vues. Les participants ont répondu que les formes étaient convexes plus souvent lorsque la lumière provenait du haut que du bas, plus souvent avec la disparité normale qu’en 2D, et plus souvent avec absence de disparité qu’avec disparité inversée. Les effets de direction d’illumination et du mode de présentation étaient additifs, c’est-à-dire qu’ils n’interagissaient pas. Cela indique que l’ombrage et la stéréoscopie contribuent indépendamment à la perception du relief des formes. La troisième étude a évalué la contribution de la stéréoscopie à la constance de forme, et son interaction avec l’expertise perceptuelle. Elle a utilisé trois tâches de discrimination séquentielle de trombones tordus ayant subi des rotations en profondeur. Les stimuli pouvaient être présentés sans stéréoscopie, avec stéréoscopie normale ou avec stéréoscopie inversée. Dans la première moitié de l’Exp. 1, dans laquelle les variations du mode de présentation étaient intra-sujets, les performances étaient meilleures en 3D qu’en 2D et qu’en 3D inversée. Ces effets ont été renversés dans la seconde moitié de l’expérience, et les coûts de rotation sont devenus plus faibles pour la 2D et la 3D inversée que pour la 3D. Dans les Exps. 2 (variations intra-sujets du mode de présentation, avec un changement de stimuli au milieu de l’expérience) et 3 (variations inter-sujets du mode de présentation), les effets de rotation étaient en tout temps plus faibles avec stéréoscopie qu’avec stéréoscopie inversée et qu’en 2D, et plus faibles avec stéréoscopie inversée que sans stéréoscopie. Ces résultats indiquent que la stéréoscopie contribue à la constance de forme. Toutefois, cela demande qu’elle soit valide avec un niveau minimal de consistance, sinon elle devient stratégiquement ignorée. En bref, les trois études présentées dans cette thèse ont permis de montrer que la stéréoscopie contribue à la reconnaissance de forme, à la perception du relief et à la constance de forme. De plus, l’ombrage et la stéréoscopie sont intégrés linéairement.

A new 3D modelling paradigm for discrete model

Until few years ago, 3D modelling was a topic confined into a professional environment. Nowadays technological innovations, the 3D printer among all, have attracted novice users to this application field. This sudden breakthrough was not supported by adequate software solutions. The 3D editing tools currently available do not assist the non-expert user during the various stages of generation, interaction and manipulation of 3D virtual models. This is mainly due to the current paradigm that is largely supported by two-dimensional input/output devices and strongly affected by obvious geometrical constraints. We have identified three main phases that characterize the creation and management of 3D virtual models. We investigated these directions evaluating and simplifying the classic editing techniques in order to propose more natural and intuitive tools in a pure 3D modelling environment. In particular, we focused on freehand sketch-based modelling to create 3D virtual models, interaction and navigation in a 3D modelling environment and advanced editing tools for free-form deformation and objects composition. To pursuing these goals we wondered how new gesture-based interaction technologies can be successfully employed in a 3D modelling environments, how we could improve the depth perception and the interaction in 3D environments and which operations could be developed to simplify the classical virtual models editing paradigm. Our main aims were to propose a set of solutions with which a common user can realize an idea in a 3D virtual model, drawing in the air just as he would on paper. Moreover, we tried to use gestures and mid-air movements to explore and interact in 3D virtual environment, and we studied simple and effective 3D form transformations. The work was carried out adopting the discrete representation of the models, thanks to its intuitiveness, but especially because it is full of open challenges.

Visual attention and perception models for assessing quality in 2D and 3D stereoscopic video

La medida de calidad de vídeo sigue siendo necesaria para definir los criterios que caracterizan una señal que cumpla los requisitos de visionado impuestos por el usuario. Las nuevas tecnologías, como el vídeo 3D estereoscópico o formatos más allá de la alta definición, imponen nuevos criterios que deben ser analizadas para obtener la mayor satisfacción posible del usuario. Entre los problemas detectados durante el desarrollo de esta tesis doctoral se han determinado fenómenos que afectan a distintas fases de la cadena de producción audiovisual y tipo de contenido variado. En primer lugar, el proceso de generación de contenidos debe encontrarse controlado mediante parámetros que eviten que se produzca el disconfort visual y, consecuentemente, fatiga visual, especialmente en lo relativo a contenidos de 3D estereoscópico, tanto de animación como de acción real. Por otro lado, la medida de calidad relativa a la fase de compresión de vídeo emplea métricas que en ocasiones no se encuentran adaptadas a la percepción del usuario. El empleo de modelos psicovisuales y diagramas de atención visual permitirían ponderar las áreas de la imagen de manera que se preste mayor importancia a los píxeles que el usuario enfocará con mayor probabilidad. Estos dos bloques se relacionan a través de la definición del término saliencia. Saliencia es la capacidad del sistema visual para caracterizar una imagen visualizada ponderando las áreas que más atractivas resultan al ojo humano. La saliencia en generación de contenidos estereoscópicos se refiere principalmente a la profundidad simulada mediante la ilusión óptica, medida en términos de distancia del objeto virtual al ojo humano. Sin embargo, en vídeo bidimensional, la saliencia no se basa en la profundidad, sino en otros elementos adicionales, como el movimiento, el nivel de detalle, la posición de los píxeles o la aparición de caras, que serán los factores básicos que compondrán el modelo de atención visual desarrollado. Con el objetivo de detectar las características de una secuencia de vídeo estereoscópico que, con mayor probabilidad, pueden generar disconfort visual, se consultó la extensa literatura relativa a este tema y se realizaron unas pruebas subjetivas preliminares con usuarios. De esta forma, se llegó a la conclusión de que se producía disconfort en los casos en que se producía un cambio abrupto en la distribución de profundidades simuladas de la imagen, aparte de otras degradaciones como la denominada “violación de ventana”. A través de nuevas pruebas subjetivas centradas en analizar estos efectos con diferentes distribuciones de profundidades, se trataron de concretar los parámetros que definían esta imagen. Los resultados de las pruebas demuestran que los cambios abruptos en imágenes se producen en entornos con movimientos y disparidades negativas elevadas que producen interferencias en los procesos de acomodación y vergencia del ojo humano, así como una necesidad en el aumento de los tiempos de enfoque del cristalino. En la mejora de las métricas de calidad a través de modelos que se adaptan al sistema visual humano, se realizaron también pruebas subjetivas que ayudaron a determinar la importancia de cada uno de los factores a la hora de enmascarar una determinada degradación. Los resultados demuestran una ligera mejora en los resultados obtenidos al aplicar máscaras de ponderación y atención visual, los cuales aproximan los parámetros de calidad objetiva a la respuesta del ojo humano. ABSTRACT Video quality assessment is still a necessary tool for defining the criteria to characterize a signal with the viewing requirements imposed by the final user. New technologies, such as 3D stereoscopic video and formats of HD and beyond HD oblige to develop new analysis of video features for obtaining the highest user’s satisfaction. Among the problems detected during the process of this doctoral thesis, it has been determined that some phenomena affect to different phases in the audiovisual production chain, apart from the type of content. On first instance, the generation of contents process should be enough controlled through parameters that avoid the occurrence of visual discomfort in observer’s eye, and consequently, visual fatigue. It is especially necessary controlling sequences of stereoscopic 3D, with both animation and live-action contents. On the other hand, video quality assessment, related to compression processes, should be improved because some objective metrics are adapted to user’s perception. The use of psychovisual models and visual attention diagrams allow the weighting of image regions of interest, giving more importance to the areas which the user will focus most probably. These two work fields are related together through the definition of the term saliency. Saliency is the capacity of human visual system for characterizing an image, highlighting the areas which result more attractive to the human eye. Saliency in generation of 3DTV contents refers mainly to the simulated depth of the optic illusion, i.e. the distance from the virtual object to the human eye. On the other hand, saliency is not based on virtual depth, but on other features, such as motion, level of detail, position of pixels in the frame or face detection, which are the basic features that are part of the developed visual attention model, as demonstrated with tests. Extensive literature involving visual comfort assessment was looked up, and the development of new preliminary subjective assessment with users was performed, in order to detect the features that increase the probability of discomfort to occur. With this methodology, the conclusions drawn confirmed that one common source of visual discomfort was when an abrupt change of disparity happened in video transitions, apart from other degradations, such as window violation. New quality assessment was performed to quantify the distribution of disparities over different sequences. The results confirmed that abrupt changes in negative parallax environment produce accommodation-vergence mismatches derived from the increasing time for human crystalline to focus the virtual objects. On the other side, for developing metrics that adapt to human visual system, additional subjective tests were developed to determine the importance of each factor, which masks a concrete distortion. Results demonstrated slight improvement after applying visual attention to objective metrics. This process of weighing pixels approximates the quality results to human eye’s response.

Visual perception for the 3D recognition of geometric pieces in robotic manipulation

During grasping and intelligent robotic manipulation tasks, the camera position relative to the scene changes dramatically because the robot is moving to adapt its path and correctly grasp objects. This is because the camera is mounted at the robot effector. For this reason, in this type of environment, a visual recognition system must be implemented to recognize and “automatically and autonomously” obtain the positions of objects in the scene. Furthermore, in industrial environments, all objects that are manipulated by robots are made of the same material and cannot be differentiated by features such as texture or color. In this work, first, a study and analysis of 3D recognition descriptors has been completed for application in these environments. Second, a visual recognition system designed from specific distributed client-server architecture has been proposed to be applied in the recognition process of industrial objects without these appearance features. Our system has been implemented to overcome problems of recognition when the objects can only be recognized by geometric shape and the simplicity of shapes could create ambiguity. Finally, some real tests are performed and illustrated to verify the satisfactory performance of the proposed system.

Changes in neural complexity during the perception of 3D images using random dot stereograms

One developing theme in consciousness research is that consciousness is not the product of any specific component of the brain, rather it is an emergent property of the changing patterns of connectivity between different specialised functional components. For example, the dynamic core hypothesis proposes that conscious experience requires high levels of neural complexity, where complexity is defined in terms of functional connectivity. To test this hypothesis, electroencephalography was recorded while participants were shown random dot-stereograms. Consistent with the dynamic core hypothesis, neural complexity increased as the participants changed from simply viewing the stereogram to consciously perceiving the hidden 3D image.

3D considerations for motion perception and visuomotor transformations

Moving through a stable, three-dimensional world is a hallmark of our motor and perceptual experience. This stability is constantly being challenged by movements of the eyes and head, inducing retinal blur and retino-spatial misalignments for which the brain must compensate. To do so, the brain must account for eye and head kinematics to transform two-dimensional retinal input into the reference frame necessary for movement or perception. The four studies in this thesis used both computational and psychophysical approaches to investigate several aspects of this reference frame transformation. In the first study, we examined the neural mechanism underlying the visuomotor transformation for smooth pursuit using a feedforward neural network model. After training, the model performed the general, three-dimensional transformation using gain modulation. This gave mechanistic significance to gain modulation observed in cortical pursuit areas while also providing several testable hypotheses for future electrophysiological work. In the second study, we asked how anticipatory pursuit, which is driven by memorized signals, accounts for eye and head geometry using a novel head-roll updating paradigm. We showed that the velocity memory driving anticipatory smooth pursuit relies on retinal signals, but is updated for the current head orientation. In the third study, we asked how forcing retinal motion to undergo a reference frame transformation influences perceptual decision making. We found that simply rolling one's head impairs perceptual decision making in a way captured by stochastic reference frame transformations. In the final study, we asked how torsional shifts of the retinal projection occurring with almost every eye movement influence orientation perception across saccades. We found a pre-saccadic, predictive remapping consistent with maintaining a purely retinal (but spatially inaccurate) orientation perception throughout the movement. Together these studies suggest that, despite their spatial inaccuracy, retinal signals play a surprisingly large role in our seamless visual experience. This work therefore represents a significant advance in our understanding of how the brain performs one of its most fundamental functions.

Gesture based persuasive interfaces for public ambient displays

Dissertação de Mestrado em Engenharia Informática 2º Semestre, 2011/2012

Accuracy estimation of a new omnidirectional 3D vision sensor

We present a computer vision system that associates omnidirectional vision with structured light with the aim of obtaining depth information for a 360 degrees field of view. The approach proposed in this article combines an omnidirectional camera with a panoramic laser projector. The article shows how the sensor is modelled and its accuracy is proved by means of experimental results. The proposed sensor provides useful information for robot navigation applications, pipe inspection, 3D scene modelling etc

Implementation of a robust coded structured light technique for dynamic 3D measurements

This paper presents the implementation details of a coded structured light system for rapid shape acquisition of unknown surfaces. Such techniques are based on the projection of patterns onto a measuring surface and grabbing images of every projection with a camera. Analyzing the pattern deformations that appear in the images, 3D information of the surface can be calculated. The implemented technique projects a unique pattern so that it can be used to measure moving surfaces. The structure of the pattern is a grid where the color of the slits are selected using a De Bruijn sequence. Moreover, since both axis of the pattern are coded, the cross points of the grid have two codewords (which permits to reconstruct them very precisely), while pixels belonging to horizontal and vertical slits have also a codeword. Different sets of colors are used for horizontal and vertical slits, so the resulting pattern is invariant to rotation. Therefore, the alignment constraint between camera and projector considered by a lot of authors is not necessary