A survey of human motion analysis using depth imagery

Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, but recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions.

Motion planning of autonomous vehicles in a non-autonomous vehicle environment without speed lanes

Planning is one of the key problems for autonomous vehicles operating in road scenarios. Present planning algorithms operate with the assumption that traffic is organised in predefined speed lanes, which makes it impossible to allow autonomous vehicles in countries with unorganised traffic. Unorganised traffic is though capable of higher traffic bandwidths when constituting vehicles vary in their speed capabilities and sizes. Diverse vehicles in an unorganised exhibit unique driving behaviours which are analysed in this paper by a simulation study. The aim of the work reported here is to create a planning algorithm for mixed traffic consisting of both autonomous and non-autonomous vehicles without any inter-vehicle communication. The awareness (e.g. vision) of every vehicle is restricted to nearby vehicles only and a straight infinite road is assumed for decision making regarding navigation in the presence of multiple vehicles. Exhibited behaviours include obstacle avoidance, overtaking, giving way for vehicles to overtake from behind, vehicle following, adjusting the lateral lane position and so on. A conflict of plans is a major issue which will almost certainly arise in the absence of inter-vehicle communication. Hence each vehicle needs to continuously track other vehicles and rectify plans whenever a collision seems likely. Further it is observed here that driver aggression plays a vital role in overall traffic dynamics, hence this has also been factored in accordingly. This work is hence a step forward towards achieving autonomous vehicles in unorganised traffic, while similar effort would be required for planning problems such as intersections, mergers, diversions and other modules like localisation.

Biologically-inspired robust motion segmentation using mutual information

This paper presents a neuroscience inspired information theoretic approach to motion segmentation. Robust motion segmentation represents a fundamental first stage in many surveillance tasks. As an alternative to widely adopted individual segmentation approaches, which are challenged in different ways by imagery exhibiting a wide range of environmental variation and irrelevant motion, this paper presents a new biologically-inspired approach which computes the multivariate mutual information between multiple complementary motion segmentation outputs. Performance evaluation across a range of datasets and against competing segmentation methods demonstrates robust performance.

Using high-fidelity virtual reality to study perception in freely moving observers

Technological innovations have had a profound influence on how we study the sensory perception in humans and other animals. One example was the introduction of affordable computers, which radically changed the nature of visual experiments. It is clear that vision research is now at cusp of a similar shift, this time driven by the use of commercially available, low-cost, high- fidelity virtual reality (VR). In this review we will focus on: (a) the research questions VR allows experimenters to address and why these research questions are important, (b) the things that need to be considered when using VR to study human perception, (c) the drawbacks of current VR systems, and (d) the future direction vision research may take, now that VR has become a viable research tool.

Disparity-defined objects moving in depth do not elicit three-dimensional shape constancy

Observers generally fail to recover three-dimensional shape accurately from binocular disparity. Typically, depth is overestimated at near distances and underestimated at far distances [Johnston, E. B. (1991). Systematic distortions of shape from stereopsis. Vision Research, 31, 1351–1360]. A simple prediction from this is that disparity-defined objects should appear to expand in depth when moving towards the observer, and compress in depth when moving away. However, additional information is provided when an object moves from which 3D Euclidean shape can be recovered, be this through the addition of structure from motion information [Richards, W. (1985). Structure from stereo and motion. Journal of the Optical Society of America A, 2, 343–349], or the use of non-generic strategies [Todd, J. T., & Norman, J. F. (2003). The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure? Perception and Psychophysics, 65, 31–47]. Here, we investigated shape constancy for objects moving in depth. We found that to be perceived as constant in shape, objects needed to contract in depth when moving toward the observer, and expand in depth when moving away, countering the effects of incorrect distance scaling (Johnston, 1991). This is a striking example of the failure of shape con- stancy, but one that is predicted if observers neither accurately estimate object distance in order to recover Euclidean shape, nor are able to base their responses on a simpler processing strategy.

A Computer Vision Framework For Finger-Tapping Evaluation In Parkinson's Disease

Objective: To define and evaluate a Computer-Vision (CV) method for scoring Paced Finger-Tapping (PFT) in Parkinson's disease (PD) using quantitative motion analysis of index-fingers and to compare the obtained scores to the UPDRS (Unified Parkinson's Disease Rating Scale) finger-taps (FT). Background: The naked-eye evaluation of PFT in clinical practice results in coarse resolution to determine PD status. Besides, sensor mechanisms for PFT evaluation may cause patients discomfort. In order to avoid cost and effort of applying wearable sensors, a CV system for non-invasive PFT evaluation is introduced. Methods: A database of 221 PFT videos from 6 PD patients was processed. The subjects were instructed to position their hands above their shoulders besides the face and tap the index-finger against the thumb consistently with speed. They were facing towards a pivoted camera during recording. The videos were rated by two clinicians between symptom levels 0-to-3 using UPDRS-FT. The CV method incorporates a motion analyzer and a face detector. The method detects the face of testee in each video-frame. The frame is split into two images from face-rectangle center. Two regions of interest are located in each image to detect index-finger motion of left and right hands respectively. The tracking of opening and closing phases of dominant hand index-finger produces a tapping time-series. This time-series is normalized by the face height. The normalization calibrates the amplitude in tapping signal which is affected by the varying distance between camera and subject (farther the camera, lesser the amplitude). A total of 15 features were classified using K-nearest neighbor (KNN) classifier to characterize the symptoms levels in UPDRS-FT. The target ratings provided by the raters were averaged. Results: A 10-fold cross validation in KNN classified 221 videos between 3 symptom levels with 75% accuracy. An area under the receiver operating characteristic curves of 82.6% supports feasibility of the obtained features to replicate clinical assessments. Conclusions: The system is able to track index-finger motion to estimate tapping symptoms in PD. It has certain advantages compared to other technologies (e.g. magnetic sensors, accelerometers etc.) for PFT evaluation to improve and automate the ratings

Motion Cues Analysis for Parkinson Gait Recognition

Background: Previous assessment methods for PG recognition used sensor mechanisms for PG that may cause discomfort. In order to avoid stress of applying wearable sensors, computer vision (CV) based diagnostic systems for PG recognition have been proposed. Main constraints in these methods are the laboratory setup procedures: Novel colored dresses for the patients were specifically designed to segment the test body from a specific colored background. Objective: To develop an image processing tool for home-assessment of Parkinson Gait(PG) by analyzing motion cues extracted during the gait cycles. Methods: The system is based on the idea that a normal body attains equilibrium during the gait by aligning the body posture with the axis of gravity. Due to the rigidity in muscular tone, persons with PD fail to align their bodies with the axis of gravity. The leaned posture of PD patients appears to fall forward. Whereas a normal posture exhibits a constant erect posture throughout the gait. Patients with PD walk with shortened stride angle (less than 15 degrees on average) with high variability in the stride frequency. Whereas a normal gait exhibits a constant stride frequency with an average stride angle of 45 degrees. In order to analyze PG, levodopa-responsive patients and normal controls were videotaped with several gait cycles. First, the test body is segmented in each frame of the gait video based on the pixel contrast from the background to form a silhouette. Next, the center of gravity of this silhouette is calculated. This silhouette is further skeletonized from the video frames to extract the motion cues. Two motion cues were stride frequency based on the cyclic leg motion and the lean frequency based on the angle between the leaned torso tangent and the axis of gravity. The differences in the peaks in stride and lean frequencies between PG and normal gait are calculated using Cosine Similarity measurements. Results: High cosine dissimilarity was observed in the stride and lean frequencies between PG and normal gait. High variations are found in the stride intervals of PG whereas constant stride intervals are found in the normal gait. Conclusions: We propose an algorithm as a source to eliminate laboratory constraints and discomfort during PG analysis. Installing this tool in a home computer with a webcam allows assessment of gait in the home environment.

Motion cue analysis for parkinsonian gait recognition

This paper presents a computer-vision based marker-free method for gait-impairment detection in Patients with Parkinson's disease (PWP). The system is based upon the idea that a normal human body attains equilibrium during the gait by aligning the body posture with Axis-of-Gravity (AOG) using feet as the base of support. In contrast, PWP appear to be falling forward as they are less-able to align their body with AOG due to rigid muscular tone. A normal gait exhibits periodic stride-cycles with stride-angle around 45o between the legs, whereas PWP walk with shortened stride-angle with high variability between the stride-cycles. In order to analyze Parkinsonian-gait (PG), subjects were videotaped with several gait-cycles. The subject's body was segmented using a color-segmentation method to form a silhouette. The silhouette was skeletonized for motion cues extraction. The motion cues analyzed were stride-cycles (based on the cyclic leg motion of skeleton) and posture lean (based on the angle between leaned torso of skeleton and AOG). Cosine similarity between an imaginary perfect gait pattern and the subject gait patterns produced 100% recognition rate of PG for 4 normal-controls and 3 PWP. Results suggested that the method is a promising tool to be used for PG assessment in home-environment.

The use of software Mapavox for teaching graphics and games to blind and low vision students: a development through practices

Current policies on education to visually impaired point for a growing trend of including students with special educational needs in regular schools. However, most often this inclusion is not accompanied by an appropriate professional trained or infrastructure, which has been presented as a big problem for regular school teachers who have students with visual impairments in their classroom. Based on this situation, the Group of Extension in Tactile Cartography from UNESP - University of the State of São Paulo - Campus de Rio Claro - SP - Brazil has been developing educational material of geography and cartography to blind students at a special school. Among the materials developed in this study highlight the development of graphics and board games provided with sound capabilities through MAPAVOX, software developed in partnership with UFRJ - Federal University from Rio de Janeiro - RJ - Brazil. Through this software, sound capabilities can be inserted into built materials, giving them a multi-sensory character. In most cases the necessary conditions for building specific materials to students with visual impairments is expensive and beyond the reach of features from a regular school, so the survey sought to use easy access and low cost materials like Cork, leaf aluminum, material for fixing and others. The development of these materials was supported by preparation in laboratory and its subsequent test through practices involving blind students. The methodology used on the survey is based on qualitative research and non comparative analysis of the results. In other words, the material is built based on the special students perception and reality construction, not being mere adaptations of visual materials, but a construction focused on the reality of the visually impaired. The results proved were quite successful as the materials prepared were effective on mediating the learning process of students with disabilities. Geographical and cartographic concepts were seized by the students through the technology used, associated with the use of materials that took into account in its building process the perception of the students.

Odometria visual baseada em técnicas de structure from motion

Visual Odometry is the process that estimates camera position and orientation based solely on images and in features (projections of visual landmarks present in the scene) extraced from them. With the increasing advance of Computer Vision algorithms and computer processing power, the subarea known as Structure from Motion (SFM) started to supply mathematical tools composing localization systems for robotics and Augmented Reality applications, in contrast with its initial purpose of being used in inherently offline solutions aiming 3D reconstruction and image based modelling. In that way, this work proposes a pipeline to obtain relative position featuring a previously calibrated camera as positional sensor and based entirely on models and algorithms from SFM. Techniques usually applied in camera localization systems such as Kalman filters and particle filters are not used, making unnecessary additional information like probabilistic models for camera state transition. Experiments assessing both 3D reconstruction quality and camera position estimated by the system were performed, in which image sequences captured in reallistic scenarios were processed and compared to localization data gathered from a mobile robotic platform

Perception-action and adaptation in postural control of children and adolescents with cerebral palsy

The aim of this study was to examine the coupling between visual information and body sway and the adaptation in this coupling of individuals with cerebral palsy (CP). Fifteen children with and 15 without CP. 6-15 years old, were required to stand upright inside of a moving room. All children first performed two trials with no movement of the room and eyes open or closed, then four trials in which the room oscillated at 0.2 or 0.5 Hz (peak velocity of 0.6 cm/s), one trial in which the room oscillated at 0.2 Hz (peak velocity of 3.5 cm/s), and finally two other trials in which the room oscillated again at 0.2 Hz (peak velocity of 0.6 cm/s). Participants with CP coupled body sway to visual information provided by the moving room, comparable to the coupling of participants without CP. However, participants with CP exhibited larger body sway in maintaining upright position and more variable sway when body sway was induced by visual manipulation. They showed adaptive sensory motor coupling, e.g. down-weighting visual influence when a larger stimulus was provided, but not with the same magnitude as typically developing participants. This indicates that participants with CP have less capability of adaptation. (C) 2011 Published by Elsevier Ltd.

Using the Hard, Randy, and Rittler Test to Evaluate Color Vision in Capuchins (Cebus libidinosus)

The identification of color vision types in primates is fundamental to understanding the evolution and biological function of color perception. The Hard, Randy, and Rittler (HRR) pseudoisochromatic test categorizes human color vision types successfully. Here we provide an experimental setup to employ HRR in a nonhuman primate, the capuchin (Cebus libidinosus), a platyrrhine with polymorphic color vision. The HRR test consists of plates with a matrix composed of gray circles that vary in size and brightness. Differently colored circles form a geometric shape (X, O, or Delta) that is discriminated visually from the gray background pattern. The ability to identify these shapes determines the type of dyschromatopsy (deficiency in color vision). We tested six capuchins in their own cages under natural sunlight. The subjects chose between two HRR plates in each trial: one with the gray pattern only and the other with a colored shape, presented on the left or right side at random. We presented the test 40 times and calculated the 95 % confidence limits for chance performance based on the binomial test. We also genotyped all subjects for exons 3 and 5 of the X-linked opsin genes. The HRR test diagnosed two subjects as protan dichromats (missing or defective L-cone), three as deutan dichromats (missing or defective M-cone), and one female as trichromat. Genetic analysis supported the behavioral data for all subjects. These findings show that the HRR test can be applied to diagnose color vision in nonhuman primates.

A variational approach to camera motion smoothing

[EN] In this paper we study a variational problem derived from a computer vision application: video camera calibration with smoothing constraint. By video camera calibration we meanto estimate the location, orientation and lens zoom-setting of the camera for each video frame taking into account image visible features. To simplify the problem we assume that the camera is mounted on a tripod, in such case, for each frame captured at time t , the calibration is provided by 3 parameters : (1) P(t) (PAN) which represents the tripod vertical axis rotation, (2) T(t) (TILT) which represents the tripod horizontal axis rotation and (3) Z(t) (CAMERA ZOOM) the camera lens zoom setting. The calibration function t -> u(t) = (P(t),T(t),Z(t)) is obtained as the minima of an energy function I[u] . In thIs paper we study the existence of minima of such energy function as well as the solutions of the associated Euler-Lagrange equations.

Verhaltensphysiologische Analyse der visuellen Wahrnehmung räumlicher Tiefe beim Goldfisch

Die Frage, wie es zur visuellen Wahrnehmung räumlicher Tiefe kommt, wenn das Retinabild nur zweidimensional ist, gehört zu den grundlegenden Proble-men der Hirnforschung. Für Tiere, die sich aktiv in ihrer Umgebung bewegen, herrscht ein großer Selektionsdruck Entfernungen und Größen richtig einzu-schätzen. Ziel der vorliegenden Arbeit war es, herauszufinden, ob und wie gut Goldfische Objekte allein aufgrund des Abstandes unterscheiden können und woraus sie Information über den Abstand gewinnen. Hierzu wurde ein Ver-suchsaufbau mit homogen weißem Hintergrund entworfen, in dem die Akkom-modation als Entfernungsinformationen verwendet werden kann, weniger je-doch die Bewegungsparallaxe. Die Goldfische lernten durch operante Konditio-nierung einen Stimulus (schwarze Kreisscheibe) in einem bestimmten Abstand zu wählen, während ein anderer, gleichgroßer Stimulus so entfernt wie möglich präsentiert wurde. Der Abstand zwischen den Stimuli wurde dann verringert, bis die Goldfische keine sichere Wahl für den Dressurstimulus mehr treffen konnten. Die Unterscheidungsleistung der Goldfische wurde mit zunehmendem Abstand des Dressurstimulus immer geringer. Eine Wiederholung der Versuche mit unscharfen Stimu¬lus¬kon¬turen brachte keine Verschlechterung in der Unter-scheidung, was Akkommodation wenig wahrscheinlich macht. Um die Größen-konstanz beim Goldfisch zu testen, wurden die Durchmesser der unterschiedlich entfernten Stimuli so angepasst, dass sie für den Goldfisch die gleiche Retina-bildgröße hatten. Unter diesen Bedingungen waren die Goldfische nicht in der Lage verschieden entfernte Stimuli zu unterscheiden und somit Größenkonstanz zu leisten. Es fand demnach keine echte Entfernungsbestimmung oder Tiefen-wahrneh¬mung statt. Die Unterscheidung der verschieden entfernten Stimuli erfolgte allein durch deren Abbildungsgröße auf der Retina. Dass die Goldfische bei diesem Experiment nicht akkommodieren, wurde durch Infrarot-Photoretinoskopie gezeigt. Somit lässt sich Akkommodation für die Entfer-nungsbestimmung in diesen Versuchen ausschließen. Für diese Leistung und die Größenkonstanz ist vermutlich die Bewegungsparallaxe entscheidend.

The posterior parietal cortex: a bridge between vision and action

The present work takes into account three posterior parietal areas, V6, V6A, and PEc, all operating on different subsets of signals (visual, somatic, motor). The work focuses on the study of their functional properties, to better understand their respective contribution in the neuronal circuits that make possible the interactions between subject and external environment. In the caudalmost pole of parietal lobe there is area V6. Functional data suggest that this area is related to the encoding of both objects motion and ego-motion. However, the sensitivity of V6 neurons to optic flow stimulations has been tested only in human fMRI experiments. Here we addressed this issue by applying on monkey the same experimental protocol used in human studies. The visual stimulation obtained with the Flow Fields stimulus was the most effective and powerful to activate area V6 in monkey, further strengthening this homology between the two primates. The neighboring areas, V6A and PEc, show different cytoarchitecture and connectivity profiles, but are both involved in the control of reaches. We studied the sensory responses present in these areas, and directly compared these.. We also studied the motor related discharges of PEc neurons during reaching movements in 3D space comparing also the direction and depth tuning of PEc cells with those of V6A. The results show that area PEc and V6A share several functional properties. Area PEc, unlike V6A, contains a richer and more complex somatosensory input, and a poorer, although complex visual one. Differences emerged also comparing the motor-related properties for reaches in depth: the incidence of depth modulations in PEc and the temporal pattern of modulation for depth and direction allow to delineate a trend among the two parietal visuomotor areas.