A survey of human motion analysis using depth imagery

Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, but recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions.

Dissociation between the impact of evidence on eye movement target choice and confidence judgements

It has been suggested that the evidence used to support a decision to move our eyes and the confidence we have in that decision are derived from a common source. Alternatively, confidence may be based on further post-decisional processes. In three experiments we examined this. In Experiment 1, participants chose between two targets on the basis of varying levels of evidence (i.e., the direction of motion coherence in a Random-Dot-Kinematogram). They indicated this choice by making a saccade to one of two targets and then indicated their confidence. Saccade trajectory deviation was taken as a measure of the inhibition of the non-selected target. We found that as evidence increased so did confidence and deviations of saccade trajectory away from the non-selected target. However, a correlational analysis suggested they were not related. In Experiment 2 an option to opt-out of the choice was offered on some trials if choice proved too difficult. In this way we isolated trials on which confidence in target selection was high (i.e., when the option to opt-out was available but not taken). Again saccade trajectory deviations were found not to differ in relation to confidence. In Experiment 3 we directly manipulated confidence, such that participants had high or low task confidence. They showed no differences in saccade trajectory deviations. These results support post-decisional accounts of confidence: evidence supporting the decision to move the eyes is reflected in saccade control, but the confidence that we have in that choice is subject to further post-decisional processes.

Visual sensor fusion for active security in robotic industrial environments

This work presents a method of information fusion involving data captured by both a standard CCD camera and a ToF camera to be used in the detection of the proximity between a manipulator robot and a human. Both cameras are assumed to be located above the work area of an industrial robot. The fusion of colour images and time of light information makes it possible to know the 3D localization of objects with respect to a world coordinate system. At the same time this allows to know their colour information. Considering that ToF information given by the range camera contains innacuracies including distance error, border error, and pixel saturation, some corrections over the ToF information are proposed and developed to improve the results. The proposed fusion method uses the calibration parameters of both cameras to reproject 3D ToF points, expressed in a common coordinate system for both cameras and a robot arm, in 2D colour images. In addition to this, using the 3D information, the motion detection in a robot industrial environment is achieved, and the fusion of information is applied to the foreground objects previously detected. This combination of information results in a matrix that links colour and 3D information, giving the possibility of characterising the object by its colour in addition to its 3D localization. Further development of these methods will make it possible to identify objects and their position in the real world, and to use this information to prevent possible collisions between the robot and such objects.

Visual sensor fusion for active security in robotic industrial environments

This work presents a method of information fusion involving data captured by both a standard charge-coupled device (CCD) camera and a time-of-flight (ToF) camera to be used in the detection of the proximity between a manipulator robot and a human. Both cameras are assumed to be located above the work area of an industrial robot. The fusion of colour images and time-of-flight information makes it possible to know the 3D localization of objects with respect to a world coordinate system. At the same time, this allows to know their colour information. Considering that ToF information given by the range camera contains innacuracies including distance error, border error, and pixel saturation, some corrections over the ToF information are proposed and developed to improve the results. The proposed fusion method uses the calibration parameters of both cameras to reproject 3D ToF points, expressed in a common coordinate system for both cameras and a robot arm, in 2D colour images. In addition to this, using the 3D information, the motion detection in a robot industrial environment is achieved, and the fusion of information is applied to the foreground objects previously detected. This combination of information results in a matrix that links colour and 3D information, giving the possibility of characterising the object by its colour in addition to its 3D localisation. Further development of these methods will make it possible to identify objects and their position in the real world and to use this information to prevent possible collisions between the robot and such objects.

Subsecond changes of global brain state in illusory multistable motion perception

This study explored transient changes in EEG microstates and spatial Omega complexity associated with changes in multistable perception. 21-channel EEG was recorded from 13 healthy subjects viewing an alternating dot pattern that induced illusory motion with ambiguous direction. Baseline epochs with stable motion direction were compared to epochs immediately preceding stimuli that were perceived with changed motion direction ('reference stimuli'). About 750 ms before reference stimuli, Omega complexity decreased as compared to baseline, and two of four classes of EEG microstates changed their probability of occurrence. About 300 ms before reference stimuli, Omega complexity increased and the previous deviations of EEG microstates were reversed. Given earlier results on Omega complexity and microstates, these sub-second EEG changes might parallel longer-lasting fluctuations in vigilance. Assumedly, the discontinuities of illusory motion thus occur during sub-second dips in arousal, and the following reconstruction of the illusion coincides with a state of relative over-arousal.

Bitmap Movement Detection: HDR for Dynamic Scenes

Exposure Fusion and other HDR techniques generate well-exposed images from a bracketed image sequence while reproducing a large dynamic range that far exceeds the dynamic range of a single exposure. Common to all these techniques is the problem that the smallest movements in the captured images generate artefacts (ghosting) that dramatically affect the quality of the final images. This limits the use of HDR and Exposure Fusion techniques because common scenes of interest are usually dynamic. We present a method that adapts Exposure Fusion, as well as standard HDR techniques, to allow for dynamic scene without introducing artefacts. Our method detects clusters of moving pixels within a bracketed exposure sequence with simple binary operations. We show that the proposed technique is able to deal with a large amount of movement in the scene and different movement configurations. The result is a ghost-free and highly detailed exposure fused image at a low computational cost.

Robust Proximal Femur Segmentation in Conventional X-Ray Images via Random Forest Regression on Multi-resolution Gradient Features

In this paper, we propose a fully automatic, robust approach for segmenting proximal femur in conventional X-ray images. Our method is based on hierarchical landmark detection by random forest regression, where the detection results of 22 global landmarks are used to do the spatial normalization, and the detection results of the 59 local landmarks serve as the image cue for instantiation of a statistical shape model of the proximal femur. To detect landmarks in both levels, we use multi-resolution HoG (Histogram of Oriented Gradients) as features which can achieve better accuracy and robustness. The efficacy of the present method is demonstrated by experiments conducted on 150 clinical x-ray images. It was found that the present method could achieve an average point-to-curve error of 2.0 mm and that the present method was robust to low image contrast, noise and occlusions caused by implants.

Fully automatic segmentation of AP pelvis X-rays via random forest regression with efficient feature selection and hierarchical sparse shape composition

In clinical practice, traditional X-ray radiography is widely used, and knowledge of landmarks and contours in anteroposterior (AP) pelvis X-rays is invaluable for computer aided diagnosis, hip surgery planning and image-guided interventions. This paper presents a fully automatic approach for landmark detection and shape segmentation of both pelvis and femur in conventional AP X-ray images. Our approach is based on the framework of landmark detection via Random Forest (RF) regression and shape regularization via hierarchical sparse shape composition. We propose a visual feature FL-HoG (Flexible- Level Histogram of Oriented Gradients) and a feature selection algorithm based on trace radio optimization to improve the robustness and the efficacy of RF-based landmark detection. The landmark detection result is then used in a hierarchical sparse shape composition framework for shape regularization. Finally, the extracted shape contour is fine-tuned by a post-processing step based on low level image features. The experimental results demonstrate that our feature selection algorithm reduces the feature dimension in a factor of 40 and improves both training and test efficiency. Further experiments conducted on 436 clinical AP pelvis X-rays show that our approach achieves an average point-to-curve error around 1.2 mm for femur and 1.9 mm for pelvis.

Blind multiuser receiver for DS-CDMA wireless system

We consider blind signal detection in an asynchronous code-division multiple-access (CDMA) system employing short spreading sequences in the presence of unknown multipath fading. This approach is capable of countering the presence of multiple-access interference (MAI) in CDMA fading channels. The proposed blind multiuser detector is based on an independent component analysis (ICA) to mitigate both MAI and noise. This algorithm has been utilised in blind source separation (BSS) of unknown sources from their mixtures. It can also be used for estimating the basis vectors of BSS. The aim is to include an ICA algorithm within a wireless receiver in order to reduce the level of interference in wideband systems. This blind multiuser detector requires no training sequence compared with the conventional multiuser detection receiver. The proposed ICA blind multiuser detector is made robust with respect to knowledge of signature waveforms and the timing of the user of interest. Several experiments are performed in order to verify the validity of the proposed ICA algorithm.

Neuromagnetic evoked responses to complex motions are greatest for expansion

We analysed evoked magnetic responses to moving random dot stimuli, initially using a 19-channel magnetoencephalography (MEG) system, and subsequently using a 151-channel MEG system. Random dot displays were used to construct complex motion sequences, which we refer to as expansion, contraction, deformation, and rotation. We also investigated lateral translation and a condition in which the directions of the dots were randomised. In all stimulus conditions, the dots were first stationary, then traveled for a brief period (317s or 542 ms), and were then stationary again. In all conditions, evoked magnetic responses were observed with a widespread bilateral distribution over the observers' heads. Initial recordings revealed a substantially larger evoked magnetic response to the expansion condition than the other conditions. In a revised study, we used a 151-channel MEG system and two stimulus diameters (9.3 and 48 deg), the smaller comparable with the first experiment. The responses were analysed using a nonparametric approach and confirmed our initial observations. In a third study, speed gradients were removed and a new design permitted direct comparisons between motion conditions. The results from all three experiments are consistent with the greater ecological validity of the expansion stimulus. © 2004 Elsevier B.V. All rights reserved.

Independent detectors for expansion and rotation, and for orthogonal components of deformation

It is well known that optic flow - the smooth transformation of the retinal image experienced by a moving observer - contains valuable information about the three-dimensional layout of the environment. From psychophysical and neurophysiological experiments, specialised mechanisms responsive to components of optic flow (sometimes called complex motion) such as expansion and rotation have been inferred. However, it remains unclear (a) whether the visual system has mechanisms for processing the component of deformation and (b) whether there are multiple mechanisms that function independently from each other. Here, we investigate these issues using random-dot patterns and a forced-choice subthreshold summation technique. In experiment 1, we manipulated the size of a test region that was permitted to contain signal and found substantial spatial summation for signal components of translation, expansion, rotation, and deformation embedded in noise. In experiment 2, little or no summation was found for the superposition of orthogonal pairs of complex motion patterns (eg expansion and rotation), consistent with probability summation between pairs of independent detectors. Our results suggest that optic-flow components are detected by mechanisms that are specialised for particular patterns of complex motion.

Investigations into visual hyperacuity

Visual hyperacuities.are a group of thresholds whose values surpass that expected by the anatomical and optical constraints of the eye. There are many variables which affect hyperacuities of which this thesis considers the following .. 1. The effect of contrast on displacement detection and bisection acuity. It is proposed that spatial summation may account for the different response of these two hyperacuities compared with the contrast response of vernier acuity. 2. The effect of references on displacement detection. These were shown to greatly enhance performance when present. Their effect was, however, dependent upon the temporal characteristics of the displacement. 3. The effect of spatial frequency on vernier acuity. Evidence from this experiment suggests that vernier performance can be explained on the basis of the output of orientationally selective spatial frequency filters. 4. Evidence for a weighting function for visual location using random dot clusters. The weighting attached to different parts of the retinal light distribution was found to alter non-linearly with increasing offset from the geometric center of the cluster. A relationship between dot density and peak amplitude of the weighting function was found. 5. Spatial scaling of vernier acuity in the peripheral field. With careful choice of a technique which did not allow separation and eccentricity to co-vary it was found possible to scale vernier acuity both for two lines and two separated dots. 6. The effect of increasing age on hyperacuity. No change in vernier acuity with age was found which contrasted with displacement detection and bisection acuity both of which showed a significant decline with increasing age.

Wind turbine rotor blade monitoring using digital image correlation: a comparison to aeroelastic simulations of a multi-megawatt wind turbine

Optical full-field measurement methods such as Digital Image Correlation (DIC) provide a new opportunity for measuring deformations and vibrations with high spatial and temporal resolution. However, application to full-scale wind turbines is not trivial. Elaborate preparation of the experiment is vital and sophisticated post processing of the DIC results essential. In the present study, a rotor blade of a 3.2 MW wind turbine is equipped with a random black-and-white dot pattern at four different radial positions. Two cameras are located in front of the wind turbine and the response of the rotor blade is monitored using DIC for different turbine operations. In addition, a Light Detection and Ranging (LiDAR) system is used in order to measure the wind conditions. Wind fields are created based on the LiDAR measurements and used to perform aeroelastic simulations of the wind turbine by means of advanced multibody codes. The results from the optical DIC system appear plausible when checked against common and expected results. In addition, the comparison of relative out-of-plane blade deflections shows good agreement between DIC results and aeroelastic simulations.

FREQUENCY DOMAIN CHARACTERIZATION OF OPTIC FLOW AND VISION-BASED OCELLAR SENSING FOR ROTATIONAL MOTION

The structure of an animal’s eye is determined by the tasks it must perform. While vertebrates rely on their two eyes for all visual functions, insects have evolved a wide range of specialized visual organs to support behaviors such as prey capture, predator evasion, mate pursuit, flight stabilization, and navigation. Compound eyes and ocelli constitute the vision forming and sensing mechanisms of some flying insects. They provide signals useful for flight stabilization and navigation. In contrast to the well-studied compound eye, the ocelli, seen as the second visual system, sense fast luminance changes and allows for fast visual processing. Using a luminance-based sensor that mimics the insect ocelli and a camera-based motion detection system, a frequency-domain characterization of an ocellar sensor and optic flow (due to rotational motion) are analyzed. Inspired by the insect neurons that make use of signals from both vision sensing mechanisms, advantages, disadvantages and complementary properties of ocellar and optic flow estimates are discussed.

Eye for an eye

Artist David Lyons and computer scientist David Flatla work collaboratively to create art that intentionally targets audiences of varying visual abilities mediated through smart device interfaces. Conceived as an investigation into theories and practices of visual perception, they explore the idea that artwork can be intentionally created to be experienced differently dependent on one’s visual abilities. They have created motion graphics and supporting recolouring and colour vision deficiency (CVD) simulation software. Some of the motion graphics communicate details specifically to those with colour blindness/CVD by containing moving imagery only seen by those with CVD. Others will contain moving images that those with typical colour vision can experience but appear to be unchanging to people with CVD. All the artwork is revealed for both audiences through the use of specially programmed smart devices, fitted with augmented reality recolouring and CVD simulation software. The visual elements come from various sources, including the Ishihara Colour Blind Test, movie marques, and game shows. The software created reflects the perceptual capabilities of most individuals with reduced colour vision. The development of the simulation software and the motion graphic series are examined and discussed from both computer science and artistic positions.