Low cost multimodal facial recognition via kinect sensors

This paper presents a novel low cost 3D facial recognition using gaming sensors such as Kinect™. The paper describes the hardware, calibration and infrared noise and pattern interference challenges of integrating multiple Kinect sensors. The preliminary results show a promising trend for low cost solutions that can be populated in crowded facilities such as malls and airports.

Kinect crowd interaction

Most of the state-of-the-art commercial simulation software mainly focuses on providing realistic animations and convincing artificial intelligence to avatars in the scenario. However, works on how to trigger the events and avatar reactions in the scenario in a natural and intuitive way are less noticed and developed. Typical events are usually triggered by predefined timestamps. Once the events are set, there is no easy way to interactively generate new events while the scene is running and therefore difficult to dynamically affect the avatar reactions. Based on this situation, we propose a framework to use human gesture as input to trigger events within a DI-Guy simulation scenario in real-time, which could greatly help users to control events and avatar reactions in the scenario. By implementing such a framework, we will be able to identify user’s intentions interactively and ensure that the avatars make corresponding reactions.

Integrating kinect and haptics for interactive STEM education in local and distributed environments

Skill shortage is a realistic social problem that Australia is currently facing, especially in the fields of Science, Technology, Engineering and Mathematics (STEM). Various approaches have been proposed to soften this issue. By now the most successful approach is to attract pre-university youth and university freshmen into those fields before they make a decision on future subjects by introducing them with interactive, modifiable and inspiring virtual environments, which incorporates most essential knowledge of STEM. We propose to design a comprehensive virtual reality platform with immersive interactions, pluggable components and flexible configurations. It also involves haptics, motion capture and gesture recognition, and could be deployed in both local and distributed environments. The platform utilizes off the shelf low cost haptics and motion capture products, however the fidelity can be maintained at a good level. The proposed platform has been implemented with different configurations and has been tested on a group of users. Preliminary test results show that the interactivity, flexibility and fidelity of the platform are highly appreciated by users. User surveys also indicate that the proposed platform could help pre-university students and university freshmen build an overview of various aspects of STEM education. Besides, users are also positive on the fact that the platform enabled them to identify the challenges for higher education in STEM by providing them opportunities to interactively modify system configurations and instantly experience the corresponding results both visually and haptically.

Multi-Kinect skeleton fusion for physical rehabilitation monitoring

 Kinect has been increasingly applied in rehabilitation as a motion capture device. However, the inherent limitations significantly hinder its further development in this important area. Although a number of Kinect fusion approaches have been proposed, only a few of them was actually considered for rehabilitation. In this paper, we propose to fuse information from multiple Kinects to achieve this. Given the specific scenario of users suffering from limited range of movements, we propose to calibrate depth cameras in multiple Kinects with 3D positions of joints on a human body rather than in a checkerboard pattern, so that patients are able to calibrate Kinects without extra support. Kalman filter is applied for skeleton-wise Kinect fusion since skeleton data (3D positions of joints) and its derivatives are preferred by physiotherapists to evaluate the exercise performance of patients. Various preliminary experiments were conducted to illustrate the accuracy of proposed calibration and fusion approach by comparing with a commercial Vicon system®, confirming the practical use of the system in rehabilitation exercise monitoring.

A general pose estimation algorithm in a multi-Kinect system

 Microsoft Kinect which has been primarily aimed at the computer gaming industry has been used in bio-kinematic research related implementations. A multi-Kinect system can be useful in exploiting spatial diversity to increase measurement accuracy. One of the main problems in deploying multi-Kinect systems is to estimate the pose, including the position and orientation of each Kinect. In this paper, a singular value decomposition (SVD) least-squares algorithm is extended to a more generic time-series based approach to solve this pose estimation problem utilising 3D positions of one or more joints in skeletons obtained from a multi-Kinect system. Additionally, computer simulations are performed to demonstrate the use and to evaluate the efficiency of the proposed algorithm. The former is further validated with a commercial Vicon system.

Safety applications using Kinect technology

 Microsoft Kinect sensor was introduced with the XBOX gaming console. It features a simple and portable motion capturing system. Kinect nowadays presents a point of interest in many fields of study and areas of research where its affordable price compared to its capabilities. The Kinect sensor has the capability to capture and track detected 3D objects with accuracy comparable to that captured by state of the art commercial systems. Human safety is considered one of the highest concerns, specially nowadays where the existence of machines and robots is widely used. In this paper we present using the Kinect technology for enhancing the safety of equipment and operations in seven different applications. These applications include 1) positioning of child’s car seat to optimise the child’s position in respected to front and side air-bags; 2) board positioning system to improve the teacher’s arm reach posture; 3) gas station safety to prevent children from accessing the gas pump; 4) indoor pool safety to avoid children access to deep pool area; 5) robot safety emergency stop; 6) Workplace safety; and 7) older adults fall prediction.

Pedestrian behaviour analysis using the microsoft kinect

An important challenge for brick-and-mortar retail businesses is how to monitor the interest of customers in displays and products in a store. We investigate the effectiveness of the Microsoft Kinect as a sensor to monitor the behaviour of pedestrians, which may reflect their interest in a store display or advertisement. In a controlled environment, participants acting as pedestrians are requested to show different levels of interest as they pass an object that is being monitored by a Kinect sensor. The sensor collected measurements such as the positions and orientation of the pedestrian's body and head, which were analysed to detect the movements of the pedestrians and thus infer the behaviour they exhibited. Our results demonstrate that when the Kinect is able to detect a behaviour that indicates a pedestrian's interest in an object, then the classification of the level of interest in terms of the type of behaviour is reasonably accurate under varying light conditions and numbers of pedestrians.

Kinect with ROS, interact with oculus: towards dynamic user interfaces for robotic teleoperation

Teleoperation remains an important aspect for robotic systems especially when deployed in unstructured environments. While a range of research strives for robots that are completely autonomous, many robotic applications still require some level of human-in-The-loop control. In any situation where teleoperation is required an effective User Interface (UI) remains a key component within the systems design. Current advancements in Virtual Reality (VR) software and hardware such as the Oculus Rift, HTC Vive and Google Cardboard combined with greater transparency to robotic systems afforded by middleware such as the Robot Operating System (ROS) provides an opportunity to rapidly improve traditional teleoperation interfaces. This paper uses a System of System (SoS) approach to present the concept of a Virtual Reality Dynamic User Interface (VRDUI) for the teleoperation of heterogeneous robots. Different geometric virtual workspaces are discussed and a cylindrical workspace aligned with interactive displays is presented as a virtual control room. A presentation mode within the proposed VRDUI is also detailed, this shows how point cloud information obtained from the Microsoft Kinect can be incorporated within the proposed virtual workspace. This point cloud data is successfully processed into an OctoMap utilizing the octree data structure to create a voxelized representation of the 3D scanned environment. The resulting OctoMap is then displayed to an operator as a 3D point cloud using the Oculus Rift Head Mounted Display (HMD).

Extracting 3D mesh skeletons using antipodal points locations

Finding the skeleton of a 3D mesh is an essential task for many applications such as mesh animation, tracking, and 3D registeration. In recent years, new technologies in computer vision such as Microsoft Kinect have proven that a mesh skeleton can be useful such as in the case of human machine interactions. To calculate the 3D mesh skeleton, the mesh properties such as topology and its components relations are utilized. In this paper, we propose the usage of a novel algorithm that can efficiently calculate a vertex antipodal point. A vertex antipodal point is the diametrically opposite point that belongs to the same mesh. The set of centers of the connecting lines between each vertex and its antipodal point represents the 3D mesh desired skeleton. Post processing is completed for smoothing and fitting centers into optimized skeleton parts. The algorithm is tested on different classes of 3D objects and produced efficient results that are comparable with the literature. The algorithm has the advantages of producing high quality skeletons as it preserves details. This is suitable for applications where the mesh skeleton mapping is required to be kept as much as possible.

Reproduction game demo

An 8 screen stereoscopic 3D Multi-user environment set in a circular formation. Paticipants are tracked by 8 kinect sensors as they move around the 360 degree installation. Participants inherit a species upon entering. Artificial Life based entities cluster towards the human particpants of their species and their songs evolve according to the interactions.

"Recognition" : Interactive motion tracking installation with neural network agent performer

Public installation running throughout the night at Cube 37 gallery. Morphing forms are projected onto the glass front of the gallery, facing the street. Human perticipants interact with the forms whose morphology changes with the movements of the participants. When no human participants are present, a neural network based agent that has learnt how to dance from a trained dancer, takes over and the forms follow the agent's movement.

Measuring depth accuracy in RGBD cameras

This paper presents the comparison between the Microsoft Kinect depth sensor and the Asus Xtion for computer vision applications. Depth sensors, known as RGBD cameras,
project an infrared pattern and calculate the depth from the reflected light using an infrared sensitive camera. In this research, we compare the depth sensing capabilities of the two sensors under various conditions. The purpose is to give the reader a background to whether use the Microsoft Kinect or Asus Xtion sensor to solve a specific computer vision problem. The properties of the two depth sensors were investigated by conducting a series of experiments evaluating the accuracy of the sensors under various conditions, which shows the advantages and disadvantages of both Microsoft Kinect and Asus Xtion sensor.

Cloud-based non-invasive tele-rehabilitation exercise monitoring

Exercise based rehabilitation plays a vital role in the recovery of various conditions, such as stroke, Parkinson’s disease (PD), chronic pain, and so on. Recently, telerehabilitation has become increasingly popular quantitative nature in assessments particularly for systematic monitoring of progress as well as cost saving for the patients as well as for the health care sector at large. However, challenges do exist in implementing a distributed bio-feedback in a cost-effective and efficient way. In this paper, we present the associated conceptual framework of cloud-based tele-rehabilitation system employing affordable non-invasive Microsoft Kinect® allowing patients to perform rehabilitation exercises in non-clinical setting such as home environments without loosing the quality of patients care. More importantly, different from existing tele-rehabilitation systems, our system not only measures whether patients can perform rehabilitation tasks, but also how well they can finish the tasks. Preliminary experiments validate its potential in training healthy subject to perform exercise motions emulating the physical rehabilitation process.

Efficacy comparison of clustering systems for limb detection

This paper presents a comparison of applying different clustering algorithms on a point cloud constructed from the depth maps captured by a RGBD camera such as Microsoft Kinect. The depth sensor is capable of returning images, where each pixel represents the distance to its corresponding point not the RGB data. This is considered as the real novelty of the RGBD camera in computer vision compared to the common video-based and stereo-based products. Depth sensors captures depth data without using markers, 2D to 3D-transition or determining feature points. The captured depth map then cluster the 3D depth points into different clusters to determine the different limbs of the human-body. The 3D points clustering is achieved by different clustering techniques. Our Experiments show good performance and results in using clustering to determine different human-body limbs.