POSE ESTIMATION AND 3D RECONSTRUCTION USING SENSOR FUSION

A camera maps 3-dimensional (3D) world space to a 2-dimensional (2D) image space. In the process it loses the depth information, i.e., the distance from the camera focal point to the imaged objects. It is impossible to recover this information from a single image. However, by using two or more images from different viewing angles this information can be recovered, which in turn can be used to obtain the pose (position and orientation) of the camera. Using this pose, a 3D reconstruction of imaged objects in the world can be computed. Numerous algorithms have been proposed and implemented to solve the above problem; these algorithms are commonly called Structure from Motion (SfM). State-of-the-art SfM techniques have been shown to give promising results. However, unlike a Global Positioning System (GPS) or an Inertial Measurement Unit (IMU) which directly give the position and orientation respectively, the camera system estimates it after implementing SfM as mentioned above. This makes the pose obtained from a camera highly sensitive to the images captured and other effects, such as low lighting conditions, poor focus or improper viewing angles. In some applications, for example, an Unmanned Aerial Vehicle (UAV) inspecting a bridge or a robot mapping an environment using Simultaneous Localization and Mapping (SLAM), it is often difficult to capture images with ideal conditions. This report examines the use of SfM methods in such applications and the role of combining multiple sensors, viz., sensor fusion, to achieve more accurate and usable position and reconstruction information. This project investigates the role of sensor fusion in accurately estimating the pose of a camera for the application of 3D reconstruction of a scene. The first set of experiments is conducted in a motion capture room. These results are assumed as ground truth in order to evaluate the strengths and weaknesses of each sensor and to map their coordinate systems. Then a number of scenarios are targeted where SfM fails. The pose estimates obtained from SfM are replaced by those obtained from other sensors and the 3D reconstruction is completed. Quantitative and qualitative comparisons are made between the 3D reconstruction obtained by using only a camera versus that obtained by using the camera along with a LIDAR and/or an IMU. Additionally, the project also works towards the performance issue faced while handling large data sets of high-resolution images by implementing the system on the Superior high performance computing cluster at Michigan Technological University.

Design and implementation of a 3D computer game controller using inertial MEMS sensors

Though 3D computer graphics has seen tremendous advancement in the past two decades, most available mechanisms for computer interaction in 3D are high cost and targeted for industry and virtual reality applications. Recent advances in Micro-Electro-Mechanical-System (MEMS) devices have brought forth a variety of new low-cost, low-power, miniature sensors with high accuracy, which are well suited for hand-held devices. In this work a novel design for a 3D computer game controller using inertial sensors is proposed, and a prototype device based on this design is implemented. The design incorporates MEMS accelerometers and gyroscopes from Analog Devices to measure the three components of the acceleration and angular velocity. From these sensor readings, the position and orientation of the hand-held compartment can be calculated using numerical methods. The implemented prototype is utilizes a USB 2.0 compliant interface for power and communication with the host system. A Microchip dsPIC microcontroller is used in the design. This microcontroller integrates the analog to digital converters, the program memory flash, as well as the core processor, on a single integrated circuit. A PC running Microsoft Windows operating system is used as the host machine. Prototype firmware for the microcontroller is developed and tested to establish the communication between the design and the host, and perform the data acquisition and initial filtering of the sensor data. A PC front-end application with a graphical interface is developed to communicate with the device, and allow real-time visualization of the acquired data.

Planar magneto-photonic and gradient-photonic structures : crystals and metamaterials.

In the field of photonics, two new types of material structures, photonic crystals and metamaterials, are presently of great interest. Both are studied in the present work, which focus on planar magnetic materials in the former and planar gradient metamaterials in the latter. These planar periodic structures are easy to handle and integrate into optical systems. The applications are promising field for future optical telecommunication systems and give rise to new optical, microwave and radio technologies. The photonic crystal part emphasizes the utilization of magnetic material based photonic crystals due to its remarkable magneto-optical characteristics. Bandgaps tuning by magnetic field in bismuth-gadolinium-substituted lutetium iron garnet (Bi0.8 Gd0.2 Lu2.0 Fe5 O12) based one- dimensional photonic crystals are investigated and demonstrated in this work. Magnetic optical switches are fabricated and tested. Waveguide formulation for band structure in magneto photonic crystals is developed. We also for the first time demonstrate and test two- dimensional magneto photonic crystals optical. We observe multi-stopbands in two- dimensional photonic waveguide system and study the origin of multi-stopbands. The second part focus on studying photonic metamaterials and planar gradient photonic metamaterial design. We systematically study the effects of varying the geometry of the fishnet unit cell on the refractive index in optical frequency. It is the first time to design and demonstrate the planar gradient structure in the high optical frequency. Optical beam bending using planar gradient photonic metamaterials is observed. The technologies needed for the fabrication of the planar gradient photonic metamaterials are investigated. Beam steering devices, shifter, gradient optical lenses and etc. can be derived from this design.

Studies of spatial clustering of inertial particles in turbulence

We present studies of the spatial clustering of inertial particles embedded in turbulent flow. A major part of the thesis is experimental, involving the technique of Phase Doppler Interferometry (PDI). The thesis also includes significant amount of simulation studies and some theoretical considerations. We describe the details of PDI and explain why it is suitable for study of particle clustering in turbulent flow with a strong mean velocity. We introduce the concept of the radial distribution function (RDF) as our chosen way of quantifying inertial particle clustering and present some original works on foundational and practical considerations related to it. These include methods of treating finite sampling size, interpretation of the magnitude of RDF and the possibility of isolating RDF signature of inertial clustering from that of large scale mixing. In experimental work, we used the PDI to observe clustering of water droplets in a turbulent wind tunnel. From that we present, in the form of a published paper, evidence of dynamical similarity (Stokes number similarity) of inertial particle clustering together with other results in qualitative agreement with available theoretical prediction and simulation results. We next show detailed quantitative comparisons of results from our experiments, direct-numerical-simulation (DNS) and theory. Very promising agreement was found for like-sized particles (mono-disperse). Theory is found to be incorrect regarding clustering of different-sized particles and we propose a empirical correction based on the DNS and experimental results. Besides this, we also discovered a few interesting characteristics of inertial clustering. Firstly, through observations, we found an intriguing possibility for modeling the RDF arising from inertial clustering that has only one (sensitive) parameter. We also found that clustering becomes saturated at high Reynolds number.

Optical hysteresis of ridge waveguide magnetic garnet films and cavities in two-dimensional magneto-photonic crystal slabs

Magnetic iron garnets as well as magnetic photonic crystals are of great interests in magneto-optic applications such as isolators, current captors, circulators, TE-TM mode conversion, wavelength accordable filters, optical sensors and switches, all of which provide a promising platform for future integrated optical circuits. In the present work, two topics are studied based on magnetic iron garnet films. In the first part, the characteristics of the magnetization are investigated for ridge waveguides fabricated on (100) oriented iron garnet thin films. The magnetic response in magneto-optic waveguides patterned on epitaxial magnetic garnet films depends on the crystallographic orientation of the waveguides and the magnetic anisotropy of the material. These can be studied by polarization rotation hysteresis loops, which are related to the component of magnetization parallel to the light propagation direction and the linear birefringence. Polarization rotation hysteresis loops for low birefringence waveguides with different orientations are experimentally investigated. Asymmetric stepped curves are obtained from waveguides along, due to the large magnetocrystalline anisotropy in the plane. A model based on the free energy density is developed to demonstrate the motion of the magnetization and can be used in the design of magneto-optic devices. The second part of this thesis focuses on the design and fabrication of high-Q cavities in two-dimensional magneto-photonic crystal slabs. The device consists of a layer of silicon and a layer of iron garnet thin film. Triangular lattice elliptical air holes are patterned in the slab. The fundamental TM band gap overlaps with the first-order TE band gap from 0374~0.431(a/λ) showing that both TE and TM polarization light can be confined in the photonic crystals. A nanocavity is designed to obtain both TE and TM defect modes in the band gaps. Additional work is needed to overlap the TE and TM defect modes and obtain a high-Q cavity so as to develop miniaturized Faraday rotators.

VISUAL-INERTIAL INTEGRATION FOR HUMAN MOTION TRACKING AND NAVIGATION IN FREE-LIVING ENVIRONMENTS

The aging population has become a burning issue for all modern societies around the world recently. There are two important issues existing now to be solved. One is how to continuously monitor the movements of those people having suffered a stroke in natural living environment for providing more valuable feedback to guide clinical interventions. The other one is how to guide those old people effectively when they are at home or inside other buildings and to make their life easier and convenient. Therefore, human motion tracking and navigation have been active research fields with the increasing number of elderly people. However, motion capture has been extremely challenging to go beyond laboratory environments and obtain accurate measurements of human physical activity especially in free-living environments, and navigation in free-living environments also poses some problems such as the denied GPS signal and the moving objects commonly presented in free-living environments. This thesis seeks to develop new technologies to enable accurate motion tracking and positioning in free-living environments. This thesis comprises three specific goals using our developed IMU board and the camera from the imaging source company: (1) to develop a robust and real-time orientation algorithm using only the measurements from IMU; (2) to develop a robust distance estimation in static free-living environments to estimate people’s position and navigate people in static free-living environments and simultaneously the scale ambiguity problem, usually appearing in the monocular camera tracking, is solved by integrating the data from the visual and inertial sensors; (3) in case of moving objects viewed by the camera existing in free-living environments, to firstly design a robust scene segmentation algorithm and then respectively estimate the motion of the vIMU system and moving objects. To achieve real-time orientation tracking, an Adaptive-Gain Orientation Filter (AGOF) is proposed in this thesis based on the basic theory of deterministic approach and frequency-based approach using only measurements from the newly developed MARG (Magnet, Angular Rate, and Gravity) sensors. To further obtain robust positioning, an adaptive frame-rate vision-aided IMU system is proposed to develop and implement fast vIMU ego-motion estimation algorithms, where the orientation is estimated in real time from MARG sensors in the first step and then used to estimate the position based on the data from visual and inertial sensors. In case of the moving objects viewed by the camera existing in free-living environments, a robust scene segmentation algorithm is firstly proposed to obtain position estimation and simultaneously the 3D motion of moving objects. Finally, corresponding simulations and experiments have been carried out.

MAGNETO-PHOTONIC CRYSTALS FOR OPTICAL SENSING APPLICATIONS

Among the optical structures investigated for optical sensing purpose, a significant amount of research has been conducted on photonic crystal based sensors. A particular advantage of photonic crystal based sensors is that they show superior sensitivity for ultra-small volume sensing. In this study we investigate polarization changes in response to the changes in the cover index of magneto-optic active photonic band gap structures. One-dimensional photonic-band gap structures fabricated on iron garnet materials yield large polarization rotations at the band gap edges. The enhanced polarization effects serve as an excellent tool for chemical sensing showing high degree of sensitivity for photonic crystal cover refractive index changes. The one dimensional waveguide photonic crystals are fabricated on single-layer bismuth-substituted rare earth iron garnet films ((Bi, Y, Lu)3(Fe, Ga)5O12 ) grown by liquid phase epitaxy on gadolinium gallium garnet substrates. Band gaps have been observed where Bragg scattering conditions links forward-going fundamental waveguide modes to backscattered high-order waveguide modes. Large near-band-edge polarization rotations which increase progressively with backscattered-mode order have been experimentally demonstrated for multiple samples with different composition, film thickness and fabrication parameters. Experimental findings are supported by theoretical analysis of Bloch modes polarization states showing that large near stop-band edge rotations are induced by the magneto-photonic crystal. Theoretical and experimental analysis conducted on polarization rotation sensitivity to waveguide photonic crystal cover refractive index changes shows a monotonic enhancement of the rotation with cover index. The sensor is further developed for selective chemical sensing by employing Polypyrrole as the photonic crystal cover layer. Polypyrrole is one of the extensively studied conducting polymers for selective analyte detection. Successful detection of aqueous ammonia and methanol has been achieved with Polypyrrole deposited magneto-photonic crystals.

Automatic Food Intake Assessment Using Camera Phones

Obesity is becoming an epidemic phenomenon in most developed countries. The fundamental cause of obesity and overweight is an energy imbalance between calories consumed and calories expended. It is essential to monitor everyday food intake for obesity prevention and management. Existing dietary assessment methods usually require manually recording and recall of food types and portions. Accuracy of the results largely relies on many uncertain factors such as user's memory, food knowledge, and portion estimations. As a result, the accuracy is often compromised. Accurate and convenient dietary assessment methods are still blank and needed in both population and research societies. In this thesis, an automatic food intake assessment method using cameras, inertial measurement units (IMUs) on smart phones was developed to help people foster a healthy life style. With this method, users use their smart phones before and after a meal to capture images or videos around the meal. The smart phone will recognize food items and calculate the volume of the food consumed and provide the results to users. The technical objective is to explore the feasibility of image based food recognition and image based volume estimation. This thesis comprises five publications that address four specific goals of this work: (1) to develop a prototype system with existing methods to review the literature methods, find their drawbacks and explore the feasibility to develop novel methods; (2) based on the prototype system, to investigate new food classification methods to improve the recognition accuracy to a field application level; (3) to design indexing methods for large-scale image database to facilitate the development of new food image recognition and retrieval algorithms; (4) to develop novel convenient and accurate food volume estimation methods using only smart phones with cameras and IMUs. A prototype system was implemented to review existing methods. Image feature detector and descriptor were developed and a nearest neighbor classifier were implemented to classify food items. A reedit card marker method was introduced for metric scale 3D reconstruction and volume calculation. To increase recognition accuracy, novel multi-view food recognition algorithms were developed to recognize regular shape food items. To further increase the accuracy and make the algorithm applicable to arbitrary food items, new food features, new classifiers were designed. The efficiency of the algorithm was increased by means of developing novel image indexing method in large-scale image database. Finally, the volume calculation was enhanced through reducing the marker and introducing IMUs. Sensor fusion technique to combine measurements from cameras and IMUs were explored to infer the metric scale of the 3D model as well as reduce noises from these sensors.

STRUCTURAL CONTROL OF A SMALL-SCALE TEST-BED SHAKER STRUCTURE USING A SPONGE-TYPE MAGNETO-RHEOLOGICAL FLUID DAMPER

Semi-active damping devices have been shown to be effective in mitigating unwanted vibrations in civil structures. These devices impart force indirectly through real-time alterations to structural properties. Simulating the complex behavior of these devices for laboratory-scale experiments is a major challenge. Commercial devices for seismic applications typically operate in the 2-10 kN range; this force is too high for small-scale testing applications where requirements typically range from 0-10 N. Several challenges must be overcome to produce damping forces at this level. In this study, a small-scale magneto-rheological (MR) damper utilizing a fluid absorbent metal foam matrix is developed and tested to accomplish this goal. This matrix allows magneto-rheological (MR) fluid to be extracted upon magnetic excitation in order to produce MR-fluid shear stresses and viscosity effects between an electromagnetic piston, the foam, and the damper housing. Dampers for uniaxial seismic excitation are traditionally positioned in the horizontal orientation allowing MR-fluid to gather in the lower part of the damper housing when partially filled. Thus, the absorbent matrix is placed in the bottom of the housing relieving the need to fill the entire device with MR-fluid, a practice that requires seals that add significant unwanted friction to the desired low-force device. The damper, once constructed, can be used in feedback control applications to reduce seismic vibrations and to test structural control algorithms and wireless command devices. To validate this device, a parametric study was performed utilizing force and acceleration measurements to characterize damper performance and controllability for this actuator. A discussion of the results is presented to demonstrate the attainment of the damper design objectives.