Scene Reconstruction And Understanding By RGB-D Sensors

This thesis investigates interactive scene reconstruction and understanding using RGB-D data only. Indeed, we believe that depth cameras will still be in the near future a cheap and low-power 3D sensing alternative suitable for mobile devices too. Therefore, our contributions build on top of state-of-the-art approaches to achieve advances in three main challenging scenarios, namely mobile mapping, large scale surface reconstruction and semantic modeling. First, we will describe an effective approach dealing with Simultaneous Localization And Mapping (SLAM) on platforms with limited resources, such as a tablet device. Unlike previous methods, dense reconstruction is achieved by reprojection of RGB-D frames, while local consistency is maintained by deploying relative bundle adjustment principles. We will show quantitative results comparing our technique to the state-of-the-art as well as detailed reconstruction of various environments ranging from rooms to small apartments. Then, we will address large scale surface modeling from depth maps exploiting parallel GPU computing. We will develop a real-time camera tracking method based on the popular KinectFusion system and an online surface alignment technique capable of counteracting drift errors and closing small loops. We will show very high quality meshes outperforming existing methods on publicly available datasets as well as on data recorded with our RGB-D camera even in complete darkness. Finally, we will move to our Semantic Bundle Adjustment framework to effectively combine object detection and SLAM in a unified system. Though the mathematical framework we will describe does not restrict to a particular sensing technology, in the experimental section we will refer, again, only to RGB-D sensing. We will discuss successful implementations of our algorithm showing the benefit of a joint object detection, camera tracking and environment mapping.

Compressed sensing in digital tomosynthesis reconstruction

In this work we study a model for the breast image reconstruction in Digital Tomosynthesis, that is a non-invasive and non-destructive method for the three-dimensional visualization of the inner structures of an object, in which the data acquisition includes measuring a limited number of low-dose two-dimensional projections of an object by moving a detector and an X-ray tube around the object within a limited angular range. The problem of reconstructing 3D images from the projections provided in the Digital Tomosynthesis is an ill-posed inverse problem, that leads to a minimization problem with an object function that contains a data fitting term and a regularization term. The contribution of this thesis is to use the techniques of the compressed sensing, in particular replacing the standard least squares problem of data fitting with the problem of minimizing the 1-norm of the residuals, and using as regularization term the Total Variation (TV). We tested two different algorithms: a new alternating minimization algorithm (ADM), and a version of the more standard scaled projected gradient algorithm (SGP) that involves the 1-norm. We perform some experiments and analyse the performance of the two methods comparing relative errors, iterations number, times and the qualities of the reconstructed images. In conclusion we noticed that the use of the 1-norm and the Total Variation are valid tools in the formulation of the minimization problem for the image reconstruction resulting from Digital Tomosynthesis and the new algorithm ADM has reached a relative error comparable to a version of the classic algorithm SGP and proved best in speed and in the early appearance of the structures representing the masses.

A picture is worth a thousand words--the utility of 3D visualization illustrated by a case of survived pancreatic transection

It is one of the most important tasks of the forensic pathologist to explain the forensically relevant medical findings to medical non-professionals. However, it is often difficult to comment on the nature and potential consequences of organ injuries in a comprehensive way to individuals with limited knowledge of anatomy and physiology. This rare case of survived pancreatic transaction after kicks to the abdomen illustrates how the application of dedicated software programs for three-dimensional reconstruction can overcome these difficulties, allowing for clear and concise visualization of complex findings.

Importance of 3D-CT imaging in single-bullet cranioencephalic gunshot wounds

PURPOSE: The aim of this paper is to demonstrate that computed tomography (CT) and three-dimensional (3D) CT imaging techniques can be useful tools for evaluating gunshot wounds of the skull in forensic medicine. Three purposes can be achieved: (1) identifying and recognising the bullet entrance wound - and exit wound, if present; (2) recognising the bullet's intracranial course by studying damage to bone and brain tissue; (3) suggesting hypotheses as to the dynamics of the event. MATERIALS AND METHODS: Ten cadavers of people who died of a fatal head injury caused by a single gunshot were imaged with total-body CT prior to conventional autoptic examination. Three-dimensional-CT reconstructions were obtained with the volume-rendering technique, and data were analysed by two independent observers and compared with autopsy results. RESULTS: In our experience, CT analysis and volumetric reconstruction techniques allowed the identification of the bullet entrance and exit wounds and intracranial trajectory, as well as helping to formulate a hypothesis on the extracranial trajectory to corroborate circumstantial evidence. CONCLUSIONS: CT imaging techniques are excellent tools for addressing the most important questions of forensic medicine in the case of gunshot wounds of the skull, with results as good as (or sometimes better than) traditional autoptic methods.

Electroanatomic reconstruction of the left atrium, pulmonary veins, and esophagus compared with the "true anatomy" on multislice computed tomography in patients undergoing catheter ablation of atrial fibrillation

BACKGROUND: Current concepts of catheter ablation for atrial fibrillation (AF) commonly use three-dimensional (3D) reconstructions of the left atrium (LA) for orientation, catheter navigation, and ablation line placement. OBJECTIVES: The purpose of this study was to compare the 3D electroanatomic reconstruction (Carto) of the LA, pulmonary veins (PVs), and esophagus with the true anatomy displayed on multislice computed tomography (CT). METHODS: In this prospective study, 100 patients undergoing AF catheter ablation underwent contrast-enhanced spiral CT scan with barium swallow and subsequent multiplanar and 3D reconstructions. Using Carto, circumferential plus linear LA lesions were placed. The esophagus was tagged and integrated into the Carto map. RESULTS: Compared with the true anatomy on CT, the electroanatomic reconstruction accurately displayed the true distance between the lower PVs; the distances between left upper PV, left lower PV, right lower PV, and center of the esophagus; the longitudinal diameter of the encircling line around the funnel of the left PVs; and the length of the mitral isthmus line. Only the distances between the upper PVs, the distance between the right upper PV and esophagus, and the diameter of the right encircling line were significantly shorter on the electroanatomic reconstructions. Furthermore, electroanatomic tagging of the esophagus reliably visualized the true anatomic relationship to the LA. On multiple tagging and repeated CT scans, the LA and esophagus showed a stable anatomic relationship, without relevant sideward shifting of the esophagus. CONCLUSION: Electroanatomic reconstruction can display with high accuracy the true 3D anatomy of the LA and PVs in most of the regions of interest for AF catheter ablation. In addition, Carto was able to visualize the true anatomic relationship between the esophagus and LA. Both structures showed a stable anatomic relationship on Carto and CT without relevant sideward shifting of the esophagus.

3D C-arm as an alternative modality to CT in postmortem imaging: technical feasibility

OBJECT: The aim of our study was to demonstrate the image quality of the new device using human cadavers, extending the horizon of available imaging modalities in forensic medicine. MATERIALS AND METHODS: Six human cadavers were examined, revealing C-arm data sets of the head, neck thorax, abdomen and pelvis. High-resolution mode was performed with 500 fluoroscopy shots during a 190 degrees orbital movement with a constant tube voltage of 100 kV and a current of 4.6 mA. Based on these data sets subsequent three-dimensional reconstructions were generated. RESULTS: Reconstructed data sets revealed high-resolution images of all skeletal structures in a near-CT quality. The same image quality was available in all reconstruction planes. Artefacts caused by restorative dental materials are less accentuated in CBCT data sets. The system configuration was not powerful enough to generate sufficient images of intracranial structures. CONCLUSION: After the here-demonstrated encouraging preliminary results, the forensic indications that would be suitable for imaging with a 3D C-arm have to be defined. Promising seems the visualization local limited region of interest as the cervical spine or the facial skeleton.

Accurate and robust reconstruction of proximal femur from sparse intraoperative data and dense point distribution model for surgical navigation

Constructing a 3D surface model from sparse-point data is a nontrivial task. Here, we report an accurate and robust approach for reconstructing a surface model of the proximal femur from sparse-point data and a dense-point distribution model (DPDM). The problem is formulated as a three-stage optimal estimation process. The first stage, affine registration, is to iteratively estimate a scale and a rigid transformation between the mean surface model of the DPDM and the sparse input points. The estimation results of the first stage are used to establish point correspondences for the second stage, statistical instantiation, which stably instantiates a surface model from the DPDM using a statistical approach. This surface model is then fed to the third stage, kernel-based deformation, which further refines the surface model. Handling outliers is achieved by consistently employing the least trimmed squares (LTS) approach with a roughly estimated outlier rate in all three stages. If an optimal value of the outlier rate is preferred, we propose a hypothesis testing procedure to automatically estimate it. We present here our validations using four experiments, which include 1 leave-one-out experiment, 2 experiment on evaluating the present approach for handling pathology, 3 experiment on evaluating the present approach for handling outliers, and 4 experiment on reconstructing surface models of seven dry cadaver femurs using clinically relevant data without noise and with noise added. Our validation results demonstrate the robust performance of the present approach in handling outliers, pathology, and noise. An average 95-percentile error of 1.7-2.3 mm was found when the present approach was used to reconstruct surface models of the cadaver femurs from sparse-point data with noise added.

Haptics in forensics: the possibilities and advantages in using the haptic device for reconstruction approaches in forensic science

Non-invasive documentation methods such as surface scanning and radiological imaging are gaining in importance in the forensic field. These three-dimensional technologies provide digital 3D data, which are processed and handled in the computer. However, the sense of touch gets lost using the virtual approach. The haptic device enables the use of the sense of touch to handle and feel digital 3D data. The multifunctional application of a haptic device for forensic approaches is evaluated and illustrated in three different cases: the representation of bone fractures of the lower extremities, by traffic accidents, in a non-invasive manner; the comparison of bone injuries with the presumed injury-inflicting instrument; and in a gunshot case, the identification of the gun by the muzzle imprint, and the reconstruction of the holding position of the gun. The 3D models of the bones are generated from the Computed Tomography (CT) images. The 3D models of the exterior injuries, the injury-inflicting tools and the bone injuries, where a higher resolution is necessary, are created by the optical surface scan. The haptic device is used in combination with the software FreeForm Modelling Plus for touching the surface of the 3D models to feel the minute injuries and the surface of tools, to reposition displaced bone parts and to compare an injury-causing instrument with an injury. The repositioning of 3D models in a reconstruction is easier, faster and more precisely executed by means of using the sense of touch and with the user-friendly movement in the 3D space. For representation purposes, the fracture lines of bones are coloured. This work demonstrates that the haptic device is a suitable and efficient application in forensic science. The haptic device offers a new way in the handling of digital data in the virtual 3D space.

The role of 3D fusion imaging in sentinel lymphadenectomy for vulvar cancer

OBJECTIVES: To evaluate the feasibility of fusion imaging compound tomography (FICT) of CT/MRI and single photon emission tomography (SPECT) versus planar scintigraphy only (plSc) in pre-surgical staging for vulvar cancer. MATERIALS AND METHODS: Analysis of consecutive patients with vulvar cancer who preoperatively underwent sentinel scintigraphy (planar and 3D-SPECT imaging) and CT or MRI. Body markers were used for exact anatomical co-registration and fusion datasets were reconstructed using SPECT and CT/MRI. The number and localisation of all intraoperatively identified and resected sentinel lymph nodes (SLN) were compared between planar and 3D fusion imaging. RESULTS: Twenty six SLN were localized on planar scintigraphy. Twelve additional SLN were identified after SPECT and CT/MRI reconstruction, all of them were confirmed intraoperatively. In seven cases where single foci were identified at plSc, fusion imaging revealed grouped individual nodes and five additional localisations were discovered at fusion imaging. In seven patients both methods identified SLN contra lateral to the primary tumor site, but only fusion imaging allowed to localise iliac SLN in four patients. All SLN predicted on fusion imaging could be localised and resected during surgery. CONCLUSIONS: Fusion imaging using SPECT and CT/MRI can detect SLN in vulvar cancer more precisely than planar imaging regarding number and anatomical localisation. FICT revealed additional information in seven out of ten cases (70%).

3D volumetric intensity reconsturction from 2D x-ray images using partial least squares regression

Reconstruction of shape and intensity from 2D x-ray images has drawn more and more attentions. Previously introduced work suffers from the long computing time due to its iterative optimization characteristics and the requirement of generating digitally reconstructed radiographs within each iteration. In this paper, we propose a novel method which uses a patient-specific 3D surface model reconstructed from 2D x-ray images as a surrogate to get a patient-specific volumetric intensity reconstruction via partial least squares regression. No DRR generation is needed. The method was validated on 20 cadaveric proximal femurs by performing a leave-one-out study. Qualitative and quantitative results demonstrated the efficacy of the present method. Compared to the existing work, the present method has the advantage of much shorter computing time and can be applied to both DXA images as well as conventional x-ray images, which may hold the potentials to be applied to clinical routine task such as total hip arthroplasty (THA).

Detergent-induced stabilization and improved 3D map of the human heteromeric amino acid transporter 4F2hc-LAT2.

Human heteromeric amino acid transporters (HATs) are membrane protein complexes that facilitate the transport of specific amino acids across cell membranes. Loss of function or overexpression of these transporters is implicated in several human diseases such as renal aminoacidurias and cancer. HATs are composed of two subunits, a heavy and a light subunit, that are covalently connected by a disulphide bridge. Light subunits catalyse amino acid transport and consist of twelve transmembrane α-helix domains. Heavy subunits are type II membrane N-glycoproteins with a large extracellular domain and are involved in the trafficking of the complex to the plasma membrane. Structural information on HATs is scarce because of the difficulty in heterologous overexpression. Recently, we had a major breakthrough with the overexpression of a recombinant HAT, 4F2hc-LAT2, in the methylotrophic yeast Pichia pastoris. Microgram amounts of purified protein made possible the reconstruction of the first 3D map of a human HAT by negative-stain transmission electron microscopy. Here we report the important stabilization of purified human 4F2hc-LAT2 using a combination of two detergents, i.e., n-dodecyl-β-D-maltopyranoside and lauryl maltose neopentyl glycol, and cholesteryl hemisuccinate. The superior quality and stability of purified 4F2hc-LAT2 allowed the measurement of substrate binding by scintillation proximity assay. In addition, an improved 3D map of this HAT could be obtained. The detergent-induced stabilization of the purified human 4F2hc-LAT2 complex presented here paves the way towards its crystallization and structure determination at high-resolution, and thus the elucidation of the working mechanism of this important protein complex at the molecular level.

Can 3D ultrasound identify trochlea dysplasia in newborns? Evaluation and applicability of a technique.

Femoro-patellar dysplasia is considered as a significant risk factor of patellar instability. Different studies suggest that the shape of the trochlea is already developed in early childhood. Therefore early identification of a dysplastic configuration might be relevant information for the treating physician. An easy applicable routine screening of the trochlea is yet not available. The purpose of this study was to establish and evaluate a screening method for femoro-patellar dysplasia using 3D ultrasound. From 2012 to 2013 we prospectively imaged 160 consecutive femoro-patellar joints in 80 newborns from the 36th to 61st gestational week that underwent a routine hip sonography (Graf). All ultrasounds were performed by a pediatric radiologist with only minimal additional time to the routine hip ultrasound. In 30° flexion of the knee, axial, coronal, and sagittal reformats were used to standardize a reconstructed axial plane through the femoral condyle and the mid-patella. The sulcus angle, the lateral-to-medial facet ratio of the trochlea and the shape of the patella (Wiberg Classification) were evaluated. In all examinations reconstruction of the standardized axial plane was achieved, the mean trochlea angle was 149.1° (SD 4.9°), the lateral-to-medial facet ratio of the trochlea ratio was 1.3 (SD 0.22), and a Wiberg type I patella was found in 95% of the newborn. No statistical difference was detected between boys and girls. Using standardized reconstructions of the axial plane allows measurements to be made with lower operator dependency and higher accuracy in a short time. Therefore 3D ultrasound is an easy applicable and powerful tool to identify trochlea dysplasia in newborns and might be used for screening for trochlea dysplasia.

Computerized Three-Dimmensional Craniofacial Reconstruction from Skulls Based on Landmarks.

Human identification from a skull is a critical process in legal and forensic medicine, specially when no other means are available. Traditional clay-based methods attempt to generate the human face, in order to identify the corresponding person. However, these reconstructions lack of objectivity and consistence, since they depend on the practitioner. Current computerized techniques are based on facial models, which introduce undesired facial features when the final reconstruction is built. This paper presents an objective 3D craniofacial reconstruction technique, implemented in a graphic application, without using any facial template. The only information required by the software tool is the 3D image of the target skull and three parameters: age, gender and Body Mass Index (BMI) of the individual. Complexity is minimized, since the application database only consists of the anthropological information provided by soft tissue depth values in a set of points of the skull.

Efficient methodologies for system matrix modelling in iterative image reconstruction for rotating high-resolution PET

A fully 3D iterative image reconstruction algorithm has been developed for high-resolution PET cameras composed of pixelated scintillator crystal arrays and rotating planar detectors, based on the ordered subsets approach. The associated system matrix is precalculated with Monte Carlo methods that incorporate physical effects not included in analytical models, such as positron range effects and interaction of the incident gammas with the scintillator material. Custom Monte Carlo methodologies have been developed and optimized for modelling of system matrices for fast iterative image reconstruction adapted to specific scanner geometries, without redundant calculations. According to the methodology proposed here, only one-eighth of the voxels within two central transaxial slices need to be modelled in detail. The rest of the system matrix elements can be obtained with the aid of axial symmetries and redundancies, as well as in-plane symmetries within transaxial slices. Sparse matrix techniques for the non-zero system matrix elements are employed, allowing for fast execution of the image reconstruction process. This 3D image reconstruction scheme has been compared in terms of image quality to a 2D fast implementation of the OSEM algorithm combined with Fourier rebinning approaches. This work confirms the superiority of fully 3D OSEM in terms of spatial resolution, contrast recovery and noise reduction as compared to conventional 2D approaches based on rebinning schemes. At the same time it demonstrates that fully 3D methodologies can be efficiently applied to the image reconstruction problem for high-resolution rotational PET cameras by applying accurate pre-calculated system models and taking advantage of the system's symmetries.

Massively parallelizable list-mode reconstruction using a Monte Carlo-based elliptical Gaussian model

Purpose: A fully three-dimensional (3D) massively parallelizable list-mode ordered-subsets expectation-maximization (LM-OSEM) reconstruction algorithm has been developed for high-resolution PET cameras. System response probabilities are calculated online from a set of parameters derived from Monte Carlo simulations. The shape of a system response for a given line of response (LOR) has been shown to be asymmetrical around the LOR. This work has been focused on the development of efficient region-search techniques to sample the system response probabilities, which are suitable for asymmetric kernel models, including elliptical Gaussian models that allow for high accuracy and high parallelization efficiency. The novel region-search scheme using variable kernel models is applied in the proposed PET reconstruction algorithm. Methods: A novel region-search technique has been used to sample the probability density function in correspondence with a small dynamic subset of the field of view that constitutes the region of response (ROR). The ROR is identified around the LOR by searching for any voxel within a dynamically calculated contour. The contour condition is currently defined as a fixed threshold over the posterior probability, and arbitrary kernel models can be applied using a numerical approach. The processing of the LORs is distributed in batches among the available computing devices, then, individual LORs are processed within different processing units. In this way, both multicore and multiple many-core processing units can be efficiently exploited. Tests have been conducted with probability models that take into account the noncolinearity, positron range, and crystal penetration effects, that produced tubes of response with varying elliptical sections whose axes were a function of the crystal's thickness and angle of incidence of the given LOR. The algorithm treats the probability model as a 3D scalar field defined within a reference system aligned with the ideal LOR. Results: This new technique provides superior image quality in terms of signal-to-noise ratio as compared with the histogram-mode method based on precomputed system matrices available for a commercial small animal scanner. Reconstruction times can be kept low with the use of multicore, many-core architectures, including multiple graphic processing units. Conclusions: A highly parallelizable LM reconstruction method has been proposed based on Monte Carlo simulations and new parallelization techniques aimed at improving the reconstruction speed and the image signal-to-noise of a given OSEM algorithm. The method has been validated using simulated and real phantoms. A special advantage of the new method is the possibility of defining dynamically the cut-off threshold over the calculated probabilities thus allowing for a direct control on the trade-off between speed and quality during the reconstruction.