3-D reconstruction of the human skeleton during motion

L’analisi del movimento umano ha come obiettivo la descrizione del movimento assoluto e relativo dei segmenti ossei del soggetto e, ove richiesto, dei relativi tessuti molli durante l’esecuzione di esercizi fisici. La bioingegneria mette a disposizione dell’analisi del movimento gli strumenti ed i metodi necessari per una valutazione quantitativa di efficacia, funzione e/o qualità del movimento umano, consentendo al clinico l’analisi di aspetti non individuabili con gli esami tradizionali. Tali valutazioni possono essere di ausilio all’analisi clinica di pazienti e, specialmente con riferimento a problemi ortopedici, richiedono una elevata accuratezza e precisione perché il loro uso sia valido. Il miglioramento della affidabilità dell’analisi del movimento ha quindi un impatto positivo sia sulla metodologia utilizzata, sia sulle ricadute cliniche della stessa. Per perseguire gli obiettivi scientifici descritti, è necessario effettuare una stima precisa ed accurata della posizione e orientamento nello spazio dei segmenti ossei in esame durante l’esecuzione di un qualsiasi atto motorio. Tale descrizione può essere ottenuta mediante la definizione di un modello della porzione del corpo sotto analisi e la misura di due tipi di informazione: una relativa al movimento ed una alla morfologia. L’obiettivo è quindi stimare il vettore posizione e la matrice di orientamento necessari a descrivere la collocazione nello spazio virtuale 3D di un osso utilizzando le posizioni di punti, definiti sulla superficie cutanea ottenute attraverso la stereofotogrammetria. Le traiettorie dei marker, così ottenute, vengono utilizzate per la ricostruzione della posizione e dell’orientamento istantaneo di un sistema di assi solidale con il segmento sotto esame (sistema tecnico) (Cappozzo et al. 2005). Tali traiettorie e conseguentemente i sistemi tecnici, sono affetti da due tipi di errore, uno associato allo strumento di misura e l’altro associato alla presenza di tessuti molli interposti tra osso e cute. La propagazione di quest’ultimo ai risultati finali è molto più distruttiva rispetto a quella dell’errore strumentale che è facilmente minimizzabile attraverso semplici tecniche di filtraggio (Chiari et al. 2005). In letteratura è stato evidenziato che l’errore dovuto alla deformabilità dei tessuti molli durante l’analisi del movimento umano provoca inaccuratezze tali da mettere a rischio l’utilizzabilità dei risultati. A tal proposito Andriacchi scrive: “attualmente, uno dei fattori critici che rallentano il progresso negli studi del movimento umano è la misura del movimento scheletrico partendo dai marcatori posti sulla cute” (Andriacchi et al. 2000). Relativamente alla morfologia, essa può essere acquisita, ad esempio, attraverso l’utilizzazione di tecniche per bioimmagini. Queste vengono fornite con riferimento a sistemi di assi locali in generale diversi dai sistemi tecnici. Per integrare i dati relativi al movimento con i dati morfologici occorre determinare l’operatore che consente la trasformazione tra questi due sistemi di assi (matrice di registrazione) e di conseguenza è fondamentale l’individuazione di particolari terne di riferimento, dette terne anatomiche. L’identificazione di queste terne richiede la localizzazione sul segmento osseo di particolari punti notevoli, detti repere anatomici, rispetto ad un sistema di riferimento solidale con l’osso sotto esame. Tale operazione prende il nome di calibrazione anatomica. Nella maggior parte dei laboratori di analisi del movimento viene implementata una calibrazione anatomica a “bassa risoluzione” che prevede la descrizione della morfologia dell’osso a partire dall’informazione relativa alla posizione di alcuni repere corrispondenti a prominenze ossee individuabili tramite palpazione. Attraverso la stereofotogrammetria è quindi possibile registrare la posizione di questi repere rispetto ad un sistema tecnico. Un diverso approccio di calibrazione anatomica può essere realizzato avvalendosi delle tecniche ad “alta risoluzione”, ovvero attraverso l’uso di bioimmagini. In questo caso è necessario disporre di una rappresentazione digitale dell’osso in un sistema di riferimento morfologico e localizzare i repere d’interesse attraverso palpazione in ambiente virtuale (Benedetti et al. 1994 ; Van Sint Jan et al. 2002; Van Sint Jan et al. 2003). Un simile approccio è difficilmente applicabile nella maggior parte dei laboratori di analisi del movimento, in quanto normalmente non si dispone della strumentazione necessaria per ottenere le bioimmagini; inoltre è noto che tale strumentazione in alcuni casi può essere invasiva. Per entrambe le calibrazioni anatomiche rimane da tenere in considerazione che, generalmente, i repere anatomici sono dei punti definiti arbitrariamente all’interno di un’area più vasta e irregolare che i manuali di anatomia definiscono essere il repere anatomico. L’identificazione dei repere attraverso una loro descrizione verbale è quindi povera in precisione e la difficoltà nella loro identificazione tramite palpazione manuale, a causa della presenza dei tessuti molli interposti, genera errori sia in precisione che in accuratezza. Tali errori si propagano alla stima della cinematica e della dinamica articolare (Ramakrishnan et al. 1991; Della Croce et al. 1999). Della Croce (Della Croce et al. 1999) ha inoltre evidenziato che gli errori che influenzano la collocazione nello spazio delle terne anatomiche non dipendono soltanto dalla precisione con cui vengono identificati i repere anatomici, ma anche dalle regole che si utilizzano per definire le terne. E’ infine necessario evidenziare che la palpazione manuale richiede tempo e può essere effettuata esclusivamente da personale altamente specializzato, risultando quindi molto onerosa (Simon 2004). La presente tesi prende lo spunto dai problemi sopra elencati e ha come obiettivo quello di migliorare la qualità delle informazioni necessarie alla ricostruzione della cinematica 3D dei segmenti ossei in esame affrontando i problemi posti dall’artefatto di tessuto molle e le limitazioni intrinseche nelle attuali procedure di calibrazione anatomica. I problemi sono stati affrontati sia mediante procedure di elaborazione dei dati, sia apportando modifiche ai protocolli sperimentali che consentano di conseguire tale obiettivo. Per quanto riguarda l’artefatto da tessuto molle, si è affrontato l’obiettivo di sviluppare un metodo di stima che fosse specifico per il soggetto e per l’atto motorio in esame e, conseguentemente, di elaborare un metodo che ne consentisse la minimizzazione. Il metodo di stima è non invasivo, non impone restrizione al movimento dei tessuti molli, utilizza la sola misura stereofotogrammetrica ed è basato sul principio della media correlata. Le prestazioni del metodo sono state valutate su dati ottenuti mediante una misura 3D stereofotogrammetrica e fluoroscopica sincrona (Stagni et al. 2005), (Stagni et al. 2005). La coerenza dei risultati raggiunti attraverso i due differenti metodi permette di considerare ragionevoli le stime dell’artefatto ottenute con il nuovo metodo. Tale metodo fornisce informazioni sull’artefatto di pelle in differenti porzioni della coscia del soggetto e durante diversi compiti motori, può quindi essere utilizzato come base per un piazzamento ottimo dei marcatori. Lo si è quindi utilizzato come punto di partenza per elaborare un metodo di compensazione dell’errore dovuto all’artefatto di pelle che lo modella come combinazione lineare degli angoli articolari di anca e ginocchio. Il metodo di compensazione è stato validato attraverso una procedura di simulazione sviluppata ad-hoc. Relativamente alla calibrazione anatomica si è ritenuto prioritario affrontare il problema associato all’identificazione dei repere anatomici perseguendo i seguenti obiettivi: 1. migliorare la precisione nell’identificazione dei repere e, di conseguenza, la ripetibilità dell’identificazione delle terne anatomiche e della cinematica articolare, 2. diminuire il tempo richiesto, 3. permettere che la procedura di identificazione possa essere eseguita anche da personale non specializzato. Il perseguimento di tali obiettivi ha portato alla implementazione dei seguenti metodi: • Inizialmente è stata sviluppata una procedura di palpazione virtuale automatica. Dato un osso digitale, la procedura identifica automaticamente i punti di repere più significativi, nella maniera più precisa possibile e senza l'ausilio di un operatore esperto, sulla base delle informazioni ricavabili da un osso digitale di riferimento (template), preliminarmente palpato manualmente. • E’ stato poi condotto uno studio volto ad indagare i fattori metodologici che influenzano le prestazioni del metodo funzionale nell’individuazione del centro articolare d’anca, come prerequisito fondamentale per migliorare la procedura di calibrazione anatomica. A tale scopo sono stati confrontati diversi algoritmi, diversi cluster di marcatori ed è stata valutata la prestazione del metodo in presenza di compensazione dell’artefatto di pelle. • E’stato infine proposto un metodo alternativo di calibrazione anatomica basato sull’individuazione di un insieme di punti non etichettati, giacenti sulla superficie dell’osso e ricostruiti rispetto ad un TF (UP-CAST). A partire dalla posizione di questi punti, misurati su pelvi coscia e gamba, la morfologia del relativo segmento osseo è stata stimata senza identificare i repere, bensì effettuando un’operazione di matching dei punti misurati con un modello digitale dell’osso in esame. La procedura di individuazione dei punti è stata eseguita da personale non specializzato nell’individuazione dei repere anatomici. Ai soggetti in esame è stato richiesto di effettuare dei cicli di cammino in modo tale da poter indagare gli effetti della nuova procedura di calibrazione anatomica sulla determinazione della cinematica articolare. I risultati ottenuti hanno mostrato, per quel che riguarda la identificazione dei repere, che il metodo proposto migliora sia la precisione inter- che intraoperatore, rispetto alla palpazione convenzionale (Della Croce et al. 1999). E’ stato inoltre riscontrato un notevole miglioramento, rispetto ad altri protocolli (Charlton et al. 2004; Schwartz et al. 2004), nella ripetibilità della cinematica 3D di anca e ginocchio. Bisogna inoltre evidenziare che il protocollo è stato applicato da operatori non specializzati nell’identificazione dei repere anatomici. Grazie a questo miglioramento, la presenza di diversi operatori nel laboratorio non genera una riduzione di ripetibilità. Infine, il tempo richiesto per la procedura è drasticamente diminuito. Per una analisi che include la pelvi e i due arti inferiori, ad esempio, l’identificazione dei 16 repere caratteristici usando la calibrazione convenzionale richiede circa 15 minuti, mentre col nuovo metodo tra i 5 e i 10 minuti.

Reconstruction from image correspondences

A single picture provides a largely incomplete representation of the scene one is looking at. Usually it reproduces only a limited spatial portion of the scene according to the standpoint and the viewing angle, besides it contains only instantaneous information. Thus very little can be understood on the geometrical structure of the scene, the position and orientation of the observer with respect to it remaining also hard to guess. When multiple views, taken from different positions in space and time, observe the same scene, then a much deeper knowledge is potentially achievable. Understanding inter-views relations enables construction of a collective representation by fusing the information contained in every single image. Visual reconstruction methods confront with the formidable, and still unanswered, challenge of delivering a comprehensive representation of structure, motion and appearance of a scene from visual information. Multi-view visual reconstruction deals with the inference of relations among multiple views and the exploitation of revealed connections to attain the best possible representation. This thesis investigates novel methods and applications in the field of visual reconstruction from multiple views. Three main threads of research have been pursued: dense geometric reconstruction, camera pose reconstruction, sparse geometric reconstruction of deformable surfaces. Dense geometric reconstruction aims at delivering the appearance of a scene at every single point. The construction of a large panoramic image from a set of traditional pictures has been extensively studied in the context of image mosaicing techniques. An original algorithm for sequential registration suitable for real-time applications has been conceived. The integration of the algorithm into a visual surveillance system has lead to robust and efficient motion detection with Pan-Tilt-Zoom cameras. Moreover, an evaluation methodology for quantitatively assessing and comparing image mosaicing algorithms has been devised and made available to the community. Camera pose reconstruction deals with the recovery of the camera trajectory across an image sequence. A novel mosaic-based pose reconstruction algorithm has been conceived that exploit image-mosaics and traditional pose estimation algorithms to deliver more accurate estimates. An innovative markerless vision-based human-machine interface has also been proposed, so as to allow a user to interact with a gaming applications by moving a hand held consumer grade camera in unstructured environments. Finally, sparse geometric reconstruction refers to the computation of the coarse geometry of an object at few preset points. In this thesis, an innovative shape reconstruction algorithm for deformable objects has been designed. A cooperation with the Solar Impulse project allowed to deploy the algorithm in a very challenging real-world scenario, i.e. the accurate measurements of airplane wings deformations.

Reconstruction of CNGS neutrino events in the emulsions of the OPERA experiment

The OPERA experiment aims at the direct observation of ν_mu -> ν_tau oscillations in the CNGS (CERN Neutrinos to Gran Sasso) neutrino beam produced at CERN; since the ν_e contamination in the CNGS beam is low, OPERA will also be able to study the sub-dominant oscillation channel ν_mu -> ν_e. OPERA is a large scale hybrid apparatus divided in two supermodules, each equipped with electronic detectors, an iron spectrometer and a highly segmented ~0.7 kton target section made of Emulsion Cloud Chamber (ECC) units. During my research work in the Bologna Lab. I have taken part to the set-up of the automatic scanning microscopes studying and tuning the scanning system performances and efficiencies with emulsions exposed to a test beam at CERN in 2007. Once the triggered bricks were distributed to the collaboration laboratories, my work was centered on the procedure used for the localization and the reconstruction of neutrino events.

Reconstruction and analysis of the NF-kB pathway interactome: a systems biology approach

Background. One of the phenomena observed in human aging is the progressive increase of a systemic inflammatory state, a condition referred to as “inflammaging”, negatively correlated with longevity. A prominent mediator of inflammation is the transcription factor NF-kB, that acts as key transcriptional regulator of many genes coding for pro-inflammatory cytokines. Many different signaling pathways activated by very diverse stimuli converge on NF-kB, resulting in a regulatory network characterized by high complexity. NF-kB signaling has been proposed to be responsible of inflammaging. Scope of this analysis is to provide a wider, systemic picture of such intricate signaling and interaction network: the NF-kB pathway interactome. Methods. The study has been carried out following a workflow for gathering information from literature as well as from several pathway and protein interactions databases, and for integrating and analyzing existing data and the relative reconstructed representations by using the available computational tools. Strong manual intervention has been necessarily used to integrate data from multiple sources into mathematically analyzable networks. The reconstruction of the NF-kB interactome pursued with this approach provides a starting point for a general view of the architecture and for a deeper analysis and understanding of this complex regulatory system. Results. A “core” and a “wider” NF-kB pathway interactome, consisting of 140 and 3146 proteins respectively, were reconstructed and analyzed through a mathematical, graph-theoretical approach. Among other interesting features, the topological characterization of the interactomes shows that a relevant number of interacting proteins are in turn products of genes that are controlled and regulated in their expression exactly by NF-kB transcription factors. These “feedback loops”, not always well-known, deserve deeper investigation since they may have a role in tuning the response and the output consequent to NF-kB pathway initiation, in regulating the intensity of the response, or its homeostasis and balance in order to make the functioning of such critical system more robust and reliable. This integrated view allows to shed light on the functional structure and on some of the crucial nodes of thet NF-kB transcription factors interactome. Conclusion. Framing structure and dynamics of the NF-kB interactome into a wider, systemic picture would be a significant step toward a better understanding of how NF-kB globally regulates diverse gene programs and phenotypes. This study represents a step towards a more complete and integrated view of the NF-kB signaling system.

Reconstruction of Etna recent flank eruptions for assessing diversion barrier project

The thesis contributed to the volcanic hazard assessment through the reconstruction of some historical flank eruptions of Etna in order to obtain quantitative data (volumes, effusion rates, etc.) for characterizing the recent effusive activity, quantifying the impact on the territory and defining mitigation actions for reducing the volcanic risk as for example containment barriers. The reconstruction was based on a quantitative approach using data extracted from aerial photographs and topographic maps. The approach allows to obtain the temporal evolution of the lava flow field and estimating the Time Average Discharge Rate (TADR) by dividing the volume emplaced over a given time interval for the corresponding duration. The analysis concerned the 2001, 1981 and 1928 Etna eruptions. The choice of these events is linked to their impact on inhabited areas. The results of the analysis showed an extraordinarily high effusion rate for the 1981 and 1928 eruptions (over 600 m^3/s), unusual for Etna eruptions. For the 1981 Etna eruption an eruptive model was proposed to explain the high discharge rate. The obtained TADRs were used as input data for simulations of the propagation of the lava flows for evaluating different scenarios of volcanic hazard and analyse different mitigation actions against lava flow invasion. It was experienced how numerical simulations could be adopted for evaluating the effectiveness of barrier construction and for supporting their optimal design. In particular, the gabions were proposed as an improvement for the construction of barriers with respect to the earthen barriers. The gabion barriers allow to create easily modular structures reducing the handled volumes and the intervention time. For evaluating operational constrain an experimental test was carried out to test the filling of the gabions with volcanic rock and evaluating their deformation during transport and placement.

Electron identification and reconstruction with the OPERA ECC bricks and search for muon neutrino to electron neutrino oscillations

The procedure for event location in OPERA ECC has been optimazed for penetrating particles while is less efficient for electrons. For this reason new procedure has been defined in order to recover event with an electromagnetic shower in its final state not located with the standard one. The new procedure include the standard procedure during which several electromagnetic shower hint has been defined by means of the available data. In case the event is not located, the presence of an electromagnetic shower hint trigger a dedicated procedure. The old and new location procedure has been then simulated in order to obtain the standard location efficiency and the possible gain due to the new one for the events with electromagnetic shower. Finally a Data-MC comparison has been performed for the 2008 and 2009 runs for what concern the NC in order to validate the Monte Carlo. Then the expected electron neutrino interactions for the 2008 and 2009 runs has been evaluated and compared with the available data.

Towards a Reconstruction of Thermal Properties of Light Nuclei from Fusion - Evaporation reactions

This thesis work has been developed in the framework of a new experimental campaign, proposed by the NUCL-EX Collaboration (INFN III Group), in order to progress in the understanding of the statistical properties of light nuclei, at excitation energies above particle emission threshold, by measuring exclusive data from fusion-evaporation reactions. The determination of the nuclear level density in the A~20 region, the understanding of the statistical behavior of light nuclei with excitation energies ~3 A.MeV, and the measurement of observables linked to the presence of cluster structures of nuclear excited levels are the main physics goals of this work. On the theory side, the contribution to this project given by this work lies in the development of a dedicated Monte-Carlo Hauser-Feshbach code for the evaporation of the compound nucleus. The experimental part of this thesis has consisted in the participation to the measurement 12C+12C at 95 MeV beam energy, at Laboratori Nazionali di Legnaro - INFN, using the GARFIELD+Ring Counter(RCo) set-up, from the beam-time request to the data taking, data reduction, detector calibrations and data analysis. Different results of the data analysis are presented in this thesis, together with a theoretical study of the system, performed with the new statistical decay code. As a result of this work, constraints on the nuclear level density at high excitation energy for light systems ranging from C up to Mg are given. Moreover, pre-equilibrium effects, tentatively interpreted as alpha-clustering effects, are put in evidence, both in the entrance channel of the reaction and in the dissipative dynamics on the path towards thermalisation.

Spectrum reconstruction from a scattering measurement using the adjoint Boltzmann transport equation for photons

Quality control of medical radiological systems is of fundamental importance, and requires efficient methods for accurately determine the X-ray source spectrum. Straightforward measurements of X-ray spectra in standard operating require the limitation of the high photon flux, and therefore the measure has to be performed in a laboratory. However, the optimal quality control requires frequent in situ measurements which can be only performed using a portable system. To reduce the photon flux by 3 magnitude orders an indirect technique based on the scattering of the X-ray source beam by a solid target is used. The measured spectrum presents a lack of information because of transport and detection effects. The solution is then unfolded by solving the matrix equation that represents formally the scattering problem. However, the algebraic system is ill-conditioned and, therefore, it is not possible to obtain a satisfactory solution. Special strategies are necessary to circumvent the ill-conditioning. Numerous attempts have been done to solve this problem by using purely mathematical methods. In this thesis, a more physical point of view is adopted. The proposed method uses both the forward and the adjoint solutions of the Boltzmann transport equation to generate a better conditioned linear algebraic system. The procedure has been tested first on numerical experiments, giving excellent results. Then, the method has been verified with experimental measurements performed at the Operational Unit of Health Physics of the University of Bologna. The reconstructed spectra have been compared with the ones obtained with straightforward measurements, showing very good agreement.

Oblique ionograms automatic scaling and eikonal based ray tracing

A method for automatic scaling of oblique ionograms has been introduced. This method also provides a rejection procedure for ionograms that are considered to lack sufficient information, depicting a very good success rate. Observing the Kp index of each autoscaled ionogram, can be noticed that the behavior of the autoscaling program does not depend on geomagnetic conditions. The comparison between the values of the MUF provided by the presented software and those obtained by an experienced operator indicate that the procedure developed for detecting the nose of oblique ionogram traces is sufficiently efficient and becomes much more efficient as the quality of the ionograms improves. These results demonstrate the program allows the real-time evaluation of MUF values associated with a particular radio link through an oblique radio sounding. The automatic recognition of a part of the trace allows determine for certain frequencies, the time taken by the radio wave to travel the path between the transmitter and receiver. The reconstruction of the ionogram traces, suggests the possibility of estimating the electron density between the transmitter and the receiver, from an oblique ionogram. The showed results have been obtained with a ray-tracing procedure based on the integration of the eikonal equation and using an analytical ionospheric model with free parameters. This indicates the possibility of applying an adaptive model and a ray-tracing algorithm to estimate the electron density in the ionosphere between the transmitter and the receiver An additional study has been conducted on a high quality ionospheric soundings data set and another algorithm has been designed for the conversion of an oblique ionogram into a vertical one, using Martyn's theorem. This allows a further analysis of oblique soundings, throw the use of the INGV Autoscala program for the automatic scaling of vertical ionograms.

The reconstruction of skeletal movement: the soft tissue artefact issue

In 3D human movement analysis performed using stereophotogrammetric systems and skin markers, bone pose can only be estimated in an indirect fashion. During a movement, soft tissue deformations make the markers move with respect to the underlying bone generating soft tissue artefact (STA). STA has devastating effects on bone pose estimation and its compensation remains an open question. The aim of this PhD thesis was to contribute to the solution of this crucial issue. Modelling STA using measurable trial-specific variables is a fundamental prerequisite for its removal from marker trajectories. Two STA model architectures are proposed. Initially, a thigh marker-level artefact model is presented. STA was modelled as a linear combination of joint angles involved in the movement. This model was calibrated using ex-vivo and in-vivo STA invasive measures. The considerable number of model parameters led to defining STA approximations. Three definitions were proposed to represent STA as a series of modes: individual marker displacements, marker-cluster geometrical transformations (MCGT), and skin envelope shape variations. Modes were selected using two criteria: one based on modal energy and another on the selection of modes chosen a priori. The MCGT allows to select either rigid or non-rigid STA components. It was also empirically demonstrated that only the rigid component affects joint kinematics, regardless of the non-rigid amplitude. Therefore, a model of thigh and shank STA rigid component at cluster-level was then defined. An acceptable trade-off between STA compensation effectiveness and number of parameters can be obtained, improving joint kinematics accuracy. The obtained results lead to two main potential applications: the proposed models can generate realistic STAs for simulation purposes to compare different skeletal kinematics estimators; and, more importantly, focusing only on the STA rigid component, the model attains a satisfactory STA reconstruction with less parameters, facilitating its incorporation in an pose estimator.

Scene Reconstruction And Understanding By RGB-D Sensors

This thesis investigates interactive scene reconstruction and understanding using RGB-D data only. Indeed, we believe that depth cameras will still be in the near future a cheap and low-power 3D sensing alternative suitable for mobile devices too. Therefore, our contributions build on top of state-of-the-art approaches to achieve advances in three main challenging scenarios, namely mobile mapping, large scale surface reconstruction and semantic modeling. First, we will describe an effective approach dealing with Simultaneous Localization And Mapping (SLAM) on platforms with limited resources, such as a tablet device. Unlike previous methods, dense reconstruction is achieved by reprojection of RGB-D frames, while local consistency is maintained by deploying relative bundle adjustment principles. We will show quantitative results comparing our technique to the state-of-the-art as well as detailed reconstruction of various environments ranging from rooms to small apartments. Then, we will address large scale surface modeling from depth maps exploiting parallel GPU computing. We will develop a real-time camera tracking method based on the popular KinectFusion system and an online surface alignment technique capable of counteracting drift errors and closing small loops. We will show very high quality meshes outperforming existing methods on publicly available datasets as well as on data recorded with our RGB-D camera even in complete darkness. Finally, we will move to our Semantic Bundle Adjustment framework to effectively combine object detection and SLAM in a unified system. Though the mathematical framework we will describe does not restrict to a particular sensing technology, in the experimental section we will refer, again, only to RGB-D sensing. We will discuss successful implementations of our algorithm showing the benefit of a joint object detection, camera tracking and environment mapping.