Algorithms for on-line image registration from multiple views

Images of a scene, static or dynamic, are generally acquired at different epochs from different viewpoints. They potentially gather information about the whole scene and its relative motion with respect to the acquisition device. Data from different (in the spatial or temporal domain) visual sources can be fused together to provide a unique consistent representation of the whole scene, even recovering the third dimension, permitting a more complete understanding of the scene content. Moreover, the pose of the acquisition device can be achieved by estimating the relative motion parameters linking different views, thus providing localization information for automatic guidance purposes. Image registration is based on the use of pattern recognition techniques to match among corresponding parts of different views of the acquired scene. Depending on hypotheses or prior information about the sensor model, the motion model and/or the scene model, this information can be used to estimate global or local geometrical mapping functions between different images or different parts of them. These mapping functions contain relative motion parameters between the scene and the sensor(s) and can be used to integrate accordingly informations coming from the different sources to build a wider or even augmented representation of the scene. Accordingly, for their scene reconstruction and pose estimation capabilities, nowadays image registration techniques from multiple views are increasingly stirring up the interest of the scientific and industrial community. Depending on the applicative domain, accuracy, robustness, and computational payload of the algorithms represent important issues to be addressed and generally a trade-off among them has to be reached. Moreover, on-line performance is desirable in order to guarantee the direct interaction of the vision device with human actors or control systems. This thesis follows a general research approach to cope with these issues, almost independently from the scene content, under the constraint of rigid motions. This approach has been motivated by the portability to very different domains as a very desirable property to achieve. A general image registration approach suitable for on-line applications has been devised and assessed through two challenging case studies in different applicative domains. The first case study regards scene reconstruction through on-line mosaicing of optical microscopy cell images acquired with non automated equipment, while moving manually the microscope holder. By registering the images the field of view of the microscope can be widened, preserving the resolution while reconstructing the whole cell culture and permitting the microscopist to interactively explore the cell culture. In the second case study, the registration of terrestrial satellite images acquired by a camera integral with the satellite is utilized to estimate its three-dimensional orientation from visual data, for automatic guidance purposes. Critical aspects of these applications are emphasized and the choices adopted are motivated accordingly. Results are discussed in view of promising future developments.

Individuazione di punti salienti in dati 3D mediante rappresentazioni strutturate

Questa tesi si inserisce nel filone di ricerca dell'elaborazione di dati 3D, e in particolare nella 3D Object Recognition, e delinea in primo luogo una panoramica sulle principali rappresentazioni strutturate di dati 3D, le quali rappresentano una prerogativa necessaria per implementare in modo efficiente algoritmi di processing di dati 3D, per poi presentare un nuovo algoritmo di 3D Keypoint Detection che è stato sviluppato e proposto dal Computer Vision Laboratory dell'Università di Bologna presso il quale ho effettuato la mia attività di tesi.

Methodological improvement of 3D fluoroscopic analysis for the robust quantification of 3D kinematics of human joints

3D video-fluoroscopy is an accurate but cumbersome technique to estimate natural or prosthetic human joint kinematics. This dissertation proposes innovative methodologies to improve the 3D fluoroscopic analysis reliability and usability. Being based on direct radiographic imaging of the joint, and avoiding soft tissue artefact that limits the accuracy of skin marker based techniques, the fluoroscopic analysis has a potential accuracy of the order of mm/deg or better. It can provide fundamental informations for clinical and methodological applications, but, notwithstanding the number of methodological protocols proposed in the literature, time consuming user interaction is exploited to obtain consistent results. The user-dependency prevented a reliable quantification of the actual accuracy and precision of the methods, and, consequently, slowed down the translation to the clinical practice. The objective of the present work was to speed up this process introducing methodological improvements in the analysis. In the thesis, the fluoroscopic analysis was characterized in depth, in order to evaluate its pros and cons, and to provide reliable solutions to overcome its limitations. To this aim, an analytical approach was followed. The major sources of error were isolated with in-silico preliminary studies as: (a) geometric distortion and calibration errors, (b) 2D images and 3D models resolutions, (c) incorrect contour extraction, (d) bone model symmetries, (e) optimization algorithm limitations, (f) user errors. The effect of each criticality was quantified, and verified with an in-vivo preliminary study on the elbow joint. The dominant source of error was identified in the limited extent of the convergence domain for the local optimization algorithms, which forced the user to manually specify the starting pose for the estimating process. To solve this problem, two different approaches were followed: to increase the optimal pose convergence basin, the local approach used sequential alignments of the 6 degrees of freedom in order of sensitivity, or a geometrical feature-based estimation of the initial conditions for the optimization; the global approach used an unsupervised memetic algorithm to optimally explore the search domain. The performances of the technique were evaluated with a series of in-silico studies and validated in-vitro with a phantom based comparison with a radiostereometric gold-standard. The accuracy of the method is joint-dependent, and for the intact knee joint, the new unsupervised algorithm guaranteed a maximum error lower than 0.5 mm for in-plane translations, 10 mm for out-of-plane translation, and of 3 deg for rotations in a mono-planar setup; and lower than 0.5 mm for translations and 1 deg for rotations in a bi-planar setups. The bi-planar setup is best suited when accurate results are needed, such as for methodological research studies. The mono-planar analysis may be enough for clinical application when the analysis time and cost may be an issue. A further reduction of the user interaction was obtained for prosthetic joints kinematics. A mixed region-growing and level-set segmentation method was proposed and halved the analysis time, delegating the computational burden to the machine. In-silico and in-vivo studies demonstrated that the reliability of the new semiautomatic method was comparable to a user defined manual gold-standard. The improved fluoroscopic analysis was finally applied to a first in-vivo methodological study on the foot kinematics. Preliminary evaluations showed that the presented methodology represents a feasible gold-standard for the validation of skin marker based foot kinematics protocols.

Spectral-Element and Adjoint 3D Full-Wave Tomography for the Lithosphere of Central Italy

The primary objective of this thesis is to obtain a better understanding of the 3D velocity structure of the lithosphere in central Italy. To this end, I adopted the Spectral-Element Method to perform accurate numerical simulations of the complex wavefields generated by the 2009 Mw 6.3 L’Aquila event and by its foreshocks and aftershocks together with some additional events within our target region. For the mainshock, the source was represented by a finite fault and different models for central Italy, both 1D and 3D, were tested. Surface topography, attenuation and Moho discontinuity were also accounted for. Three-component synthetic waveforms were compared to the corresponding recorded data. The results of these analyses show that 3D models, including all the known structural heterogeneities in the region, are essential to accurately reproduce waveform propagation. They allow to capture features of the seismograms, mainly related to topography or to low wavespeed areas, and, combined with a finite fault model, result into a favorable match between data and synthetics for frequencies up to ~0.5 Hz. We also obtained peak ground velocity maps, that provide valuable information for seismic hazard assessment. The remaining differences between data and synthetics led us to take advantage of SEM combined with an adjoint method to iteratively improve the available 3D structure model for central Italy. A total of 63 events and 52 stations in the region were considered. We performed five iterations of the tomographic inversion, by calculating the misfit function gradient - necessary for the model update - from adjoint sensitivity kernels, constructed using only two simulations for each event. Our last updated model features a reduced traveltime misfit function and improved agreement between data and synthetics, although further iterations, as well as refined source solutions, are necessary to obtain a new reference 3D model for central Italy tomography.

La Ricostruzione 3D di Bologna nel XVI Secolo

Il lavoro presentato ha come oggetto la ricostruzione tridimensionale della città di Bologna nella sua fase rinascimentale. Tale lavoro vuole fornire un modello 3D delle architetture e degli spazi urbani utilizzabile sia per scopi di ricerca nell’ambito della storia delle città sia per un uso didattico-divulgativo nel settore del turismo culturale. La base del lavoro è una fonte iconografica di grande importanza: l’affresco raffigurante Bologna risalente al 1575 e situato in Vaticano; questa è una veduta a volo d’uccello di grandi dimensioni dell’intero tessuto urbano bolognese all’interno della terza cerchia di mura. In esso sono rappresentate in maniera particolareggiata le architetture civili e ecclesiastiche, gli spazi ortivi e cortilivi interni agli isolati e alcune importanti strutture urbane presenti in città alla fine del Cinquecento, come l’area portuale e i canali interni alla città, oggi non più visibili. La ricostruzione tridimensionale è stata realizzata tramite Blender, software per la modellazione 3D opensource, attraverso le fasi di modellazione, texturing e creazione materiali (mediante campionamento delle principali cromie presenti nell’affresco), illuminazione e animazione. Una parte della modellazione è stata poi testata all’interno di un GIS per verificare l’utilizzo delle geometrie 3D come elementi collegabili ad altre fonti storiche relative allo sviluppo urbano e quindi sfruttabili per la ricerca storica. Grande attenzione infine è stata data all’uso dei modelli virtuali a scopo didattico-divulgativo e per il turismo culturale. La modellazione è stata utilizzata all’interno di un motore grafico 3D per costruire un ambiente virtuale interattivo nel quale un utente anche non esperto possa muoversi per esplorare gli spazi urbani della Bologna del Cinquecento. In ultimo è stato impostato lo sviluppo di un’applicazione per sistemi mobile (Iphone e Ipad) al fine di fornire uno strumento per la conoscenza della città storica in mobilità, attraverso la comparazione dello stato attuale con quello ricostruito virtualmente.

3D-Cryo-Elektronenmikroskopie des 4x6-Hämocyanins der Vogelspinne Eurypelma californicum unter Oxy- und Deoxybedingungen

Arthropodenhämocyanine und Molluskenhämocyanine, die extrazellulären Atmungsproteine der Arthropoden und Mollusken, unterscheiden sich grundsätzlich im Aufbau, besitzen aber ähnliche aktive Zentren, welche in ihrer oxydierten Form für die Blaufärbung der Hämocyanine verantwortlich sind. Sauerstoff wird im Bindungszentrum zwischen zwei, von sechs Histidinen ligandierten, Kupfer(I)Ionen gebunden. Arthropodenhämocyanine bauen sich artspezifisch aus 1, 2, 4, 6, oder 8 Hexameren mit D3-Symmetrie auf. Die Untereinheiten von je ca. 75 kDa falten sich in drei Domänen unterschiedlicher Funktionen. Der komplexe, hierarchische Zusammenbau der Arthropodenhämocyanine hängt von der Heterogenität der Untereinheiten ab. Die 7 verschieden Sequenzen des 4x6-Hämocyanins von Eurypelma californicum (EcHc) sind biochemisch in der Quartärstruktur lokalisiert. Bislang fehlte noch ein unabhängig erstelltes 3D-Modell der geometrischen Gesamtstruktur welche die hexamere und monomere Topographie eindeutig zeigt. Dessen Erstellung war Gegenstand dieser Arbeit, in Verbindung mit der Zielsetzung, die 3D-Rekonstruktion in den beiden extremen physiologischen Zuständen, mit und ohne gebundenen Sauerstoff, zu erzeugen. Dazu wurden in einer eigens entwickelten Atmosphären-Präparationskammer die Proteine in Lösung schockgefrorenen und mittels Cryo-3D-Elektronenmikroskopie gemessen. Aus den daraus gewonnen Projektionsbildern ließen sich mit der ”Single Particle Analyse“ die 3D-Informationen zurückberechnen. Die 3D-Rekonstruktionen wurden mit der publizierten Röntgenkristallstruktur des hexameren Referenz-Hämocyanins der Languste Panulirus interruptus verifiziert. Die Rekonstruktionen erlaubten die eindeutige Messung diverser in der Literatur diskutierter Parameter der Architektur des 4x6-EcHc und darüber hinaus weiterer geometrischer Parameter, welche hier erstmals veröffentlicht werden. SAXS-Daten sagen extreme Translationen und Rotationen von Teilquartärstrukturen zwischen oxy- und deoxy-EcHc voraus, was von den 3D-Rekonstruktionen der beiden Zustände nicht bestätigt werden konnte: Die 16 Å Rekonstruktion der Deoxyform weicht geometrisch nicht von der 21 Å Rekonstruktion der Oxyform ab. Die Einpassung der publizierten Röntgenstruktur der Untereinheit II des Hämocyanin des Pfeilschwanzkrebses Limulus polyphemus in die Rekonstruktionen unterstützt eine auf der hexameren Hierarchieebene lokalisierte Dynamik der Oxygenierung. Mittels Einpassung modellierter molekularer Strukturen der EcHc-Sequenzen konnte eine erste Vermutung zur Lokalisation der beiden zentralen Linker-Untereinheiten b und c des 4x6-Moleküls gemacht werden: Demnach würde Untereinheit b in den exponierten Hexameren des Moleküls liegen. Aussagen über die Quartärstrukturbindungen auf molekularer Ebene aufgrund der Einpassung modellierter molekularer Daten in die Rekonstruktionen sind als spekulativ einzustufen: a) Die Auflösung der Rekonstruktion ist verbesserungswürdig. b) Es gibt keine adäquate Vorlage für eine verlässliche Strukturvorhersage; die verschiedenen EcHc-Sequenzen liegen nur als Modellierung vor. c) Es wäre eine flexible Einpassung notwendig, um Ungenauigkeiten in den modellierten Strukturen durch Sekundärstrukturanpassung zu minimieren.