Algorithms for super-resolution of images based on Sparse Representation and Manifolds

lmage super-resolution is defined as a class of techniques that enhance the spatial resolution of images. Super-resolution methods can be subdivided in single and multi image methods. This thesis focuses on developing algorithms based on mathematical theories for single image super­ resolution problems. lndeed, in arder to estimate an output image, we adopta mixed approach: i.e., we use both a dictionary of patches with sparsity constraints (typical of learning-based methods) and regularization terms (typical of reconstruction-based methods). Although the existing methods already per- form well, they do not take into account the geometry of the data to: regularize the solution, cluster data samples (samples are often clustered using algorithms with the Euclidean distance as a dissimilarity metric), learn dictionaries (they are often learned using PCA or K-SVD). Thus, state-of-the-art methods still suffer from shortcomings. In this work, we proposed three new methods to overcome these deficiencies. First, we developed SE-ASDS (a structure tensor based regularization term) in arder to improve the sharpness of edges. SE-ASDS achieves much better results than many state-of-the- art algorithms. Then, we proposed AGNN and GOC algorithms for determining a local subset of training samples from which a good local model can be computed for recon- structing a given input test sample, where we take into account the underlying geometry of the data. AGNN and GOC methods outperform spectral clustering, soft clustering, and geodesic distance based subset selection in most settings. Next, we proposed aSOB strategy which takes into account the geometry of the data and the dictionary size. The aSOB strategy outperforms both PCA and PGA methods. Finally, we combine all our methods in a unique algorithm, named G2SR. Our proposed G2SR algorithm shows better visual and quantitative results when compared to the results of state-of-the-art methods.

High throughput 3D super-resolution microscopy reveals Caulobacter crescentus in vivo Z-ring organization.

We created a high-throughput modality of photoactivated localization microscopy (PALM) that enables automated 3D PALM imaging of hundreds of synchronized bacteria during all stages of the cell cycle. We used high-throughput PALM to investigate the nanoscale organization of the bacterial cell division protein FtsZ in live Caulobacter crescentus. We observed that FtsZ predominantly localizes as a patchy midcell band, and only rarely as a continuous ring, supporting a model of "Z-ring" organization whereby FtsZ protofilaments are randomly distributed within the band and interact only weakly. We found evidence for a previously unidentified period of rapid ring contraction in the final stages of the cell cycle. We also found that DNA damage resulted in production of high-density continuous Z-rings, which may obstruct cytokinesis. Our results provide a detailed quantitative picture of in vivo Z-ring organization.

Super-resolution of remotely sensed images with variable-pixel linear reconstruction

This paper describes the development and applications of a super-resolution method, known as Super-Resolution Variable-Pixel Linear Reconstruction. The algorithm works combining different lower resolution images in order to obtain, as a result, a higher resolution image. We show that it can make significant spatial resolution improvements to satellite images of the Earth¿s surface allowing recognition of objects with size approaching the limiting spatial resolution of the lower resolution images. The algorithm is based on the Variable-Pixel Linear Reconstruction algorithm developed by Fruchter and Hook, a well-known method in astronomy but never used for Earth remote sensing purposes. The algorithm preserves photometry, can weight input images according to the statistical significance of each pixel, and removes the effect of geometric distortion on both image shape and photometry. In this paper, we describe its development for remote sensing purposes, show the usefulness of the algorithm working with images as different to the astronomical images as the remote sensing ones, and show applications to: 1) a set of simulated multispectral images obtained from a real Quickbird image; and 2) a set of multispectral real Landsat Enhanced Thematic Mapper Plus (ETM+) images. These examples show that the algorithm provides a substantial improvement in limiting spatial resolution for both simulated and real data sets without significantly altering the multispectral content of the input low-resolution images, without amplifying the noise, and with very few artifacts.

An efficient total variation algorithm for super-resolution in fetal brain MRI with adaptive regularization.

Although fetal anatomy can be adequately viewed in new multi-slice MR images, many critical limitations remain for quantitative data analysis. To this end, several research groups have recently developed advanced image processing methods, often denoted by super-resolution (SR) techniques, to reconstruct from a set of clinical low-resolution (LR) images, a high-resolution (HR) motion-free volume. It is usually modeled as an inverse problem where the regularization term plays a central role in the reconstruction quality. Literature has been quite attracted by Total Variation energies because of their ability in edge preserving but only standard explicit steepest gradient techniques have been applied for optimization. In a preliminary work, it has been shown that novel fast convex optimization techniques could be successfully applied to design an efficient Total Variation optimization algorithm for the super-resolution problem. In this work, two major contributions are presented. Firstly, we will briefly review the Bayesian and Variational dual formulations of current state-of-the-art methods dedicated to fetal MRI reconstruction. Secondly, we present an extensive quantitative evaluation of our SR algorithm previously introduced on both simulated fetal and real clinical data (with both normal and pathological subjects). Specifically, we study the robustness of regularization terms in front of residual registration errors and we also present a novel strategy for automatically select the weight of the regularization as regards the data fidelity term. Our results show that our TV implementation is highly robust in front of motion artifacts and that it offers the best trade-off between speed and accuracy for fetal MRI recovery as in comparison with state-of-the art methods.

Combining Near-Field and Far-Field Microscopy: A new method for nanoscale super-resolution imaging

Lähikenttä- ja kaukokenttämikroskopian yhdistäminen: Uusi korkearesoluutioinen menetelmä nanokuvantamiseen. Osteoporoosi on sairaus, jossa luun uudistumisprosessi ei ole enää tasapainossa. Uuden luun muodostuminen on hitaampaa johtuen osteoblastien laskeneesta aktiivisuudesta. Yksi keino estää osteoporoosin syntyä on estää osteoklastien sitoutuminen luun pinnalle, jolloin ne eivät aloita luun syömisprosessia. Tämän Pro gradu -tutkielman tarkoituksena on luoda uusi työkalu osteoklastien sitoutumisen tutkimiseen samanaikaisesti fluoresenssi- ja atomivoimamikroskoopilla. Tätä tarkoitusta varten yhdistettiin atomivoimamikroskooppi sekä STED mikroskooppi. Kirjallisuuskatsauksessa käydään läpi yksityiskohtaisesti molempien mikroskooppitekniikoiden teoriat. Kokeellisessa osiossa esitetään käytetyt metodit ja alustavat tulokset uudella systeemillä. Lisäksi keskustellaan lyhyesti kuvan analysoinnista ImageJohjelmalla. Konfokaalisen fluoresenssimikroskoopin ja atomivoimamikroskoopin yhdistelmä on keksitty jo aikaisemmin, mutta tavallisen konfokaalimikroskoopin erottelukyvyn raja on noin 200 nanometriä johtuen valon diffraktioluonteesta. Yksityiskohdat eivät erotu, jos ne ovat pienempiä kuin puolet käytettävästä aallonpituudesta. STED mikroskooppi mahdollistaa fluoresenssikuvien taltioimisen solunsisäisistä prosesseista 50 nanometrin lateraalisella erotuskyvyllä ja atomivoimamikroskooppi antaa topografista tietoa näytteestä nanometrien erotuskyvyllä. Biologisia näytteitä kuvannettaessa atomivoimamikroskoopin erotuskyky kuitenkin huononee ja yleensä saavutetaan 30-50 nanometrin erotuskyky. Kuvien kerrostaminen vaatii vertauspisteitä ja tätä varten käytettiin atomivoimamikroskoopin kärjen tunnistamista ja referenssipartikkeleita. Kuva-analysointi suoritettiin ImageJ-kuvankäsittelyohjelmalla. Tuloksista nähdään, että referenssipartikkelit ovat hyviä, mutta niiden sijoittaminen tarkasti tietylle kohdealueelle on hankalaa nanoskaalassa. Tästä johtuen kärjen havaitseminen fluoresenssikuvassa on parempi metodi. Atomivoimamikroskoopin kärki voidaan päällystää fluoresoivalla aineella, mutta tämä lisää kärjen aiheuttamaa konvoluutiota mittausdataan. Myös valon takaisinsirontaa kärjestä voidaan tutkia, jolloin konvoluutio ei lisäänny. Ensimmäisten kuvien kerrostamisessa käytettiin hyväksi fluoresoivalla aineella päällystettyä kärkeä ja lopputuloksessa oli vain 50 nanometrin yhteensopimattomuus fluoresenssi- ja topografiakuvan kanssa. STED mikroskoopin avulla nähdään leimattujen proteiinien tarkat sijainnit tiettynä ajankohtana elävän solun sisällä. Samaan aikaan pystytään kuvantamaan solun fyysisiä muotoja tai mitata adheesiovoimia atomivoimamikroskoopilla. Lisäksi voidaan käyttää funktinalisoitua kärkeä, jolla voidaan laukaista signalointitapahtumia solun ja soluväliaineen välillä. Sitoutuminen soluväliaineeseen voidaan rekisteröidä samoin kuin adheesiomediaattorien sijainnit sitoutumisalueella. Nämä dynaamiset havainnot tuottavat uutta informaatiota solun signaloinnista, kun osteoklasti kiinnittyy luun pintaan. Tämä teknologia tarjoaa uuden näkökulman monimutkaisiin signalointiprosesseihin nanoskaalassa ja tulee ratkaisemaan lukemattoman määrän biologisia ongelmia.

Development of 3D super-resolution tomographic STED microscopy and its application to studies on bone resorption

In this doctoral thesis, a tomographic STED microscopy technique for 3D super-resolution imaging was developed and utilized to observebone remodeling processes. To improve upon existing methods, wehave used a tomographic approach using a commercially available stimulated emission depletion (STED) microscope. A certain region of interest (ROI) was observed at two oblique angles: one at a standard inverted configuration from below (bottom view) and another from the side (side view) via a micro-mirror positioned close to the ROI. The two viewing angles were reconstructed into a final tomogram. The technique, named as tomographic STED microscopy, was able to achieve an axial resolution of approximately 70 nm on microtubule structures in a fixed biological specimen. High resolution imaging of osteoclasts (OCs) that are actively resorbing bone was achieved by creating an optically transparent coating on a microscope coverglass that imitates a fractured bone surface. 2D super-resolution STED microscopy on the bone layer showed approximately 60 nm of lateral resolution on a resorption associated organelle allowing these structures to be imaged with super-resolution microscopy for the first time. The developed tomographic STED microscopy technique was further applied to study resorption mechanisms of OCs cultured on the bone coating. The technique revealed actin cytoskeleton with specific structures, comet-tails, some of which were facing upwards and some others were facing downwards. This, in our opinion, indicated that during bone resorption, an involvement of the actin cytoskeleton in vesicular exocytosis and endocytosis is present. The application of tomographic STED microscopy in bone biology demonstrated that 3D super-resolution techniques can provide new insights into biological 3D nano-structures that are beyond the diffraction-limit when the optical constraints of super-resolution imaging are carefully taken into account.

Bioimage informatics in STED super-resolution microscopy

Optical microscopy is living its renaissance. The diffraction limit, although still physically true, plays a minor role in the achievable resolution in far-field fluorescence microscopy. Super-resolution techniques enable fluorescence microscopy at nearly molecular resolution. Modern (super-resolution) microscopy methods rely strongly on software. Software tools are needed all the way from data acquisition, data storage, image reconstruction, restoration and alignment, to quantitative image analysis and image visualization. These tools play a key role in all aspects of microscopy today – and their importance in the coming years is certainly going to increase, when microscopy little-by-little transitions from single cells into more complex and even living model systems. In this thesis, a series of bioimage informatics software tools are introduced for STED super-resolution microscopy. Tomographic reconstruction software, coupled with a novel image acquisition method STED< is shown to enable axial (3D) super-resolution imaging in a standard 2D-STED microscope. Software tools are introduced for STED super-resolution correlative imaging with transmission electron microscopes or atomic force microscopes. A novel method for automatically ranking image quality within microscope image datasets is introduced, and it is utilized to for example select the best images in a STED microscope image dataset.

An Improved Color Video Super-Resolution Using Kernel Regression and Fuzzy Enhancement

An improved color video super-resolution technique using kernel regression and fuzzy enhancement is presented in this paper. A high resolution frame is computed from a set of low resolution video frames by kernel regression using an adaptive Gaussian kernel. A fuzzy smoothing filter is proposed to enhance the regression output. The proposed technique is a low cost software solution to resolution enhancement of color video in multimedia applications. The performance of the proposed technique is evaluated using several color videos and it is found to be better than other techniques in producing high quality high resolution color videos

Single Frame Image Super Resolution Using Learned Directionlets

In this paper, a new directionally adaptive, learning based, single image super resolution method using multiple direction wavelet transform, called Directionlets is presented. This method uses directionlets to effectively capture directional features and to extract edge information along different directions of a set of available high resolution images .This information is used as the training set for super resolving a low resolution input image and the Directionlet coefficients at finer scales of its high-resolution image are learned locally from this training set and the inverse Directionlet transform recovers the super-resolved high resolution image. The simulation results showed that the proposed approach outperforms standard interpolation techniques like Cubic spline interpolation as well as standard Wavelet-based learning, both visually and in terms of the mean squared error (mse) values. This method gives good result with aliased images also.

Development of Directionally Adaptive Techniques for Single Image Super-Resolution

Super Resolution problem is an inverse problem and refers to the process of producing a High resolution (HR) image, making use of one or more Low Resolution (LR) observations. It includes up sampling the image, thereby, increasing the maximum spatial frequency and removing degradations that arise during the image capture namely aliasing and blurring. The work presented in this thesis is based on learning based single image super-resolution. In learning based super-resolution algorithms, a training set or database of available HR images are used to construct the HR image of an image captured using a LR camera. In the training set, images are stored as patches or coefficients of feature representations like wavelet transform, DCT, etc. Single frame image super-resolution can be used in applications where database of HR images are available. The advantage of this method is that by skilfully creating a database of suitable training images, one can improve the quality of the super-resolved image. A new super resolution method based on wavelet transform is developed and it is better than conventional wavelet transform based methods and standard interpolation methods. Super-resolution techniques based on skewed anisotropic transform called directionlet transform are developed to convert a low resolution image which is of small size into a high resolution image of large size. Super-resolution algorithm not only increases the size, but also reduces the degradations occurred during the process of capturing image. This method outperforms the standard interpolation methods and the wavelet methods, both visually and in terms of SNR values. Artifacts like aliasing and ringing effects are also eliminated in this method. The super-resolution methods are implemented using, both critically sampled and over sampled directionlets. The conventional directionlet transform is computationally complex. Hence lifting scheme is used for implementation of directionlets. The new single image super-resolution method based on lifting scheme reduces computational complexity and thereby reduces computation time. The quality of the super resolved image depends on the type of wavelet basis used. A study is conducted to find the effect of different wavelets on the single image super-resolution method. Finally this new method implemented on grey images is extended to colour images and noisy images

Super-resolution Properties of the Maxwell Fish-Eyes

In this paper we present an analysis that shows the Maxwell Fish Eye (MFE) only has super-resolution property for some particular frequencies (for other frequencies, the MFE behaves as conventional imaging lens). These frequencies are directly connected with the Schumann resonance frequencies of spherical symmetric systems. The analysis have been done using a thin spherical waveguide (two concentric spheres with constant index between them), which is a dual form of the MFE (the electrical fields in the MFE can be mapped into the radial electrical fields in the spherical waveguide). In the spherical waveguide the fields are guided inside the space between the concentric spheres. A microwave circuit comprising three elements: the spherical waveguide, the source and the receiver (two coaxial cables) is designed in COMSOL. The super-resolution is demonstrated by calculation of Scaterring (S) parameters for different position of the coaxial cables and different frequencies of the input signal.

Super-Resolution in Respiratory Synchronized Positron Emission Tomography

Respiratory motion is a major source of reduced quality in positron emission tomography (PET). In order to minimize its effects, the use of respiratory synchronized acquisitions, leading to gated frames, has been suggested. Such frames, however, are of low signal-to-noise ratio (SNR) as they contain reduced statistics. Super-resolution (SR) techniques make use of the motion in a sequence of images in order to improve their quality. They aim at enhancing a low-resolution image belonging to a sequence of images representing different views of the same scene. In this work, a maximum a posteriori (MAP) super-resolution algorithm has been implemented and applied to respiratory gated PET images for motion compensation. An edge preserving Huber regularization term was used to ensure convergence. Motion fields were recovered using a B-spline based elastic registration algorithm. The performance of the SR algorithm was evaluated through the use of both simulated and clinical datasets by assessing image SNR, as well as the contrast, position and extent of the different lesions. Results were compared to summing the registered synchronized frames on both simulated and clinical datasets. The super-resolution image had higher SNR (by a factor of over 4 on average) and lesion contrast (by a factor of 2) than the single respiratory synchronized frame using the same reconstruction matrix size. In comparison to the motion corrected or the motion free images a similar SNR was obtained, while improvements of up to 20% in the recovered lesion size and contrast were measured. Finally, the recovered lesion locations on the SR images were systematically closer to the true simulated lesion positions. These observations concerning the SNR, lesion contrast and size were confirmed on two clinical datasets included in the study. In conclusion, the use of SR techniques applied to respiratory motion synchronized images lead to motion compensation combined with improved image SNR and contrast, without any increase in the overall acquisition times.

Super-resolution for a point source using positive refraction

Leonhardt demonstrated (2009) that the 2D Maxwell Fish Eye lens (MFE) can focus perfectly 2D Helmholtz waves of arbitrary frequency, i.e., it can transport perfectly an outward (monopole) 2D Helmholtz wave field, generated by a point source, towards a receptor called "perfect drain" (PD) located at the corresponding MFE image point. The PD has the property of absorbing the complete radiation without radiation or scattering and it has been claimed as necessary to obtain super-resolution (SR) in the MFE. However, a prototype using a "drain" different from the PD has shown λ/5 resolution for microwave frequencies (Ma et al, 2010). Recently, the SR properties of a device equivalent to the MFE, called the Spherical Geodesic Waveguide (SGW) (Miñano et al, 2012) have been analyzed. The reported results show resolution up to λ /3000, for the SGW loaded with the perfect drain, and up to λ /500 f for the SGW without perfect drain. The perfect drain was realized as a coaxial probe loaded with properly calculated impedance. The SGW provides SR only in a narrow band of frequencies close to the resonance Schumann frequencies. Here we analyze the SGW loaded with a small "perfect drain region" (González et al, 2011). This drain is designed as a region made of a material with complex permittivity. The comparative results show that there is no significant difference in the SR properties for both perfect drain designs.

COMSOL Multyphisics Super Resolution Analysis of a Spherical Geodesic Waveguide Suitable for Manufacturing

Recently it has been proved theoretically (Miñano et al, 2011) that the super-resolution up to ?/500 can be achieved using an ideal metallic Spherical Geodesic Waveguide (SGW). This SGW is a theoretical design, in which the conductive walls are considered to be lossless conductors with zero thickness. In this paper, we study some key parameters that might influence the super resolution properties reported in (Miñano et al, 2011), such as losses, metal type, the thickness of conductive walls and the deformation from perfect sphere. We implement a realistic SGW in COMSOL multiphysics and analyze its super-resolution properties. The realistic model is designed in accordance with the manufacturing requirements and technological limitations.