Efficient Image and Video Representations for Retrieval

Image (Video) retrieval is an interesting problem of retrieving images (videos) similar to the query. Images (Videos) are represented in an input (feature) space and similar images (videos) are obtained by finding nearest neighbors in the input representation space. Numerous input representations both in real valued and binary space have been proposed for conducting faster retrieval. In this thesis, we present techniques that obtain improved input representations for retrieval in both supervised and unsupervised settings for images and videos. Supervised retrieval is a well known problem of retrieving same class images of the query. We address the practical aspects of achieving faster retrieval with binary codes as input representations for the supervised setting in the first part, where binary codes are used as addresses into hash tables. In practice, using binary codes as addresses does not guarantee fast retrieval, as similar images are not mapped to the same binary code (address). We address this problem by presenting an efficient supervised hashing (binary encoding) method that aims to explicitly map all the images of the same class ideally to a unique binary code. We refer to the binary codes of the images as `Semantic Binary Codes' and the unique code for all same class images as `Class Binary Code'. We also propose a new class­ based Hamming metric that dramatically reduces the retrieval times for larger databases, where only hamming distance is computed to the class binary codes. We also propose a Deep semantic binary code model, by replacing the output layer of a popular convolutional Neural Network (AlexNet) with the class binary codes and show that the hashing functions learned in this way outperforms the state­ of ­the art, and at the same time provide fast retrieval times. In the second part, we also address the problem of supervised retrieval by taking into account the relationship between classes. For a given query image, we want to retrieve images that preserve the relative order i.e. we want to retrieve all same class images first and then, the related classes images before different class images. We learn such relationship aware binary codes by minimizing the similarity between inner product of the binary codes and the similarity between the classes. We calculate the similarity between classes using output embedding vectors, which are vector representations of classes. Our method deviates from the other supervised binary encoding schemes as it is the first to use output embeddings for learning hashing functions. We also introduce new performance metrics that take into account the related class retrieval results and show significant gains over the state­ of­ the art. High Dimensional descriptors like Fisher Vectors or Vector of Locally Aggregated Descriptors have shown to improve the performance of many computer vision applications including retrieval. In the third part, we will discuss an unsupervised technique for compressing high dimensional vectors into high dimensional binary codes, to reduce storage complexity. In this approach, we deviate from adopting traditional hyperplane hashing functions and instead learn hyperspherical hashing functions. The proposed method overcomes the computational challenges of directly applying the spherical hashing algorithm that is intractable for compressing high dimensional vectors. A practical hierarchical model that utilizes divide and conquer techniques using the Random Select and Adjust (RSA) procedure to compress such high dimensional vectors is presented. We show that our proposed high dimensional binary codes outperform the binary codes obtained using traditional hyperplane methods for higher compression ratios. In the last part of the thesis, we propose a retrieval based solution to the Zero shot event classification problem - a setting where no training videos are available for the event. To do this, we learn a generic set of concept detectors and represent both videos and query events in the concept space. We then compute similarity between the query event and the video in the concept space and videos similar to the query event are classified as the videos belonging to the event. We show that we significantly boost the performance using concept features from other modalities.

IMPROMPTU: FROM PUBLISHER'S IMPULSE TO AN ESTABLISHED SMALL­ SCALE GENRE IN PIANO LITERATURE

The word 'impromptu' began to appear in music literature in the early 19th century, specifically as title for a relatively short composition written for solo piano. The first impromptus appear to have been named so by the publishers. However, the composers themselves soon embraced the title to indicate, for the most part, fairly short character pieces. Impromptus do not follow any specific structural pattern, although many are cast in ternary form. The formal design ranges from strict compound ternary in the early impromptus to through-composed and variation forms. The peak of impromptu's popularity undoubtedly came during the middle and late19th century. However, they are still being composed today, albeit much less frequently. Although there have been many variants of impromptus in relation to formal design and harmonic language over the years, the essence of impromptu remains the same: it is still a short character piece with a general feeling of spontaneity. Overall, impromptus may be categorized into several different groups: some appear as part of a larger cycle, such as Dvorak's G minor Impromptu from his Piano Pieces, B. 110; many others use an element of an additional genre that enhances the character ofthe impromptu, such as Liszt's Valse-Impromptu and Antonio Bibalo's Tango Impromptu; yet another group consists of works based on opera themes, such as Liszt's Impromptu Brillant sur des themes de Rossini et Spontini and Czerny's Impromptus et variations sur Oberon, Op. 134. My recording project includes well-known impromptus, such as Schubert's Op. 142 and the four by Chopin, as well as lesser known works that have not been performed or recorded often. There are four impromptus that have been recorded here for the first time, including those written by Leopold Godowsky, Antonio Bibalo, Altin Volaj, and Nikolay Mazhara. I personally requested the two last named composers to contribute impromptus to this project. My selection represents works by twenty composers and reflects the different types of impromptus that have been encountered through almost three hundred years of the genre's existence, from approximately 1817 (VoriSek) to 2008 (Volaj and Mazhara).

A Video Game-Integrated Electromyography Biofeedback Device for Use in Physical Therapy

We present a novel system to be used in the rehabilitation of patients with forearm injuries. The system uses surface electromyography (sEMG) recordings from a wireless sleeve to control video games designed to provide engaging biofeedback to the user. An integrated hardware/software system uses a neural net to classify the signals from a user’s muscles as they perform one of a number of common forearm physical therapy exercises. These classifications are used as input for a suite of video games that have been custom-designed to hold the patient’s attention and decrease the risk of noncompliance with the physical therapy regimen necessary to regain full function in the injured limb. The data is transmitted wirelessly from the on-sleeve board to a laptop computer using a custom-designed signal-processing algorithm that filters and compresses the data prior to transmission. We believe that this system has the potential to significantly improve the patient experience and efficacy of physical therapy using biofeedback that leverages the compelling nature of video games.