Distributed Web Interface for Real-Time Spatial Audio Reproduction System

SSR es el acrónimo de SoundScape Renderer (tool for real-time spatial audio reproduction providing a variety of rendering algorithms), es un programa escrito en su mayoría en C++. El programa permite al usuario escuchar tanto sonidos grabados con anterioridad como sonidos en directo. El sonido o los sonidos se oirán, desde el punto de vista del oyente, como si el sonido se produjese en el punto que el programa decida, lo interesante de este proyecto es que el sonido podrá cambiar de lugar, moverse, etc. Todo en tiempo real. Esto se consigue sin modificar el sonido al grabarlo pero sí al emitirlo, el programa calcula las variaciones necesarias para que al emitir el sonido al oyente le llegue como si el sonido realmente se generase en un punto del espacio o lo más parecido posible. La sensación de movimiento no deja de ser el punto anterior cambiando de lugar. La idea era crear una aplicación web basada en Canvas de HTML5 que se comunicará con esta interfaz de usuario remota. Así se solucionarían todos los problemas de compatibilidad ya que cualquier dispositivo con posibilidad de visualizar páginas web podría correr una aplicación basada en estándares web, por ejemplo un sistema con Windows o un móvil con navegador. El protocolo debía de ser WebSocket porque es un protocolo HTML5 y ofrece las “garantías” de latencia que una aplicación con necesidades de información en tiempo real requiere. Nos permite una comunicación full-dúplex asíncrona sin mucho payload que es justo lo que se venía a evitar al no usar polling normal de HTML. El problema que surgió fue que la interfaz de usuario de red que tenía el programa no era compatible con WebSocket debido a un handshacking inicial y obligatorio que realiza el protocolo, por lo que se necesitaba otra interfaz de red. Se decidió entonces cambiar a JSON como formato para el intercambio de mensajes. Al final el proyecto comprende no sólo la aplicación web basada en Canvas sino también un servidor funcional y la definición de una nueva interfaz de usuario de red con su protocolo añadido. ABSTRACT. This project aims to become a part of the SSR tool to extend its capabilities in the field of the access. SSR is an acronym for SoundScape Renderer, is a program mostly written in C++ that allows you to hear already recorded or live sound with a variety of sound equipment as if the sound came from a desired place in the space. Like the web-page of the SSR says surely better explained: “The SoundScape Renderer (SSR) is a tool for real-time spatial audio reproduction providing a variety of rendering algorithms.” The application can be used with a graphical interface written in Qt but has also a network interface for external applications to use it. This network interface communicates using XML messages. A good example of it is the Android client. This Android client is already working. In order to use the application should be run it by loading an audio source and the wanted environment so that the renderer knows what to do. In that moment the server binds and anyone can use the network interface. Since the network interface is documented everyone can make an application to interact with this network interface. So the application can have as many user interfaces as wanted. The part that is developed in this project has nothing to do neither with audio rendering nor even with the reproduction of the spatial audio. The part that is developed here is about the interface used in the SSR application. As it can be deduced from the title: “Distributed Web Interface for Real-Time Spatial Audio Reproduction System”, this work aims only to offer the interface via web for the SSR (“Real-Time Spatial Audio Reproduction System”). The idea is not to make a new graphical interface for SSR but to allow more types of interfaces and communication. To accomplish the objective of allowing more graphical interfaces this project is going to use a new network interface. By now the SSR application is using only XML for data interchange but this new network interface support JSON. This project comprehends the server that launch the application, the user interface and the new network interface. It is done with these modules in order to allow creating new user interfaces that can communicate with the server or new servers that can communicate with the user interface by defining a complete network interface for data interchange.

Reduced-parameter model of head-related transfer functions for synthesized spatial audio

Digital systems can generate left and right audio channels that create the effect of virtual sound source placement (spatialization) by processing an audio signal through pairs of Head-Related Transfer Functions (HRTFs) or, equivalently, Head-Related Impulse Responses (HRIRs). The spatialization effect is better when individually-measured HRTFs or HRIRs are used than when generic ones (e.g., from a mannequin) are used. However, the measurement process is not available to the majority of users. There is ongoing interest to find mechanisms to customize HRTFs or HRIRs to a specific user, in order to achieve an improved spatialization effect for that subject. Unfortunately, the current models used for HRTFs and HRIRs contain over a hundred parameters and none of those parameters can be easily related to the characteristics of the subject. This dissertation proposes an alternative model for the representation of HRTFs, which contains at most 30 parameters, all of which have a defined functional significance. It also presents methods to obtain the value of parameters in the model to make it approximately equivalent to an individually-measured HRTF. This conversion is achieved by the systematic deconstruction of HRIR sequences through an augmented version of the Hankel Total Least Squares (HTLS) decomposition approach. An average 95% match (fit) was observed between the original HRIRs and those re-constructed from the Damped and Delayed Sinusoids (DDSs) found by the decomposition process, for ipsilateral source locations. The dissertation also introduces and evaluates an HRIR customization procedure, based on a multilinear model implemented through a 3-mode tensor, for mapping of anatomical data from the subjects to the HRIR sequences at different sound source locations. This model uses the Higher-Order Singular Value Decomposition (HOSVD) method to represent the HRIRs and is capable of generating customized HRIRs from easily attainable anatomical measurements of a new intended user of the system. Listening tests were performed to compare the spatialization performance of customized, generic and individually-measured HRIRs when they are used for synthesized spatial audio. Statistical analysis of the results confirms that the type of HRIRs used for spatialization is a significant factor in the spatialization success, with the customized HRIRs yielding better results than generic HRIRs.

3-D Audio in Mobile Communication Devices: Methods for Mobile Head-Tracking

Future generations of mobile communication devices will serve more and more as multimedia platforms capable of reproducing high quality audio. In order to achieve a 3-D sound perception the reproduction quality of audio via headphones can be significantly increased by applying binaural technology. To be independent of individual head-related transfer functions (HRTFs) and to guarantee a good performance for all listeners, an adaptation of the synthesized sound field to the listener's head movements is required. In this article several methods of head-tracking for mobile communication devices are presented and compared. A system for testing the identified methods is set up and experiments are performed to evaluate the prosand cons of each method. The implementation of such a device in a 3-D audio system is described and applications making use of such a system are identified and discussed.

3-D Audio in Mobile Communication Devices: Effects of Self-Created and External Sounds on Presence in Auditory Virtual Environments

This article describes a series of experiments which were carried out to measure the sense of presence in auditory virtual environments. Within the study a comparison of self-created signals to signals created by the surrounding environment is drawn. Furthermore, it is investigated if the room characteristics of the simulated environment have consequences on the perception of presence during vocalization or when listening to speech. Finally the experiments give information about the influence of background signals on the sense of presence. In the experiments subjects rated the degree of perceived presence in an auditory virtual environment on a perceptual scale. It is described which parameters have the most influence on the perception of presence and which ones are of minor influence. The results show that on the one hand an external speaker has more influence on the sense of presence than an adequate presentation of one’s own voice. On the other hand both room reflections and adequately presented background signals significantly increase the perceived presence in the virtual environment.

Multi-microphone recordings (EigenMike32) inside a WFS system (Iosono) through Ambisonics techniques using Matlab

El audio multicanal ha avanzado a pasos agigantados en los últimos años, y no solo en las técnicas de reproducción, sino que en las de capitación también. Por eso en este proyecto se encuentran ambas cosas: un array microfónico, EigenMike32 de MH Acoustics, y un sistema de reproducción con tecnología Wave Field Synthesis, instalado Iosono en la Jade Höchscule Oldenburg. Para enlazar estos dos puntos de la cadena de audio se proponen dos tipos distintos de codificación: la reproducción de la toma horizontal del EigenMike32; y el 3er orden de Ambisonics (High Order Ambisonics, HOA), una técnica de codificación basada en Armónicos Esféricos mediante la cual se simula el campo acústico en vez de simular las distintas fuentes. Ambas se desarrollaron en el entorno Matlab y apoyadas por la colección de scripts de Isophonics llamada Spatial Audio Matlab Toolbox. Para probar éstas se llevaron a cabo una serie de test en los que se las comparó con las grabaciones realizadas a la vez con un Dummy Head, a la que se supone el método más aproximado a nuestro modo de escucha. Estas pruebas incluían otras grabaciones hechas con un Doble MS de Schoeps que se explican en el proyecto “Sally”. La forma de realizar éstas fue, una batería de 4 audios repetida 4 veces para cada una de las situaciones garbadas (una conversación, una clase, una calle y un comedor universitario). Los resultados fueron inesperados, ya que la codificación del tercer orden de HOA quedo por debajo de la valoración Buena, posiblemente debido a la introducción de material hecho para un array tridimensional dentro de uno de 2 dimensiones. Por el otro lado, la codificación que consistía en extraer los micrófonos del plano horizontal se mantuvo en el nivel de Buena en todas las situaciones. Se concluye que HOA debe seguir siendo probado con mayores conocimientos sobre Armónicos Esféricos; mientras que el otro codificador, mucho más sencillo, puede ser usado para situaciones sin mucha complejidad en cuanto a espacialidad. In the last years the multichannel audio has increased in leaps and bounds and not only in the playback techniques, but also in the recording ones. That is the reason of both things being in this project: a microphone array, EigenMike32 from MH Acoustics; and a playback system with Wave Field Synthesis technology, installed by Iosono in Jade Höchscule Oldenburg. To link these two points of the audio chain, 2 different kinds of codification are proposed: the reproduction of the EigenMike32´s horizontal take, and the Ambisonics´ third order (High Order Ambisonics, HOA), a codification technique based in Spherical Harmonics through which the acoustic field is simulated instead of the different sound sources. Both have been developed inside Matlab´s environment and supported by the Isophonics´ scripts collection called Spatial Audio Matlab Toolbox. To test these, a serial of tests were made in which they were compared with recordings made at the time by a Dummy Head, which is supposed to be the closest method to our hearing way. These tests included other recording and codifications made by a Double MS (DMS) from Schoeps which are explained in the project named “3D audio rendering through Ambisonics techniques: from multi-microphone recordings (DMS Schoeps) to a WFS system, through Matlab”. The way to perform the tests was, a collection made of 4 audios repeated 4 times for each recorded situation (a chat, a class, a street and college canteen or Mensa). The results were unexpected, because the HOA´s third order stood under the Well valuation, possibly caused by introducing material made for a tridimensional array inside one made only by 2 dimensions. On the other hand, the codification that consisted of extracting the horizontal plane microphones kept the Well valuation in all the situations. It is concluded that HOA should keep being tested with larger knowledge about Spherical Harmonics; while the other coder, quite simpler, can be used for situations without a lot of complexity with regards to spatiality.

3-dimensional binaural sound rendering using acoustic measurements stored in the SOFA format

Several groups all over the world are researching in several ways to render 3D sounds. One way to achieve this is to use Head Related Transfer Functions (HRTFs). These measurements contain the Frequency Response of the human head and torso for each angle. Some years ago, was only possible to measure these Frequency Responses only in the horizontal plane. Nowadays, several improvements have made possible to measure and use 3D data for this purpose. The problem was that the groups didn't have a standard format file to store the data. That was a problem when a third part wanted to use some different HRTFs for 3D audio rendering. Every of them have different ways to store the data. The Spatially Oriented Format for Acoustics or SOFA was created to provide a solution to this problem. It is a format definition to unify all the previous different ways of storing any kind of acoustics data. At the moment of this project they have defined some basis for the format and some recommendations to store HRTFs. It is actually under development, so several changes could come. The SOFA[1] file format uses a numeric container called netCDF[2], specifically the Enhaced data model described in netCDF 4 that is based on HDF5[3]. The SoundScape Renderer (SSR) is a tool for real-time spatial audio reproduction providing a variety of rendering algorithms. The SSR was developed at the Quality and Usability Lab at TU Berlin and is now further developed at the Institut für Nachrichtentechnik at Universität Rostock [4]. This project is intended to be an introduction to the use of SOFA files, providing a C++ API to manipulate them and adapt the binaural renderer of the SSR for working with the SOFA format. RESUMEN. El SSR (SoundScape Renderer) es un programa que está siendo desarrollado actualmente por la Universität Rostock, y previamente por la Technische Universität Berlin. El SSR es una herramienta diseñada para la reproducción y renderización de audio 2D en tiempo real. Para ello utiliza diversos algoritmos, algunos orientados a sistemas formados por arrays de altavoces en diferentes configuraciones y otros algoritmos diseñados para cascos. El principal objetivo de este proyecto es dotar al SSR de la capacidad de renderizar sonidos binaurales en 3D. Este proyecto está centrado en el binaural renderer del SSR. Este algoritmo se basa en el uso de HRTFs (Head Related Transfer Function). Las HRTFs representan la función de transferencia del sistema formado por la cabeza y el torso del oyente. Esta función es medida desde diferentes ángulos. Con estos datos el binaural renderer puede generar audio en tiempo real simulando la posición de diferentes fuentes. Para poder incluir una base de datos con HRTFs en 3D se ha hecho uso del nuevo formato SOFA (Spatially Oriented Format for Acoustics). Este nuevo formato se encuentra en una fase bastante temprana de su desarrollo. Está pensado para servir como formato estándar para almacenar HRTFs y cualquier otro tipo de medidas acústicas, ya que actualmente cada laboratorio cuenta con su propio formato de almacenamiento y esto hace bastante difícil usar varias bases de datos diferentes en un mismo proyecto. El formato SOFA hace uso del contenedor numérico netCDF, que a su vez esta basado en un contenedor más básico llamado HRTF-5. Para poder incluir el formato SOFA en el binaural renderer del SSR se ha desarrollado una API en C++ para poder crear y leer archivos SOFA con el fin de utilizar los datos contenidos en ellos dentro del SSR.

In-car sound linear simulation

In recent years, vehicle acoustics have gained significant importance in new car development: increasingly advanced infotainment systems for spatial audio and sound enhancement algorithms have become the norm in modern vehicles. In the past, car manufacturers had to build numerous prototypes to study the sound behaviour inside the car cabin or the effect of new algorithms under development. Nowadays, advanced simulation techniques can reduce development costs and time. In this work, after selecting the reference test vehicle, a modern luxury sedan equipped with a high-end sound system, two independent tools were developed: a simulation tool created in the Comsol Multiphysics environment and an auralization tool developed in the Cycling ‘74 MAX environment. The simulation tool can calculate the impulse response and acoustic spectrum at a specific position inside the cockpit. Its input data are the vehicle’s geometry, acoustic absorption parameters of materials, the acoustic characteristics and position of loudspeakers, and the type and position of virtual microphones (or microphone arrays). The simulation tool can also provide binaural impulse responses thanks to Head Related Transfer Functions (HRTFs) and an innovative algorithm able to compute the HRTF at any distance and angle from the head. Impulse responses from simulations or acoustic measurements inside the car cabin are processed and fed into the auralization tool, enabling real-time interaction by applying filters, changing the channels gain or displaying the acoustic spectrum. Since the acoustic simulation of a vehicle involves multiple topics, the focus of this work has not only been the development of two tools but also the study and application of new techniques for acoustic characterization of the materials that compose the cockpit and the loudspeaker simulation. Specifically, three different methods have been applied for material characterization through the use of a pressure-velocity probe, a Laser Doppler Vibrometer (LDV), and a microphone array.

Wildspace VR. Il wellness per le nuove generazioni - Studio su come la psicologia percettiva applicata alla Virtual Reality può migliorare gli effetti della Mindfulness

Dai recenti studi effettuati sulle nuove generazioni (Z e Alpha) è emerso che stress e ansia colpiscono di più i giovani che si trovano in queste fasce di età. Questa tesi, partendo dalle ricerche socio-culturali, vuole proporre una soluzione progettuale a questi problemi. Si vuole elaborare quindi un progetto sul benessere mentale attraverso la Mindfulness. Le nuove tecnologie emergenti, come ad esempio l’Extended reality, stanno trovando consenso tra le nuove generazioni. Per questo motivo è stata scelta questa tecnologia per erogare il contenuto del servizio. Si ipotizza di poter migliorare gli effetti della Mindfulness utilizzando la Psicologia Percettiva. Questo campo di studio consente di sfruttare l’alterazione della percezione nella Virtual Reality attraverso ‘Spatial Visualization’ e ‘Spatial Audio’. Queste ipotesi sono stata valutate attraverso uno studio indiretto del mercato del Wellness e consolidate attraverso una User Research. Dopo questa fase di ricerca si è valutata la strada progettuale comprendendo a fondo i bisogni del target: i giovani lavoratori. Sono stati utilizzati metodi di Design Thinking e Service Design per intraprendere il progetto, tenendo ben saldi i valori etici. Il progetto pensato, Wildspace VR, è un servizio per aziende che permette ai dipendenti di meditare con contenuti di Mindfulness in Realtà Virtuale. I contenuti proposti mirano a rispondere ai bisogni degli utenti elaborando per loro percorsi personalizzati. Le aziende traggono ricavo dal servizio avendo dipendenti profittevoli salvaguardando il loro stato di salute. L’obiettivo è aiutare i giovani lavoratori offrendogli la possibilità di praticare la Mindfulness dal luogo di lavoro grazie alla Realtà Virtuale, dando loro uno strumento per affrontare lo stress quotidiano in modo innovativo. Si è infine fatto un M.V.P. dell’esperienza, elaborando un prototipo Virtuale e stilando conclusioni ed idee per sviluppi futuri.

Investigating human audio-visual object perception with a combination of hypothesis-generating and hypothesis-testing fMRI analysis tools

Primate multisensory object perception involves distributed brain regions. To investigate the network character of these regions of the human brain, we applied data-driven group spatial independent component analysis (ICA) to a functional magnetic resonance imaging (fMRI) data set acquired during a passive audio-visual (AV) experiment with common object stimuli. We labeled three group-level independent component (IC) maps as auditory (A), visual (V), and AV, based on their spatial layouts and activation time courses. The overlap between these IC maps served as definition of a distributed network of multisensory candidate regions including superior temporal, ventral occipito-temporal, posterior parietal and prefrontal regions. During an independent second fMRI experiment, we explicitly tested their involvement in AV integration. Activations in nine out of these twelve regions met the max-criterion (A < AV > V) for multisensory integration. Comparison of this approach with a general linear model-based region-of-interest definition revealed its complementary value for multisensory neuroimaging. In conclusion, we estimated functional networks of uni- and multisensory functional connectivity from one dataset and validated their functional roles in an independent dataset. These findings demonstrate the particular value of ICA for multisensory neuroimaging research and using independent datasets to test hypotheses generated from a data-driven analysis.

Speech Intelligibility in Noise With a Single-Unit Cochlear Implant Audio Processor.

INTRODUCTION The Rondo is a single-unit cochlear implant (CI) audio processor comprising the identical components as its behind-the-ear predecessor, the Opus 2. An interchange of the Opus 2 with the Rondo leads to a shift of the microphone position toward the back of the head. This study aimed to investigate the influence of the Rondo wearing position on speech intelligibility in noise. METHODS Speech intelligibility in noise was measured in 4 spatial configurations with 12 experienced CI users using the German adaptive Oldenburg sentence test. A physical model and a numerical model were used to enable a comparison of the observations. RESULTS No statistically significant differences of the speech intelligibility were found in the situations in which the signal came from the front and the noise came from the frontal, ipsilateral, or contralateral side. The signal-to-noise ratio (SNR) was significantly better with the Opus 2 in the case with the noise presented from the back (4.4 dB, p < 0.001). The differences in the SNR were significantly worse with the Rondo processors placed further behind the ear than closer to the ear. CONCLUSION The study indicates that CI users with the receiver/stimulator implanted in positions further behind the ear are expected to have higher difficulties in noisy situations when wearing the single-unit audio processor.

Measuring Spatial Presence: Introducing and Validating the Pictorial Presence SAM

The aim of the present study was to develop a pictorial presence scale using selfassessment- manikins (SAM). The instrument assesses presence sub-dimensions (selflocation and possible actions) as well as presence determinants (attention allocation, spatial situation model, higher cognitive involvement, and suspension of disbelief). To qualitatively validate the scale, think-aloud protocols and interviews (n = 12) were conducted. The results reveal that the SAM items are quickly filled out as well as easily, intuitively, and unambiguously understood. Furthermore, the instrument’s validity and sensitivity was quantitatively examined in a two-factorial design (n = 317). Factors were medium (written story, audio book, video, and computer game) and distraction (non-distraction vs. distraction). Factor analyses reveal that the SAM presence dimensions and determinants closely correspond to those of the MEC Spatial Presence Questionnaire, which was used as a comparison measure. The findings of the qualitative and quantitative validation procedures show that the Pictorial Presence SAM successfully assesses spatial presence. In contrast to the verbal questionnaire data (MEC), the significant distraction effect suggests that the new scale is even more sensitive. This points out that the scale can be a useful alternative to existing verbal presence selfreport measures.

3D Videoconferencing system using spatial scalability

An implementation of a real-time 3D videoconferencing system using the currently available technology is presented. This appr oach is based on the side by side spatial compression of the stereoscopic images . The encoder and the decoder have b een implemented in a standard personal computer and a conventional 3D comp atible TV has been used to present the frames. Moreover, the users without 3D technology can use the system because 2D compatibility mode has been implemented in the decoder. The performance res ults show that a conventional computer can be used for encod ing/decoding audio and video streams and the delay in the transmission is lower than 200 ms.

The speech, spatial and qualities of hearing scale: tradução e adaptação cultural para o português europeu

A deficiência auditiva afecta milhões de pessoas em todo o mundo, originando vários problemas, nomeadamente a nível psicossocial, que comprometem a qualidade de vida do indivíduo. A deficiência auditiva influencia o comportamento, particularmente ao dificultar a comunicação. Com o avanço tecnológico, os produtos de apoio, em particular os aparelhos auditivos e o implante coclear, melhoram essa qualidade de vida, através da melhoria da comunicação. Com as escalas de avaliação determinamos o modo como a deficiência auditiva influencia a vida diária, com ou sem amplificação, e de que forma afecta o desempenho psicossocial, emocional ou profissional do indivíduo, sendo esta informação importante para determinar a necessidade e o sucesso de amplificação, independentemente do tipo e grau da deficiência auditiva. O objectivo do presente estudo foi a tradução e adaptação para a cultura portuguesa da escala The Speech, Spatial and Qualities of Hearing Scale (SSQ), desenvolvida por Stuart Gatehouse e William Noble em 2004. Este trabalho foi realizado nos centros auditivos da Widex Portugal. Após os procedimentos de tradução e retroversão, a versão portuguesa foi testada em 12 indivíduos, com idades compreendidas entre os 36 anos e os 80 anos, dos quais 6 utilizavam prótese auditiva há mais de um ano, um utilizava prótese há menos de um ano e 5 nunca tinham utilizado. Com a tradução e adaptação cultural para o Português Europeu do “Questionário sobre as Qualidades Espaciais do Discurso – SSQ”, contribuímos para uma melhor avaliação dos indivíduos que estejam, ou venham a estar, a cumprir programas de reabilitação auditiva.

Non-gaussian Spatial Correlations Dramatically Weaken Localization.

We perform variational studies of the interaction-localization problem to describe the interaction-induced renormalizations of the effective (screened) random potential seen by quasiparticles. Here we present results of careful finite-size scaling studies for the conductance of disordered Hubbard chains at half-filling and zero temperature. While our results indicate that quasiparticle wave functions remain exponentially localized even in the presence of moderate to strong repulsive interactions, we show that interactions produce a strong decrease of the characteristic conductance scale g^{*} signaling the crossover to strong localization. This effect, which cannot be captured by a simple renormalization of the disorder strength, instead reflects a peculiar non-Gaussian form of the spatial correlations of the screened disordered potential, a hitherto neglected mechanism to dramatically reduce the impact of Anderson localization (interference) effects.

Metodo de sincronização de cameras de video utilizando a banda de audio

Universidade Estadual de Campinas . Faculdade de Educação Física