Towards an emotion-driven interactive music system: ridging the gaps between affect, physiology and sound generation

Report for the scientific sojourn at the Stanford University from January until June 2007. Music is well known for affecting human emotional states, yet the relationship between specific musical parameters and emotional responses is still not clear. With the advent of new human-computer interaction (HCI) technologies, it is now possible to derive emotion-related information from physiological data and use it as an input to interactive music systems. Providing such implicit musical HCI will be highly relevant for a number of applications including music therapy, diagnosis, nteractive gaming, and physiologically-based musical instruments. A key question in such physiology-based compositions is how sound synthesis parameters can be mapped to emotional states of valence and arousal. We used both verbal and heart rate responses to evaluate the affective power of five musical parameters. Our results show that a significant correlation exists between heart rate and the subjective evaluation of well-defined musical parameters. Brightness and loudness showed to be arousing parameters on subjective scale while harmonicity and even partial attenuation factor resulted in heart rate changes typically associated to valence. This demonstrates that a rational approach to designing emotion-driven music systems for our public installations and music therapy applications is possible.

Sound Texture Modelling

In the PhD thesis “Sound Texture Modeling” we deal with statistical modelling or textural sounds like water, wind, rain, etc. For synthesis and classification. Our initial model is based on a wavelet tree signal decomposition and the modeling of the resulting sequence by means of a parametric probabilistic model, that can be situated within the family of models trainable via expectation maximization (hidden Markov tree model ). Our model is able to capture key characteristics of the source textures (water, rain, fire, applause, crowd chatter ), and faithfully reproduces some of the sound classes. In terms of a more general taxonomy of natural events proposed by Graver, we worked on models for natural event classification and segmentation. While the event labels comprise physical interactions between materials that do not have textural propierties in their enterity, those segmentation models can help in identifying textural portions of an audio recording useful for analysis and resynthesis. Following our work on concatenative synthesis of musical instruments, we have developed a pattern-based synthesis system, that allows to sonically explore a database of units by means of their representation in a perceptual feature space. Concatenative syntyhesis with “molecules” built from sparse atomic representations also allows capture low-level correlations in perceptual audio features, while facilitating the manipulation of textural sounds based on their physical and perceptual properties. We have approached the problem of sound texture modelling for synthesis from different directions, namely a low-level signal-theoretic point of view through a wavelet transform, and a more high-level point of view driven by perceptual audio features in the concatenative synthesis setting. The developed framework provides unified approach to the high-quality resynthesis of natural texture sounds. Our research is embedded within the Metaverse 1 European project (2008-2011), where our models are contributting as low level building blocks within a semi-automated soundscape generation system.

Ab initio absorption spectrum of NiO combining molecular dynamics with the embedded cluster approach in a discrete reaction field

We developed a procedure that combines three complementary computational methodologies to improve the theoretical description of the electronic structure of nickel oxide. The starting point is a Car-Parrinello molecular dynamics simulation to incorporate vibrorotational degrees of freedom into the material model. By means ofcomplete active space self-consistent field second-order perturbation theory (CASPT2) calculations on embedded clusters extracted from the resulting trajectory, we describe localized spectroscopic phenomena on NiO with an efficient treatment of electron correlation. The inclusion of thermal motion into the theoretical description allowsus to study electronic transitions that, otherwise, would be dipole forbidden in the ideal structure and results in a natural reproduction of the band broadening. Moreover, we improved the embedded cluster model by incorporating self-consistently at the complete active space self-consistent field (CASSCF) level a discrete (or direct) reaction field (DRF) in the cluster surroundings. The DRF approach offers an efficient treatment ofelectric response effects of the crystalline embedding to the electronic transitions localized in the cluster. We offer accurate theoretical estimates of the absorption spectrum and the density of states around the Fermi level of NiO, and a comprehensive explanation of the source of the broadening and the relaxation of the charge transferstates due to the adaptation of the environment

Vibration and two-photon absorption

The rigorous and transparent treatment of the effects of nuclear vibrational motion in two-photon absorption (TPA) was discussed. Perturbation formula for diatomic molecules were developed and applied to the X¹Σ+–A¹Π transition in CO. The analysis showed that the vibrations played an important role in TPA, just as their role in the calculation of conventional nonlinear optical (NLO) hyperpolarizabilities

The effect of atmospheric absorption of sunlight on the runaway greenhouse point

The longwave emission of planetary atmospheres that contain a condensable absorbing gas in the infrared (i.e., longwave), which is in equilibrium with its liquid phase at the surface, may exhibit an upper bound. Here we analyze the effect of the atmospheric absorption of sunlight on this radiation limit. We assume that the atmospheric absorption of infrared radiation is independent of wavelength except within the spectral width of the atmospheric window, where it is zero. The temperature profile in radiative equilibrium is obtained analytically as a function of the longwave optical thickness. For illustrative purposes, numerical values for the infrared atmospheric absorption (i.e., greenhouse effect) and the liquid vapor equilibrium curve of the condensable absorbing gas refer to water. Values for the atmospheric absorption of sunlight (i.e., antigreenhouse effect) take a wide range since our aim is to provide a qualitative view of their effects. We find that atmospheres with a transparent region in the infrared spectrum do not present an absolute upper bound on the infrared emission. This result may be also found in atmospheres opaque at all infrared wavelengths if the fraction of absorbed sunlight in the atmosphere increases with the longwave opacity

Methodology for studying the numerical speed of sound in finite differences schemes

The space and time discretization inherent to all FDTD schemesintroduce non-physical dispersion errors, i.e. deviations ofthe speed of sound from the theoretical value predicted bythe governing Euler differential equations. A generalmethodologyfor computing this dispersion error via straightforwardnumerical simulations of the FDTD schemes is presented.The method is shown to provide remarkable accuraciesof the order of 1/1000 in a wide variety of twodimensionalfinite difference schemes.

Analysis by optical absorption and transmission electron microscopy of the strain inhomogeneities in InGaAs/InP strained layers

Optical absorption spectra and transmission electron microscopy (TEM) observations on InGaAs/InP layers under compressive strain are reported. From the band¿gap energy dispersion, the magnitude of the strain inhomogeneities. Is quantified and its microscopic origin is analyzed in view of the layer microstructure. TEM observations reveal a dislocation network at the layer interface the density of which correlates with ¿¿. It is concluded that local variations of dislocation density are responsible for the inhomogeneous strain field together with another mechanism that dominates when the dislocation density is very low.

Two and three nucleon induced photon absorption

We have investigated the mechanisms leading to two and three body photon absorption in nuclei. At photon energies around the pion production threshold we obtain a fraction of three body absorption of less than 10% of the total, contradicting previous theoretical claims that it dominates the absorption process. The strength of the three body channel grows smoothly with the photon energy reaching a maximum of about 60% of the total direct absorption at energies of the photon around 400 MeV.

Optical absorption and graphitic clusters of hydrogenated amorphophous carbon thin films

The optical absorption of hydrogenated amorphous carbon films (a‐C:H) was measured by spectroscopic ellipsometry. The a‐C:H films were deposited at different substrate temperatures by rf‐plasma of methane. A volume distribution of graphitic cluster size was assumed to reproduce the experimental spectra of the absorption coefficient. The changes in the absorption coefficient and the optical gap, induced by deposition temperature, have been interpreted in terms of changes in the graphitic cluster size of the network. The increase in the deposition temperature produces an increase in the size of the graphitic clusters.

Microwave absorption and magnetization tunneling in Mn12-acetate molecular clusters

In this paper we study the effect of microwave absorption on the quantum relaxation rate of Mn12 molecular clusters. We have determined first the resonant frequencies of a microwave resonator containing a single crystal of Mn12-Acetate and measured initial isothermal magnetization curves while microwave power was put into the resonator. We have found that the tunneling rate changes one order of magnitude for certain frequencies. This suggests that the microwave shaking of the nuclear spin and molecular vibrational degrees of freedom is responsible for the huge increasing of the tunneling rate.

Black holes as fundamental strings: Comparing the absorption of scalars

The recently proposed correspondence principle of Horowitz and Polchinski provides a concrete means to relate (among others) black holes with electric Neveu-SchwarzNeveu-Schwarz charges to fundamental strings and correctly match their entropies. We further test this correspondence by examining the greybody factors in the absorption rates of neutral, minimally coupled scalars by a near extremal black hole. Perhaps surprisingly, the results disagree in general with the absorption by weakly coupled strings. Though this does not disprove the correspondence, it indicates that it might not be simple in this region of the black hole parameter space.

Statistical modeling of bowing control applied to violin sound synthesis

Excitation-continuous music instrument control patterns are often not explicitly represented in current sound synthesis techniques when applied to automatic performance. Both physical model-based and sample-based synthesis paradigmswould benefit from a flexible and accurate instrument control model, enabling the improvement of naturalness and realism. Wepresent a framework for modeling bowing control parameters inviolin performance. Nearly non-intrusive sensing techniques allow for accurate acquisition of relevant timbre-related bowing control parameter signals.We model the temporal contour of bow velocity, bow pressing force, and bow-bridge distance as sequences of short Bézier cubic curve segments. Considering different articulations, dynamics, and performance contexts, a number of note classes are defined. Contours of bowing parameters in a performance database are analyzed at note-level by following a predefined grammar that dictates characteristics of curve segment sequences for each of the classes in consideration. As a result, contour analysis of bowing parameters of each note yields an optimal representation vector that is sufficient for reconstructing original contours with significant fidelity. From the resulting representation vectors, we construct a statistical model based on Gaussian mixtures suitable for both the analysis and synthesis of bowing parameter contours. By using the estimated models, synthetic contours can be generated through a bow planning algorithm able to reproduce possible constraints caused by the finite length of the bow. Rendered contours are successfully used in two preliminary synthesis frameworks: digital waveguide-based bowed stringphysical modeling and sample-based spectral-domain synthesis.