On cortical coding of vocal communication sounds in primates

Understanding how the brain processes vocal communication sounds is one of the most challenging problems in neuroscience. Our understanding of how the cortex accomplishes this unique task should greatly facilitate our understanding of cortical mechanisms in general. Perception of species-specific communication sounds is an important aspect of the auditory behavior of many animal species and is crucial for their social interactions, reproductive success, and survival. The principles of neural representations of these behaviorally important sounds in the cerebral cortex have direct implications for the neural mechanisms underlying human speech perception. Our progress in this area has been relatively slow, compared with our understanding of other auditory functions such as echolocation and sound localization. This article discusses previous and current studies in this field, with emphasis on nonhuman primates, and proposes a conceptual platform to further our exploration of this frontier. It is argued that the prerequisite condition for understanding cortical mechanisms underlying communication sound perception and production is an appropriate animal model. Three issues are central to this work: (i) neural encoding of statistical structure of communication sounds, (ii) the role of behavioral relevance in shaping cortical representations, and (iii) sensory–motor interactions between vocal production and perception systems.

Acoustic overstimulation increases outer hair cell Ca2+ concentrations and causes dynamic contractions of the hearing organ

The dynamic responses of the hearing organ to acoustic overstimulation were investigated using the guinea pig isolated temporal bone preparation. The organ was loaded with the fluorescent Ca2+ indicator Fluo-3, and the cochlear electric responses to low-level tones were recorded through a microelectrode in the scala media. After overstimulation, the amplitude of the cochlear potentials decreased significantly. In some cases, rapid recovery was seen with the potentials returning to their initial amplitude. In 12 of 14 cases in which overstimulation gave a decrease in the cochlear responses, significant elevations of the cytoplasmic [Ca2+] in the outer hair cells were seen. [Ca2+] increases appeared immediately after terminating the overstimulation, with partial recovery taking place in the ensuing 30 min in some preparations. Such [Ca2+] changes were not seen in preparations that were stimulated at levels that did not cause an amplitude change in the cochlear potentials. The overstimulation also gave rise to a contraction, evident as a decrease of the width of the organ of Corti. The average contraction in 10 preparations was 9 μm (SE 2 μm). Partial or complete recovery was seen within 30–45 min after the overstimulation. The [Ca2+] changes and the contraction are likely to produce major functional alterations and consequently are suggested to be a factor contributing strongly to the loss of function seen after exposure to loud sounds.

Echo-delay resolution in sonar images of the big brown bat, Eptesicus fuscus

Echolocating big brown bats (Eptesicus fuscus) broadcast ultrasonic frequency-modulated (FM) biosonar sounds (20–100 kHz frequencies; 10–50 μs periods) and perceive target range from echo delay. Knowing the acuity for delay resolution is essential to understand how bats process echoes because they perceive target shape and texture from the delay separation of multiple reflections. Bats can separately perceive the delays of two concurrent electronically generated echoes arriving as little as 2 μs apart, thus resolving reflecting points as close together as 0.3 mm in range (two-point threshold). This two-point resolution is roughly five times smaller than the shortest periods in the bat’s sounds. Because the bat’s broadcasts are 2,000–4,500 μs long, the echoes themselves overlap and interfere with each other, to merge together into a single sound whose spectrum is shaped by their mutual interference depending on the size of the time separation. To separately perceive the delays of overlapping echoes, the bat has to recover information about their very small delay separation that was transferred into the spectrum when the two echoes interfered with each other, thus explicitly reconstructing the range profile of targets from the echo spectrum. However, the bat’s 2-μs resolution limit is so short that the available spectral cues are extremely limited. Resolution of delay seems overly sharp just for interception of flying insects, which suggests that the bat’s biosonar images are of higher quality to suit a wider variety of orientation tasks, and that biosonar echo processing is correspondingly more sophisticated than has been suspected.

Transient “deafness” accompanies auditory development during metamorphosis from tadpole to frog

During metamorphosis, ranid frogs shift from a purely aquatic to a partly terrestrial lifestyle. The central auditory system undergoes functional and neuroanatomical reorganization in parallel with the development of new sound conduction pathways adapted for the detection of airborne sounds. Neural responses to sounds can be recorded from the auditory midbrain of tadpoles shortly after hatching, with higher rates of synchronous neural activity and lower sharpness of tuning than observed in postmetamorphic animals. Shortly before the onset of metamorphic climax, there is a brief “deaf” period during which no auditory activity can be evoked from the midbrain, and a loss of connectivity is observed between medullary and midbrain auditory nuclei. During the final stages of metamorphic development, auditory function and neural connectivity are restored. The acoustic communication system of the adult frog emerges from these periods of anatomical and physiological plasticity during metamorphosis.

Cortical processing of change detection: Dissociation between natural vowels and two-frequency complex tones

We compared magnetoencephalographic responses for natural vowels and for sounds consisting of two pure tones that represent the two lowest formant frequencies of these vowels. Our aim was to determine whether spectral changes in successive stimuli are detected differently for speech and nonspeech sounds. The stimuli were presented in four blocks applying an oddball paradigm (20% deviants, 80% standards): (i) /α/ tokens as deviants vs. /i/ tokens as standards; (ii) /e/ vs. /i/; (iii) complex tones representing /α/ formants vs. /i/ formants; and (iv) complex tones representing /e/ formants vs. /i/ formants. Mismatch fields (MMFs) were calculated by subtracting the source waveform produced by standards from that produced by deviants. As expected, MMF amplitudes for the complex tones reflected acoustic deviation: the amplitudes were stronger for the complex tones representing /α/ than /e/ formants, i.e., when the spectral difference between standards and deviants was larger. In contrast, MMF amplitudes for the vowels were similar despite their different spectral composition, whereas the MMF onset time was longer for /e/ than for /α/. Thus the degree of spectral difference between standards and deviants was reflected by the MMF amplitude for the nonspeech sounds and by the MMF latency for the vowels.

Reactivation of encoding-related brain activity during memory retrieval

Neuronal models predict that retrieval of specific event information reactivates brain regions that were active during encoding of this information. Consistent with this prediction, this positron-emission tomography study showed that remembering that visual words had been paired with sounds at encoding activated some of the auditory brain regions that were engaged during encoding. After word-sound encoding, activation of auditory brain regions was also observed during visual word recognition when there was no demand to retrieve auditory information. Collectively, these observations suggest that information about the auditory components of multisensory event information is stored in auditory responsive cortex and reactivated at retrieval, in keeping with classical ideas about “redintegration,” that is, the power of part of an encoded stimulus complex to evoke the whole experience.

Memory's echo: Vivid remembering reactivates sensory-specific cortex

A fundamental question in human memory is how the brain represents sensory-specific information during the process of retrieval. One hypothesis is that regions of sensory cortex are reactivated during retrieval of sensory-specific information (1). Here we report findings from a study in which subjects learned a set of picture and sound items and were then given a recall test during which they vividly remembered the items while imaged by using event-related functional MRI. Regions of visual and auditory cortex were activated differentially during retrieval of pictures and sounds, respectively. Furthermore, the regions activated during the recall test comprised a subset of those activated during a separate perception task in which subjects actually viewed pictures and heard sounds. Regions activated during the recall test were found to be represented more in late than in early visual and auditory cortex. Therefore, results indicate that retrieval of vivid visual and auditory information can be associated with a reactivation of some of the same sensory regions that were activated during perception of those items.

How do owls localize interaurally phase-ambiguous signals?

Owls and other animals, including humans, use the difference in arrival time of sounds between the ears to determine the direction of a sound source in the horizontal plane. When an interaural time difference (ITD) is conveyed by a narrowband signal such as a tone, human beings may fail to derive the direction represented by that ITD. This is because they cannot distinguish the true ITD contained in the signal from its phase equivalents that are ITD ± nT, where T is the period of the stimulus tone and n is an integer. This uncertainty is called phase-ambiguity. All ITD-sensitive neurons in birds and mammals respond to an ITD and its phase equivalents when the ITD is contained in narrowband signals. It is not known, however, if these animals show phase-ambiguity in the localization of narrowband signals. The present work shows that barn owls (Tyto alba) experience phase-ambiguity in the localization of tones delivered by earphones. We used sound-induced head-turning responses to measure the sound-source directions perceived by two owls. In both owls, head-turning angles varied as a sinusoidal function of ITD. One owl always pointed to the direction represented by the smaller of the two ITDs, whereas a second owl always chose the direction represented by the larger ITD (i.e., ITD − T).

How the owl resolves auditory coding ambiguity

The barn owl (Tyto alba) uses interaural time difference (ITD) cues to localize sounds in the horizontal plane. Low-order binaural auditory neurons with sharp frequency tuning act as narrow-band coincidence detectors; such neurons respond equally well to sounds with a particular ITD and its phase equivalents and are said to be phase ambiguous. Higher-order neurons with broad frequency tuning are unambiguously selective for single ITDs in response to broad-band sounds and show little or no response to phase equivalents. Selectivity for single ITDs is thought to arise from the convergence of parallel, narrow-band frequency channels that originate in the cochlea. ITD tuning to variable bandwidth stimuli was measured in higher-order neurons of the owl’s inferior colliculus to examine the rules that govern the relationship between frequency channel convergence and the resolution of phase ambiguity. Ambiguity decreased as stimulus bandwidth increased, reaching a minimum at 2–3 kHz. Two independent mechanisms appear to contribute to the elimination of ambiguity: one suppressive and one facilitative. The integration of information carried by parallel, distributed processing channels is a common theme of sensory processing that spans both modality and species boundaries. The principles underlying the resolution of phase ambiguity and frequency channel convergence in the owl may have implications for other sensory systems, such as electrolocation in electric fish and the computation of binocular disparity in the avian and mammalian visual systems.

Pitch perception: A dynamical-systems perspective

Two and a half millennia ago Pythagoras initiated the scientific study of the pitch of sounds; yet our understanding of the mechanisms of pitch perception remains incomplete. Physical models of pitch perception try to explain from elementary principles why certain physical characteristics of the stimulus lead to particular pitch sensations. There are two broad categories of pitch-perception models: place or spectral models consider that pitch is mainly related to the Fourier spectrum of the stimulus, whereas for periodicity or temporal models its characteristics in the time domain are more important. Current models from either class are usually computationally intensive, implementing a series of steps more or less supported by auditory physiology. However, the brain has to analyze and react in real time to an enormous amount of information from the ear and other senses. How is all this information efficiently represented and processed in the nervous system? A proposal of nonlinear and complex systems research is that dynamical attractors may form the basis of neural information processing. Because the auditory system is a complex and highly nonlinear dynamical system, it is natural to suppose that dynamical attractors may carry perceptual and functional meaning. Here we show that this idea, scarcely developed in current pitch models, can be successfully applied to pitch perception.

Auditory neuroscience: Development, transduction, and integration

Hearing underlies our ability to locate sound sources in the environment, our appreciation of music, and our ability to communicate. Participants in the National Academy of Sciences colloquium on Auditory Neuroscience: Development, Transduction, and Integration presented research results bearing on four key issues in auditory research. How does the complex inner ear develop? How does the cochlea transduce sounds into electrical signals? How does the brain's ability to compute the location of a sound source develop? How does the forebrain analyze complex sounds, particularly species-specific communications? This article provides an introduction to the papers stemming from the meeting.

Detection of synchrony in the activity of auditory nerve fibers by octopus cells of the mammalian cochlear nucleus

The anatomical and biophysical specializations of octopus cells allow them to detect the coincident firing of groups of auditory nerve fibers and to convey the precise timing of that coincidence to their targets. Octopus cells occupy a sharply defined region of the most caudal and dorsal part of the mammalian ventral cochlear nucleus. The dendrites of octopus cells cross the bundle of auditory nerve fibers just proximal to where the fibers leave the ventral and enter the dorsal cochlear nucleus, each octopus cell spanning about one-third of the tonotopic array. Octopus cells are excited by auditory nerve fibers through the activation of rapid, calcium-permeable, α-amino-3-hydroxy-5-methyl-4-isoxazole-propionate receptors. Synaptic responses are shaped by the unusual biophysical characteristics of octopus cells. Octopus cells have very low input resistances (about 7 MΩ), and short time constants (about 200 μsec) as a consequence of the activation at rest of a hyperpolarization-activated mixed-cation conductance and a low-threshold, depolarization-activated potassium conductance. The low input resistance causes rapid synaptic currents to generate rapid and small synaptic potentials. Summation of small synaptic potentials from many fibers is required to bring an octopus cell to threshold. Not only does the low input resistance make individual excitatory postsynaptic potentials brief so that they must be generated within 1 msec to sum but also the voltage-sensitive conductances of octopus cells prevent firing if the activation of auditory nerve inputs is not sufficiently synchronous and depolarization is not sufficiently rapid. In vivo in cats, octopus cells can fire rapidly and respond with exceptionally well-timed action potentials to periodic, broadband sounds such as clicks. Thus both the anatomical specializations and the biophysical specializations make octopus cells detectors of the coincident firing of their auditory nerve fiber inputs.

Mechanisms and streams for processing of “what” and “where” in auditory cortex

The functional specialization and hierarchical organization of multiple areas in rhesus monkey auditory cortex were examined with various types of complex sounds. Neurons in the lateral belt areas of the superior temporal gyrus were tuned to the best center frequency and bandwidth of band-passed noise bursts. They were also selective for the rate and direction of linear frequency modulated sweeps. Many neurons showed a preference for a limited number of species-specific vocalizations (“monkey calls”). These response selectivities can be explained by nonlinear spectral and temporal integration mechanisms. In a separate series of experiments, monkey calls were presented at different spatial locations, and the tuning of lateral belt neurons to monkey calls and spatial location was determined. Of the three belt areas the anterolateral area shows the highest degree of specificity for monkey calls, whereas neurons in the caudolateral area display the greatest spatial selectivity. We conclude that the cortical auditory system of primates is divided into at least two processing streams, a spatial stream that originates in the caudal part of the superior temporal gyrus and projects to the parietal cortex, and a pattern or object stream originating in the more anterior portions of the lateral belt. A similar division of labor can be seen in human auditory cortex by using functional neuroimaging.

Plasticity in the neural coding of auditory space in the mammalian brain

Sound localization relies on the neural processing of monaural and binaural spatial cues that arise from the way sounds interact with the head and external ears. Neurophysiological studies of animals raised with abnormal sensory inputs show that the map of auditory space in the superior colliculus is shaped during development by both auditory and visual experience. An example of this plasticity is provided by monaural occlusion during infancy, which leads to compensatory changes in auditory spatial tuning that tend to preserve the alignment between the neural representations of visual and auditory space. Adaptive changes also take place in sound localization behavior, as demonstrated by the fact that ferrets raised and tested with one ear plugged learn to localize as accurately as control animals. In both cases, these adjustments may involve greater use of monaural spectral cues provided by the other ear. Although plasticity in the auditory space map seems to be restricted to development, adult ferrets show some recovery of sound localization behavior after long-term monaural occlusion. The capacity for behavioral adaptation is, however, task dependent, because auditory spatial acuity and binaural unmasking (a measure of the spatial contribution to the “cocktail party effect”) are permanently impaired by chronically plugging one ear, both in infancy but especially in adulthood. Experience-induced plasticity allows the neural circuitry underlying sound localization to be customized to individual characteristics, such as the size and shape of the head and ears, and to compensate for natural conductive hearing losses, including those associated with middle ear disease in infancy.

State of the art in continuous speech recognition.

In the past decade, tremendous advances in the state of the art of automatic speech recognition by machine have taken place. A reduction in the word error rate by more than a factor of 5 and an increase in recognition speeds by several orders of magnitude (brought about by a combination of faster recognition search algorithms and more powerful computers), have combined to make high-accuracy, speaker-independent, continuous speech recognition for large vocabularies possible in real time, on off-the-shelf workstations, without the aid of special hardware. These advances promise to make speech recognition technology readily available to the general public. This paper focuses on the speech recognition advances made through better speech modeling techniques, chiefly through more accurate mathematical modeling of speech sounds.