Seeing and hearing speech, sounds, and signs: functional magnetic resonance studies on fluent and dyslexic readers

Speech has both auditory and visual components (heard speech sounds and seen articulatory gestures). During all perception, selective attention facilitates efficient information processing and enables concentration on high-priority stimuli. Auditory and visual sensory systems interact at multiple processing levels during speech perception and, further, the classical motor speech regions seem also to participate in speech perception. Auditory, visual, and motor-articulatory processes may thus work in parallel during speech perception, their use possibly depending on the information available and the individual characteristics of the observer. Because of their subtle speech perception difficulties possibly stemming from disturbances at elemental levels of sensory processing, dyslexic readers may rely more on motor-articulatory speech perception strategies than do fluent readers. This thesis aimed to investigate the neural mechanisms of speech perception and selective attention in fluent and dyslexic readers. We conducted four functional magnetic resonance imaging experiments, during which subjects perceived articulatory gestures, speech sounds, and other auditory and visual stimuli. Gradient echo-planar images depicting blood oxygenation level-dependent contrast were acquired during stimulus presentation to indirectly measure brain hemodynamic activation. Lip-reading activated the primary auditory cortex, and selective attention to visual speech gestures enhanced activity within the left secondary auditory cortex. Attention to non-speech sounds enhanced auditory cortex activity bilaterally; this effect showed modulation by sound presentation rate. A comparison between fluent and dyslexic readers' brain hemodynamic activity during audiovisual speech perception revealed stronger activation of predominantly motor speech areas in dyslexic readers during a contrast test that allowed exploration of the processing of phonetic features extracted from auditory and visual speech. The results show that visual speech perception modulates hemodynamic activity within auditory cortex areas once considered unimodal, and suggest that the left secondary auditory cortex specifically participates in extracting the linguistic content of seen articulatory gestures. They are strong evidence for the importance of attention as a modulator of auditory cortex function during both sound processing and visual speech perception, and point out the nature of attention as an interactive process (influenced by stimulus-driven effects). Further, they suggest heightened reliance on motor-articulatory and visual speech perception strategies among dyslexic readers, possibly compensating for their auditory speech perception difficulties.

East Asian audio-visual collaboration and the global expansion of Chinese media

In recent years, many of the world’s leading media producers, screenwriters, technicians and investors, particularly those in the Asia-Pacific region, have been drawn to work in the People's Republic of China (hereafter China or Mainland China). Media projects with a lighter commercial entertainment feel – compared with the heavy propaganda-oriented content of the past – have multiplied, thanks to the Chinese state’s newfound willingness to consider collaboration with foreign partners. This is no more evident than in film. Despite their long-standing reputation for rigorous censorship, state policymakers are now encouraging Chinese media entrepreneurs to generate fresh ideas and to develop products that will revitalise the stagnant domestic production sector. It is hoped that an increase in both the quality and quantity of domestic feature films, stimulated by an infusion of creativity and cutting-edge technology from outside the country, will help reverse China’s ‘cultural trade deficit’ (wenhua maoyi chizi) (Keane 2007).

Instagrammatics and digital methods: Studying visual social media, from selfies and GIFs to memes and emoji

Visual content is a critical component of everyday social media, on platforms explicitly framed around the visual (Instagram and Vine), on those offering a mix of text and images in myriad forms (Facebook, Twitter, and Tumblr), and in apps and profiles where visual presentation and provision of information are important considerations. However, despite being so prominent in forms such as selfies, looping media, infographics, memes, online videos, and more, sociocultural research into the visual as a central component of online communication has lagged behind the analysis of popular, predominantly text-driven social media. This paper underlines the increasing importance of visual elements to digital, social, and mobile media within everyday life, addressing the significant research gap in methods for tracking, analysing, and understanding visual social media as both image-based and intertextual content. In this paper, we build on our previous methodological considerations of Instagram in isolation to examine further questions, challenges, and benefits of studying visual social media more broadly, including methodological and ethical considerations. Our discussion is intended as a rallying cry and provocation for further research into visual (and textual and mixed) social media content, practices, and cultures, mindful of both the specificities of each form, but also, and importantly, the ongoing dialogues and interrelations between them as communication forms.

Conformational studies of the triphenylphosphazenyl side chain in cyclophosphazenes. I. Crystal and molecular structure of N3P3C15(NPPh3)

The title compound, C t8H~sC15NaP4, crystallizes in the monoclinic space group P2~/n with a = 20.14 (2), b = 8.69 (1), c = 14.92 (2) A, fl = 98.8 (3) ° , Z = 4. The structure was determined from visual data and refined to R = 0-069 for 1450 reflections. The cyclophosphazene ring is non-planar. The exocyclic NPPh 3 group exhibits type I conformation [R. A. Shaw (1975). Pure Appl. Chem. 44, 317-341] in which the N-P bond is perpendicular to the adjacent P-CI bond.

Pain measurement and management in elderly patients : Clinical studies in long term hospital care and after cardiac surgery

The proportion of patients over 75 years of age, receiving all different types of healthcare, is constantly increasing. The elderly undergo surgery and anaesthetic procedures more often than middle-aged patients. Poor pain management in the elderly is still an issue. Although the elderly consumes the greatest proportion of prescribed medicines in Western Europe, most clinical pharmacological studies have been performed in healthy volunteers or middle-aged patients. The aim of this study was to investigate pain measurement and management in cognitively impaired patients in long term hospital care and in cognitively normal elderly patients after cardiac surgery. This thesis incorporated 366 patients, including 86 home-dwelling or hospitalized elderly with chronic pain and 280 patients undergoing cardiac surgery with acute pain. The mean age of patients was 77 (SD ± 8) years and approximately 8400 pain measurements were performed with four pain scales: Verbal Rating Scale (VRS), the Visual Analogue Scale (VAS), the Red Wedge Scale (RWS), and the Facial Pain Scale (FPS). Cognitive function, depression, functional ability in daily life, postoperative sedation and postoperative confusion were assessed with MMSE, GDS, Barthel Index, RASS, and CAM-ICU, respectively. The effects and plasma concentrations of fentanyl and oxycodone were measured in elderly (≥ 75 years) and middle-aged patients (≤ 60 years) and the opioid-sparing effect of pregabalin was studied after cardiac surgery. The VRS pain scores after movement correlated with the Barthel Index. The VRS was most successful in the groups of demented patients (MMSE 17-23, 11-16 and ≤ 10) and in elderly patients on the first day after cardiac surgery. The elderly had a higher plasma concentration of fentanyl at the end of surgery than younger patients. The plasma concentrations of oxycodone were comparable between the groups. Pain intensity on the VRS was lower and the sedation scores were higher in the elderly. Total oxycodone consumption during five postoperative days was reduced by 48% and the CAM-ICU scores were higher on the first postoperative day in the pregabalin group. The incidence of postoperative pain during movement was lower in the pregabalin group three months after surgery. This investigation demonstrates that chronic pain did not seem to impair daily activities in home-dwelling Finnish elderly. The VRS appeared to be applicable for elderly patients with clear cognitive dysfunction (MMSE ≤17) and it was the most feasible pain scale for the early postoperative period after cardiac surgery. After cardiac surgery, plasma concentrations of fentanyl in elderly were elevated, although oxycodone concentrations were at similar level compared to middle-aged patients. The elderly had less pain and were more sedated after doses of oxycodone. Therefore, particular attention must be given to individual dosing of the opioids in elderly surgical patients, who often need a smaller amount for adequate analgesia than middle-aged patients. The administration of pregabalin reduced postoperative oxycodone consumption after cardiac surgery. Pregabalin-treated patients had less confusion, and additionally to less postoperative pain on the first postoperative day and during movement at three months post-surgery. Pregabalin might be a new alternative as analgesic for acute postoperative and chronic pain management in the elderly. Its clinical role and safety remains to be verified in large-scale randomized and controlled studies. In the future, many clinical trials in the older category of patients will be needed to facilitate improvements in health care methods.

Turning visual search time on its head

Our everyday visual experience frequently involves searching for objects in clutter. Why are some searches easy and others hard? It is generally believed that the time taken to find a target increases as it becomes similar to its surrounding distractors. Here, I show that while this is qualitatively true, the exact relationship is in fact not linear. In a simple search experiment, when subjects searched for a bar differing in orientation from its distractors, search time was inversely proportional to the angular difference in orientation. Thus, rather than taking search reaction time (RT) to be a measure of target-distractor similarity, we can literally turn search time on its head (i.e. take its reciprocal 1/RT) to obtain a measure of search dissimilarity that varies linearly over a large range of target-distractor differences. I show that this dissimilarity measure has the properties of a distance metric, and report two interesting insights come from this measure: First, for a large number of searches, search asymmetries are relatively rare and when they do occur, differ by a fixed distance. Second, search distances can be used to elucidate object representations that underlie search - for example, these representations are roughly invariant to three-dimensional view. Finally, search distance has a straightforward interpretation in the context of accumulator models of search, where it is proportional to the discriminative signal that is integrated to produce a response. This is consistent with recent studies that have linked this distance to neuronal discriminability in visual cortex. Thus, while search time remains the more direct measure of visual search, its reciprocal also has the potential for interesting and novel insights. (C) 2012 Elsevier Ltd. All rights reserved.

Signature-tagged transposon mutagenesis studies demonstrate the dynamic nature of cecal colonization of 2-week-old chickens by Campylobacter jejuni.

We have constructed plasmids to be used for in vitro signature-tagged mutagenesis (STM) of Campylobacter jejuni and used these to generate STM libraries in three different strains. Statistical analysis of the transposon insertion sites in the C. jejuni NCTC 11168 chromosome and the plasmids of strain 81-176 indicated that their distribution was not uniform. Visual inspection of the distribution suggested that deviation from uniformity was not due to preferential integration of the transposon into a limited number of hot spots but rather that there was a bias towards insertions around the origin. We screened pools of mutants from the STM libraries for their ability to colonize the ceca of 2-week-old chickens harboring a standardized gut flora. We observed high-frequency random loss of colonization proficient mutants. When cohoused birds were individually inoculated with different tagged mutants, random loss of colonization-proficient mutants was similarly observed, as was extensive bird-to-bird transmission of mutants. This indicates that the nature of campylobacter colonization in chickens is complex and dynamic, and we hypothesize that bottlenecks in the colonization process and between-bird transmission account for these observations.

Analysis of temporal structure in spike trains of visual cortical area MT

The temporal structure of neuronal spike trains in the visual cortex can provide detailed information about the stimulus and about the neuronal implementation of visual processing. Spike trains recorded from the macaque motion area MT in previous studies (Newsome et al., 1989a; Britten et al., 1992; Zohary et al., 1994) are analyzed here in the context of the dynamic random dot stimulus which was used to evoke them. If the stimulus is incoherent, the spike trains can be highly modulated and precisely locked in time to the stimulus. In contrast, the coherent motion stimulus creates little or no temporal modulation and allows us to study patterns in the spike train that may be intrinsic to the cortical circuitry in area MT. Long gaps in the spike train evoked by the preferred direction motion stimulus are found, and they appear to be symmetrical to bursts in the response to the anti-preferred direction of motion. A novel cross-correlation technique is used to establish that the gaps are correlated between pairs of neurons. Temporal modulation is also found in psychophysical experiments using a modified stimulus. A model is made that can account for the temporal modulation in terms of the computational theory of biological image motion processing. A frequency domain analysis of the stimulus reveals that it contains a repeated power spectrum that may account for psychophysical and electrophysiological observations.

Some neurons tend to fire bursts of action potentials while others avoid burst firing. Using numerical and analytical models of spike trains as Poisson processes with the addition of refractory periods and bursting, we are able to account for peaks in the power spectrum near 40 Hz without assuming the existence of an underlying oscillatory signal. A preliminary examination of the local field potential reveals that stimulus-locked oscillation appears briefly at the beginning of the trial.

Social saliency : visual psychophysics and single-neuron recordings in humans

My thesis studies how people pay attention to other people and the environment. How does the brain figure out what is important and what are the neural mechanisms underlying attention? What is special about salient social cues compared to salient non-social cues? In Chapter I, I review social cues that attract attention, with an emphasis on the neurobiology of these social cues. I also review neurological and psychiatric links: the relationship between saliency, the amygdala and autism. The first empirical chapter then begins by noting that people constantly move in the environment. In Chapter II, I study the spatial cues that attract attention during locomotion using a cued speeded discrimination task. I found that when the motion was expansive, attention was attracted towards the singular point of the optic flow (the focus of expansion, FOE) in a sustained fashion. The more ecologically valid the motion features became (e.g., temporal expansion of each object, spatial depth structure implied by distribution of the size of the objects), the stronger the attentional effects. However, compared to inanimate objects and cues, people preferentially attend to animals and faces, a process in which the amygdala is thought to play an important role. To directly compare social cues and non-social cues in the same experiment and investigate the neural structures processing social cues, in Chapter III, I employ a change detection task and test four rare patients with bilateral amygdala lesions. All four amygdala patients showed a normal pattern of reliably faster and more accurate detection of animate stimuli, suggesting that advantageous processing of social cues can be preserved even without the amygdala, a key structure of the “social brain”. People not only attend to faces, but also pay attention to others’ facial emotions and analyze faces in great detail. Humans have a dedicated system for processing faces and the amygdala has long been associated with a key role in recognizing facial emotions. In Chapter IV, I study the neural mechanisms of emotion perception and find that single neurons in the human amygdala are selective for subjective judgment of others’ emotions. Lastly, people typically pay special attention to faces and people, but people with autism spectrum disorders (ASD) might not. To further study social attention and explore possible deficits of social attention in autism, in Chapter V, I employ a visual search task and show that people with ASD have reduced attention, especially social attention, to target-congruent objects in the search array. This deficit cannot be explained by low-level visual properties of the stimuli and is independent of the amygdala, but it is dependent on task demands. Overall, through visual psychophysics with concurrent eye-tracking, my thesis found and analyzed socially salient cues and compared social vs. non-social cues and healthy vs. clinical populations. Neural mechanisms underlying social saliency were elucidated through electrophysiology and lesion studies. I finally propose further research questions based on the findings in my thesis and introduce my follow-up studies and preliminary results beyond the scope of this thesis in the very last section, Future Directions.

O ser negro à vista: construção verbo-visual do negro na propaganda impressa

Esta tese destina-se a desenvolver estudo semiótico de propagandas impressas em que a pessoa negra é posta em presença. Investiga-se como a propaganda veiculada em revista, mediante seleção e combinação sígnicas, efetiva a construção da imagem verbo-visual do negro, tendo em vista o produto anunciado e os projetos comunicativos do enunciador e, assim, como finda por ratificar ou (re)elaborar significados sociais acerca desse sujeito. Pautando-se na Semiótica de Charles Sanders Peirce e tendo por suporte, fundamentalmente, a Teoria da Iconicidade Verbal de Simões (2009), a pesquisa aborda todos os signos verbais e não verbais em diálogo, como dotados de potencial icônico, não só revelador dos projetos de texto, como também ativador de interpretações/leituras e, ainda, delineador de sentidos, posteriormente cristalizados e convertidos em significados imanentes aos signos e aos objetos reportados. Por ter, como material constitutivo do corpus, textos elaborados em linguagem mista, a pesquisa propõe a aplicação da Teoria da Iconicidade Verbal ao universo dos signos lato sensu. O debate apresenta o texto de propaganda como excelente material, não só para implementar os estudos de História e Cultura Afro-brasileiras, como prevê a Lei 10.639/03, como também para subsidiar o ensino de língua portuguesa, com ênfase para os estudos sobre leitura e produção textual, para que se forme sujeito dotado de habilidades que lhe permitam reconhecer no verbal e no não verbal a revelação e a geração de sentidos sociais

1. Ultrasonic studies of binary liquid structure in the critical region. Theory and experiment for the 2,6-lutidine/water system. 2. Hartree-Fock calculations of electric polarizabilities of some simple atoms and molecules, and their practicality. 3. Calculation of vibrational transition probabilities in collinear atom-diatom and diatom-diatom collisions with Lennard-Jones interaction

Part 1. Many interesting visual and mechanical phenomena occur in the critical region of fluids, both for the gas-liquid and liquid-liquid transitions. The precise thermodynamic and transport behavior here has some broad consequences for the molecular theory of liquids. Previous studies in this laboratory on a liquid-liquid critical mixture via ultrasonics supported a basically classical analysis of fluid behavior by M. Fixman (e. g., the free energy is assumed analytic in intensive variables in the thermodynamics)--at least when the fluid is not too close to critical. A breakdown in classical concepts is evidenced close to critical, in some well-defined ways. We have studied herein a liquid-liquid critical system of complementary nature (possessing a lower critical mixing or consolute temperature) to all previous mixtures, to look for new qualitative critical behavior. We did not find such new behavior in the ultrasonic absorption ascribable to the critical fluctuations, but we did find extra absorption due to chemical processes (yet these are related to the mixing behavior generating the lower consolute point). We rederived, corrected, and extended Fixman's analysis to interpret our experimental results in these more complex circumstances. The entire account of theory and experiment is prefaced by an extensive introduction recounting the general status of liquid state theory. The introduction provides a context for our present work, and also points out problems deserving attention. Interest in these problems was stimulated by this work but also by work in Part 3.

Part 2. Among variational theories of electronic structure, the Hartree-Fock theory has proved particularly valuable for a practical understanding of such properties as chemical binding, electric multipole moments, and X-ray scattering intensity. It also provides the most tractable method of calculating first-order properties under external or internal one-electron perturbations, either developed explicitly in orders of perturbation theory or in the fully self-consistent method. The accuracy and consistency of first-order properties are poorer than those of zero-order properties, but this is most often due to the use of explicit approximations in solving the perturbed equations, or to inadequacy of the variational basis in size or composition. We have calculated the electric polarizabilities of H₂, He, Li, Be, LiH, and N₂ by Hartree-Fock theory, using exact perturbation theory or the fully self-consistent method, as dictated by convenience. By careful studies on total basis set composition, we obtained good approximations to limiting Hartree-Fock values of polarizabilities with bases of reasonable size. The values for all species, and for each direction in the molecular cases, are within 8% of experiment, or of best theoretical values in the absence of the former. Our results support the use of unadorned Hartree-Pock theory for static polarizabilities needed in interpreting electron-molecule scattering data, collision-induced light scattering experiments, and other phenomena involving experimentally inaccessible polarizabilities.

Part 3. Numerical integration of the close-coupled scattering equations has been carried out to obtain vibrational transition probabilities for some models of the electronically adiabatic H₂-H₂ collision. All the models use a Lennard-Jones interaction potential between nearest atoms in the collision partners. We have analyzed the results for some insight into the vibrational excitation process in its dependence on the energy of collision, the nature of the vibrational binding potential, and other factors. We conclude also that replacement of earlier, simpler models of the interaction potential by the Lennard-Jones form adds very little realism for all the complication it introduces. A brief introduction precedes the presentation of our work and places it in the context of attempts to understand the collisional activation process in chemical reactions as well as some other chemical dynamics.

A crowding model of visual clutter.

Visual information is difficult to search and interpret when the density of the displayed information is high or the layout is chaotic. Visual information that exhibits such properties is generally referred to as being "cluttered." Clutter should be avoided in information visualizations and interface design in general because it can severely degrade task performance. Although previous studies have identified computable correlates of clutter (such as local feature variance and edge density), understanding of why humans perceive some scenes as being more cluttered than others remains limited. Here, we explore an account of clutter that is inspired by findings from visual perception studies. Specifically, we test the hypothesis that the so-called "crowding" phenomenon is an important constituent of clutter. We constructed an algorithm to predict visual clutter in arbitrary images by estimating the perceptual impairment due to crowding. After verifying that this model can reproduce crowding data we tested whether it can also predict clutter. We found that its predictions correlate well with both subjective clutter assessments and search performance in cluttered scenes. These results suggest that crowding and clutter may indeed be closely related concepts and suggest avenues for further research.

Expressive visual text-to-speech using active appearance models

This paper presents a complete system for expressive visual text-to-speech (VTTS), which is capable of producing expressive output, in the form of a 'talking head', given an input text and a set of continuous expression weights. The face is modeled using an active appearance model (AAM), and several extensions are proposed which make it more applicable to the task of VTTS. The model allows for normalization with respect to both pose and blink state which significantly reduces artifacts in the resulting synthesized sequences. We demonstrate quantitative improvements in terms of reconstruction error over a million frames, as well as in large-scale user studies, comparing the output of different systems. © 2013 IEEE.

Texture Segregation By Visual Cortex: Perceptual Grouping, Attention, and Learning

A neural model is proposed of how laminar interactions in the visual cortex may learn and recognize object texture and form boundaries. The model brings together five interacting processes: region-based texture classification, contour-based boundary grouping, surface filling-in, spatial attention, and object attention. The model shows how form boundaries can determine regions in which surface filling-in occurs; how surface filling-in interacts with spatial attention to generate a form-fitting distribution of spatial attention, or attentional shroud; how the strongest shroud can inhibit weaker shrouds; and how the winning shroud regulates learning of texture categories, and thus the allocation of object attention. The model can discriminate abutted textures with blurred boundaries and is sensitive to texture boundary attributes like discontinuities in orientation and texture flow curvature as well as to relative orientations of texture elements. The model quantitatively fits a large set of human psychophysical data on orientation-based textures. Object boundar output of the model is compared to computer vision algorithms using a set of human segmented photographic images. The model classifies textures and suppresses noise using a multiple scale oriented filterbank and a distributed Adaptive Resonance Theory (dART) classifier. The matched signal between the bottom-up texture inputs and top-down learned texture categories is utilized by oriented competitive and cooperative grouping processes to generate texture boundaries that control surface filling-in and spatial attention. Topdown modulatory attentional feedback from boundary and surface representations to early filtering stages results in enhanced texture boundaries and more efficient learning of texture within attended surface regions. Surface-based attention also provides a self-supervising training signal for learning new textures. Importance of the surface-based attentional feedback in texture learning and classification is tested using a set of textured images from the Brodatz micro-texture album. Benchmark studies vary from 95.1% to 98.6% with attention, and from 90.6% to 93.2% without attention.