Perception of co-speech gestures in aphasic patients: A visual exploration study during the observation of dyadic conversations

Background: Co-speech gestures are part of nonverbal communication during conversations. They either support the verbal message or provide the interlocutor with additional information. Furthermore, they prompt as nonverbal cues the cooperative process of turn taking. In the present study, we investigated the influence of co-speech gestures on the perception of dyadic dialogue in aphasic patients. In particular, we analysed the impact of co-speech gestures on gaze direction (towards speaker or listener) and fixation of body parts. We hypothesized that aphasic patients, who are restricted in verbal comprehension, adapt their visual exploration strategies. Methods: Sixteen aphasic patients and 23 healthy control subjects participated in the study. Visual exploration behaviour was measured by means of a contact-free infrared eye-tracker while subjects were watching videos depicting spontaneous dialogues between two individuals. Cumulative fixation duration and mean fixation duration were calculated for the factors co-speech gesture (present and absent), gaze direction (to the speaker or to the listener), and region of interest (ROI), including hands, face, and body. Results: Both aphasic patients and healthy controls mainly fixated the speaker’s face. We found a significant co-speech gesture x ROI interaction, indicating that the presence of a co-speech gesture encouraged subjects to look at the speaker. Further, there was a significant gaze direction x ROI x group interaction revealing that aphasic patients showed reduced cumulative fixation duration on the speaker’s face compared to healthy controls. Conclusion: Co-speech gestures guide the observer’s attention towards the speaker, the source of semantic input. It is discussed whether an underlying semantic processing deficit or a deficit to integrate audio-visual information may cause aphasic patients to explore less the speaker’s face. Keywords: Gestures, visual exploration, dialogue, aphasia, apraxia, eye movements

The SanaViz: Human Centered Geovisualization to facilitate visual exploration of Public Health data

These three manuscripts are presented as a PhD dissertation for the study of using GeoVis application to evaluate telehealth programs. The primary reason of this research was to understand how the GeoVis applications can be designed and developed using combined approaches of HC approach and cognitive fit theory and in terms utilized to evaluate telehealth program in Brazil. First manuscript The first manuscript in this dissertation presented a background about the use of GeoVisualization to facilitate visual exploration of public health data. The manuscript covered the existing challenges that were associated with an adoption of existing GeoVis applications. The manuscript combines the principles of Human Centered approach and Cognitive Fit Theory and a framework using a combination of these approaches is developed that lays the foundation of this research. The framework is then utilized to propose the design, development and evaluation of “the SanaViz” to evaluate telehealth data in Brazil, as a proof of concept. Second manuscript The second manuscript is a methods paper that describes the approaches that can be employed to design and develop “the SanaViz” based on the proposed framework. By defining the various elements of the HC approach and CFT, a mixed methods approach is utilized for the card sorting and sketching techniques. A representative sample of 20 study participants currently involved in the telehealth program at the NUTES telehealth center at UFPE, Recife, Brazil was enrolled. The findings of this manuscript helped us understand the needs of the diverse group of telehealth users, the tasks that they perform and helped us determine the essential features that might be necessary to be included in the proposed GeoVis application “the SanaViz”. Third manuscript The third manuscript involved mix- methods approach to compare the effectiveness and usefulness of the HC GeoVis application “the SanaViz” against a conventional GeoVis application “Instant Atlas”. The same group of 20 study participants who had earlier participated during Aim 2 was enrolled and a combination of quantitative and qualitative assessments was done. Effectiveness was gauged by the time that the participants took to complete the tasks using both the GeoVis applications, the ease with which they completed the tasks and the number of attempts that were taken to complete each task. Usefulness was assessed by System Usability Scale (SUS), a validated questionnaire tested in prior studies. In-depth interviews were conducted to gather opinions about both the GeoVis applications. This manuscript helped us in the demonstration of the usefulness and effectiveness of HC GeoVis applications to facilitate visual exploration of telehealth data, as a proof of concept. Together, these three manuscripts represent challenges of combining principles of Human Centered approach, Cognitive Fit Theory to design and develop GeoVis applications as a method to evaluate Telehealth data. To our knowledge, this is the first study to explore the usefulness and effectiveness of GeoVis to facilitate visual exploration of telehealth data. The results of the research enabled us to develop a framework for the design and development of GeoVis applications related to the areas of public health and especially telehealth. The results of our study showed that the varied users were involved with the telehealth program and the tasks that they performed. Further it enabled us to identify the components that might be essential to be included in these GeoVis applications. The results of our research answered the following questions; (a) Telehealth users vary in their level of understanding about GeoVis (b) Interaction features such as zooming, sorting, and linking and multiple views and representation features such as bar chart and choropleth maps were considered the most essential features of the GeoVis applications. (c) Comparing and sorting were two important tasks that the telehealth users would perform for exploratory data analysis. (d) A HC GeoVis prototype application is more effective and useful for exploration of telehealth data than a conventional GeoVis application. Future studies should be done to incorporate the proposed HC GeoVis framework to enable comprehensive assessment of the users and the tasks they perform to identify the features that might be necessary to be a part of the GeoVis applications. The results of this study demonstrate a novel approach to comprehensively and systematically enhance the evaluation of telehealth programs using the proposed GeoVis Framework.

Analysis of exploratory eye movements in patients with schizophrenia during visual scanning of projective tests' figures

OBJECTIVE: Compare pattern of exploratory eye movements during visual scanning of the Rorschach and TAT test cards in people with schizophrenia and controls. METHOD: 10 participants with schizophrenia and 10 controls matched by age, schooling and intellectual level participated in the study. Severity of symptoms was evaluated with the Positive and Negative Syndrome Scale. Test cards were divided into three groups: TAT cards with scenes content, TAT cards with interaction content (TAT-faces), and Rorschach cards with abstract images. Eye movements were analyzed for: total number, duration and location of fixation; and length of saccadic movements. RESULTS: Different pattern of eye movement was found, with schizophrenia participants showing lower number of fixations but longer fixation duration in Rorschach cards and TAT-faces. The biggest difference was observed in Rorschach, followed by TAT-faces and TAT-scene cards. CONCLUSIONS: Results suggest alteration in visual exploration mechanisms possibly related to integration of abstract visual information.

Interaction of visual and haptic information in simulated environments: texture perception

This paper describes experiments relating to the perception of the roughness of simulated surfaces via the haptic and visual senses. Subjects used a magnitude estimation technique to judge the roughness of “virtual gratings” presented via a PHANToM haptic interface device, and a standard visual display unit. It was shown that under haptic perception, subjects tended to perceive roughness as decreasing with increased grating period, though this relationship was not always statistically significant. Under visual exploration, the exact relationship between spatial period and perceived roughness was less well defined, though linear regressions provided a reliable approximation to individual subjects’ estimates.

The BioDICE Taverna plugin for clustering and visualization of biological data: a workflow for molecular compounds exploration

Background: In many experimental pipelines, clustering of multidimensional biological datasets is used to detect hidden structures in unlabelled input data. Taverna is a popular workflow management system that is used to design and execute scientific workflows and aid in silico experimentation. The availability of fast unsupervised methods for clustering and visualization in the Taverna platform is important to support a data-driven scientific discovery in complex and explorative bioinformatics applications. Results: This work presents a Taverna plugin, the Biological Data Interactive Clustering Explorer (BioDICE), that performs clustering of high-dimensional biological data and provides a nonlinear, topology preserving projection for the visualization of the input data and their similarities. The core algorithm in the BioDICE plugin is Fast Learning Self Organizing Map (FLSOM), which is an improved variant of the Self Organizing Map (SOM) algorithm. The plugin generates an interactive 2D map that allows the visual exploration of multidimensional data and the identification of groups of similar objects. The effectiveness of the plugin is demonstrated on a case study related to chemical compounds. Conclusions: The number and variety of available tools and its extensibility have made Taverna a popular choice for the development of scientific data workflows. This work presents a novel plugin, BioDICE, which adds a data-driven knowledge discovery component to Taverna. BioDICE provides an effective and powerful clustering tool, which can be adopted for the explorative analysis of biological datasets.

Visual analysis of image collections

Multidimensional Visualization techniques are invaluable tools for analysis of structured and unstructured data with variable dimensionality. This paper introduces PEx-Image-Projection Explorer for Images-a tool aimed at supporting analysis of image collections. The tool supports a methodology that employs interactive visualizations to aid user-driven feature detection and classification tasks, thus offering improved analysis and exploration capabilities. The visual mappings employ similarity-based multidimensional projections and point placement to layout the data on a plane for visual exploration. In addition to its application to image databases, we also illustrate how the proposed approach can be successfully employed in simultaneous analysis of different data types, such as text and images, offering a common visual representation for data expressed in different modalities.

Ergonomia de tablets em uso ocupacional: atividade visual e biomecânica postural sob a influência de ofuscamentos reflexivos causados pela iluminação

Pós-graduação em Design - FAAC

Multidimensional Projections for Visual Analysis of Social Networks

Visual analysis of social networks is usually based on graph drawing algorithms and tools. However, social networks are a special kind of graph in the sense that interpretation of displayed relationships is heavily dependent on context. Context, in its turn, is given by attributes associated with graph elements, such as individual nodes, edges, and groups of edges, as well as by the nature of the connections between individuals. In most systems, attributes of individuals and communities are not taken into consideration during graph layout, except to derive weights for force-based placement strategies. This paper proposes a set of novel tools for displaying and exploring social networks based on attribute and connectivity mappings. These properties are employed to layout nodes on the plane via multidimensional projection techniques. For the attribute mapping, we show that node proximity in the layout corresponds to similarity in attribute, leading to easiness in locating similar groups of nodes. The projection based on connectivity yields an initial placement that forgoes force-based or graph analysis algorithm, reaching a meaningful layout in one pass. When a force algorithm is then applied to this initial mapping, the final layout presents better properties than conventional force-based approaches. Numerical evaluations show a number of advantages of pre-mapping points via projections. User evaluation demonstrates that these tools promote ease of manipulation as well as fast identification of concepts and associations which cannot be easily expressed by conventional graph visualization alone. In order to allow better space usage for complex networks, a graph mapping on the surface of a sphere is also implemented.

The gender typicality of faces and its impact on visual processing and on hiring decisions

Past research has shown that the gender typicality of applicants’ faces affects leadership selection irrespective of a candidate’s gender: A masculine facial appearance is congruent with masculine-typed leadership roles, thus masculine-looking applicants are hired more certainly than feminine-looking ones. In the present study, we extended this line of research by investigating hiring decisions for both masculine- and feminine-typed professional roles. Furthermore, we used eye tracking to examine the visual exploration of applicants’ portraits. Our results indicate that masculine-looking applicants were favored for the masculine-typed role (leader) and feminine-looking applicants for the feminine-typed role (team member). Eye movement patterns showed that information about gender category and facial appearance was integrated during first fixations of the portraits. Hiring decisions, however, were not based on this initial analysis, but occurred at a second stage, when the portrait was viewed in the context of considering the applicant for a specific job.

The influence of naturalistic, directionally non-specific motion on the spatial deployment of visual attention in right-hemispheric stroke

An impairment of the spatial deployment of visual attention during exploration of static (i.e., motionless) stimuli is a common finding after an acute, right-hemispheric stroke. However, less is known about how these deficits: a) are modulated through naturalistic motion (i.e., without directional, specific spatial features); and, b) evolve in the subacute/chronic post-stroke phase. In the present study, we investigated free visual exploration in three patient groups with subacute/chronic right-hemispheric stroke and in healthy subjects. The first group included patients with left visual neglect and a left visual field defect (VFD), the second patients with a left VFD but no neglect, and the third patients without neglect or VFD. Eye movements were measured in all participants while they freely explored a traffic scene without (static condition) and with (dynamic condition) naturalistic motion, i.e., cars moving from the right or left. In the static condition, all patient groups showed similar deployment of visual exploration (i.e., as measured by the cumulative fixation duration) as compared to healthy subjects, suggesting that recovery processes took place, with normal spatial allocation of attention. However, the more demanding dynamic condition with moving cars elicited different re-distribution patterns of visual attention, quite similar to those typically observed in acute stroke. Neglect patients with VFD showed a significant decrease of visual exploration in the contralesional space, whereas patients with VFD but no neglect showed a significant increase of visual exploration in the contralesional space. No differences, as compared to healthy subjects, were found in patients without neglect or VFD. These results suggest that naturalistic motion, without directional, specific spatial features, may critically influence the spatial distribution of visual attention in subacute/chronic stroke patients.

Exploratory geospatial data analysis using self-organizing maps

Dissertação apresentada como requisito parcial para obtenção do grau de Mestre em Ciência e Sistemas de Informação Geográfica

Attachment style impacts behavior and early oculomotor response to positive, but not negative, pictures

The present study investigated whether oculomotor behavior is influenced by attachment styles. The Relationship Scales Questionnaire was used to assess attachment styles of forty-eight voluntary university students and to classify them into attachment groups (secure, preoccupied, fearful, and dismissing). Eye-tracking was recorded while participants engaged in a 3-seconds free visual exploration of stimuli presenting either a positive or a negative picture together with a neutral picture, all depicting social interactions. The task consisted in identifying whether the two pictures depicted the same emotion. Results showed that the processing of negative pictures was impermeable to attachment style, while the processing of positive pictures was significantly influenced by individual differences in insecure attachment. The groups highly avoidant regarding to attachment (dismissing and fearful) showed reduced accuracy, suggesting a higher threshold for recognizing positive emotions compared to the secure group. The groups with higher attachment anxiety (preoccupied and fearful) showed differences in automatic capture of attention, in particular an increased delay preceding the first fixation to a picture of positive emotional valence. Despite lenient statistical thresholds induced by the limited sample size of some groups (p < 0.05 uncorrected for multiple comparisons), the current findings suggest that the processing of positive emotions is affected by attachment styles. These results are discussed within a broader evolutionary framework.

Vision, spatial cognition and intellectual disability.

Vision is the most synthetic sensory channel and it provides specific information about the relative position of distant landmarks during visual exploration. In this paper we propose that visual exploration, as assessed by the recording of eye movements, offers an original method to analyze spatial cognition and to reveal alternative adaptation strategies in people with intellectual disabilities (ID). Our general assumption is that eye movement exploration may simultaneously reveal whether, why, and how, compensatory strategies point to specific difficulties related to neurological symptoms. An understanding of these strategies will also help in the development of optimal rehabilitation procedures.

Validation d’une situation de stimulation suscitant des comportements stéréotypés et des intérêts restreints chez le jeune enfant autiste

Les comportements stéréotypés et les intérêts restreints sont des comportements à valeur diagnostique dans l’autisme. Pourtant, il y a des lacunes en clinique, dans la façon de détecter ces comportements, considérant l’absence d’instruments standardisés les suscitant et en recherche, dans la façon de documenter ces comportements pour arriver à les définir de façon opérationnelle. Cette thèse a pour objectif de mieux documenter, par une situation d’observation, les comportements stéréotypés et les intérêts restreints en bas âge dans l’autisme, et de permettre l’utilisation de cette situation en clinique. Deux étapes préliminaires ont permis de documenter les comportements stéréotypés et les intérêts restreints en bas âge dans l’autisme. La première, l’élaboration d’un questionnaire sur les comportements stéréotypés et les intérêts restreints et les objets qui les déclenchent complété par des experts dans le domaine. Ce questionnaire a permis de construire la grille de cotation et la situation de stimulation. La seconde la construction d’une grille de cotation qui apporte une définition opérationnelle des comportements stéréotypés et des intérêts restreints en bas âge dans l’autisme et vise à les colliger. L’étape principale de la présente recherche consiste en l’élaboration d’une situation de stimulation suscitant des comportements stéréotypés et des intérêts restreints par l’exposition à des objets qui les déclenchent. Cette situation a permis de documenter, par observation, les comportements stéréotypés et les intérêts restreints en bas âge dans l’autisme. La validation de la situation de stimulation a été appliquée auprès de deux groupes d’enfants âgés de 24 à 72 mois appariés en âge chronologique, 21 enfants portant un diagnostic d’autisme et 24 enfants au développement typique Les résultats montrent que la situation de stimulation est un instrument suffisamment sensible pour détecter des comportements stéréotypés et des intérêts restreints en bas âge dans l’autisme et d’identifier des objets d’intérêt. En effet, lors de l’exposition à la situation de stimulation, les enfants autistes se distinguent des enfants typiques sur la base du nombre et de la durée des comportements stéréotypés et des intérêts restreints qu’ils présentent. Les enfants autistes montrent une fréquence significativement plus élevé pour les CSIR suivants: maniérismes des mains et des doigts, crispation des doigts, sautillement, doigts dans la bouche, objets dans la bouche, exploration visuelle: regard rapproché, met les objets en mouvement non circulaire. Les enfants autistes se distinguent également des enfants typiques sur la base de l’exploration des objets, en fréquence et en durée, significativement, pour les objets: Bateau: marteau et balles et lettres et chiffres. Cette étude est la première qui passe par un protocole d’observation systématique, pour documenter les comportements stéréotypés et les intérêts restreints, ainsi que les objets qui les déclenchent, des objets d’intérêt, en bas âge dans l’autisme. Cette situation pourrait ultimement faire partie du processus d’évaluation diagnostique ou de dépistage de l’autisme permettant d’identifier en bas âge des enfants autistes ou à risque d’autisme.

Les apports de la réalité virtuelle et de la vidéo-oculographie à l’évaluation des intérêts sexuels

La pléthysmographie pénienne est reconnue comme la méthode la plus rigoureuse pour l’évaluation des intérêts sexuels. Cet instrument pourrait cependant bénéficier d’amélioration au niveau de sa validité écologique et de sa vulnérabilité aux tentatives de falsification. Cet outil est également confronté aux contraintes éthiques et légales liées à la présentation de stimuli produits à partir de modèle humain pour susciter l’excitation sexuelle. La présente thèse doctorale propose de pallier aux lacunes de la procédure pléthysmographique en la combinant à des stimuli générés par ordinateur présentés en immersion virtuelle et à la vidéo-oculographie. Dans le cadre d’une première étude empirique, il s’agit de déterminer si l’usage de stimuli générés par ordinateur présentés en immersion virtuelle est préférable à la modalité auditive actuellement utilisée. Les réponses érectiles d’agresseurs sexuels d’enfants et d’hommes sans paraphilie connue sont enregistrées et la capacité des deux modalités à produire des profils d’excitation sexuels et des indices de déviance propres aux intérêts sexuels est comparée. Dans le cadre d’une seconde étude expérimentale, les mouvements oculaires sont analysés afin de détecter les tentatives de contrôle volontaire de la réponse érectile lors d’une procédure d’évaluation pléthysmographique. Les réponses érectiles et les mouvements oculaires d’hommes sans dossier judiciaire sont enregistrés lors du visionnement de stimuli sous trois différentes conditions, dont une tâche de suppression de la réponse érectile. Dans l’ensemble, les résultats démontrent que les stimuli générés par ordinateur présentés en immersion virtuelle génèrent des profils d’excitation sexuelle ainsi que des indices de déviance ayant une précision de classification et de discrimination significativement supérieure à la modalité auditive. D’autre part, les résultats soulignent la capacité des mouvements oculaires à identifier une signature oculaire propre à l’utilisation d’une stratégie cognitive d’inhibition de la réponse érectile qui se caractérise notamment par la mise à l’écart du contenu sexuel et le ralentissement global du processus d’exploration visuel. Ce projet de recherche souligne les avantages inhérents à la présentation en immersion virtuelle de stimuli générés par ordinateur ainsi qu’à l’étude des mouvements oculaires lors de l’évaluation pléthysmographique et vient appuyer l’implantation progressive de cette modalité dans les milieux d’évaluations cliniques.