Visual search and eye movements: Studies of perceptual span

In visual search one tries to find the currently relevant item among other, irrelevant items. In the present study, visual search performance for complex objects (characters, faces, computer icons and words) was investigated, and the contribution of different stimulus properties, such as luminance contrast between characters and background, set size, stimulus size, colour contrast, spatial frequency, and stimulus layout were investigated. Subjects were required to search for a target object among distracter objects in two-dimensional stimulus arrays. The outcome measure was threshold search time, that is, the presentation duration of the stimulus array required by the subject to find the target with a certain probability. It reflects the time used for visual processing separated from the time used for decision making and manual reactions. The duration of stimulus presentation was controlled by an adaptive staircase method. The number and duration of eye fixations, saccade amplitude, and perceptual span, i.e., the number of items that can be processed during a single fixation, were measured. It was found that search performance was correlated with the number of fixations needed to find the target. Search time and the number of fixations increased with increasing stimulus set size. On the other hand, several complex objects could be processed during a single fixation, i.e., within the perceptual span. Search time and the number of fixations depended on object type as well as luminance contrast. The size of the perceptual span was smaller for more complex objects, and decreased with decreasing luminance contrast within object type, especially for very low contrasts. In addition, the size and shape of perceptual span explained the changes in search performance for different stimulus layouts in word search. Perceptual span was scale invariant for a 16-fold range of stimulus sizes, i.e., the number of items processed during a single fixation was independent of retinal stimulus size or viewing distance. It is suggested that saccadic visual search consists of both serial (eye movements) and parallel (processing within perceptual span) components, and that the size of the perceptual span may explain the effectiveness of saccadic search in different stimulus conditions. Further, low-level visual factors, such as the anatomical structure of the retina, peripheral stimulus visibility and resolution requirements for the identification of different object types are proposed to constrain the size of the perceptual span, and thus, limit visual search performance. Similar methods were used in a clinical study to characterise the visual search performance and eye movements of neurological patients with chronic solvent-induced encephalopathy (CSE). In addition, the data about the effects of different stimulus properties on visual search in normal subjects were presented as simple practical guidelines, so that the limits of human visual perception could be taken into account in the design of user interfaces.

Studies of working memory in interrupted human-computer interaction

Distraction in the workplace is increasingly more common in the information age. Several tasks and sources of information compete for a worker's limited cognitive capacities in human-computer interaction (HCI). In some situations even very brief interruptions can have detrimental effects on memory. Nevertheless, in other situations where persons are continuously interrupted, virtually no interruption costs emerge. This dissertation attempts to reveal the mental conditions and causalities differentiating the two outcomes. The explanation, building on the theory of long-term working memory (LTWM; Ericsson and Kintsch, 1995), focuses on the active, skillful aspects of human cognition that enable the storage of task information beyond the temporary and unstable storage provided by short-term working memory (STWM). Its key postulate is called a retrieval structure an abstract, hierarchical knowledge representation built into long-term memory that can be utilized to encode, update, and retrieve products of cognitive processes carried out during skilled task performance. If certain criteria of practice and task processing are met, LTWM allows for the storage of large representations for long time periods, yet these representations can be accessed with the accuracy, reliability, and speed typical of STWM. The main thesis of the dissertation is that the ability to endure interruptions depends on the efficiency in which LTWM can be recruited for maintaing information. An observational study and a field experiment provide ecological evidence for this thesis. Mobile users were found to be able to carry out heavy interleaving and sequencing of tasks while interacting, and they exhibited several intricate time-sharing strategies to orchestrate interruptions in a way sensitive to both external and internal demands. Interruptions are inevitable, because they arise as natural consequences of the top-down and bottom-up control of multitasking. In this process the function of LTWM is to keep some representations ready for reactivation and others in a more passive state to prevent interference. The psychological reality of the main thesis received confirmatory evidence in a series of laboratory experiments. They indicate that after encoding into LTWM, task representations are safeguarded from interruptions, regardless of their intensity, complexity, or pacing. However, when LTWM cannot be deployed, the problems posed by interference in long-term memory and the limited capacity of the STWM surface. A major contribution of the dissertation is the analysis of when users must resort to poorer maintenance strategies, like temporal cues and STWM-based rehearsal. First, one experiment showed that task orientations can be associated with radically different patterns of retrieval cue encodings. Thus the nature of the processing of the interface determines which features will be available as retrieval cues and which must be maintained by other means. In another study it was demonstrated that if the speed of encoding into LTWM, a skill-dependent parameter, is slower than the processing speed allowed for by the task, interruption costs emerge. Contrary to the predictions of competing theories, these costs turned out to involve intrusions in addition to omissions. Finally, it was learned that in rapid visually oriented interaction, perceptual-procedural expectations guide task resumption, and neither STWM nor LTWM are utilized due to the fact that access is too slow. These findings imply a change in thinking about the design of interfaces. Several novel principles of design are presented, basing on the idea of supporting the deployment of LTWM in the main task.

Cortical processing of musical pitch as reflected by behavioural and electrophysiological evidence

In a musical context, the pitch of sounds is encoded according to domain-general principles not confined to music or even to audition overall but common to other perceptual and cognitive processes (such as multiple pattern encoding and feature integration), and to domain-specific and culture-specific properties related to a particular musical system only (such as the pitch steps of the Western tonal system). The studies included in this thesis shed light on the processing stages during which pitch encoding occurs on the basis of both domain-general and music-specific properties, and elucidate the putative brain mechanisms underlying pitch-related music perception. Study I showed, in subjects without formal musical education, that the pitch and timbre of multiple sounds are integrated as unified object representations in sensory memory before attentional intervention. Similarly, multiple pattern pitches are simultaneously maintained in non-musicians' sensory memory (Study II). These findings demonstrate the degree of sophistication of pitch processing at the sensory memory stage, requiring neither attention nor any special expertise of the subjects. Furthermore, music- and culture-specific properties, such as the pitch steps of the equal-tempered musical scale, are automatically discriminated in sensory memory even by subjects without formal musical education (Studies III and IV). The cognitive processing of pitch according to culture-specific musical-scale schemata hence occurs as early as at the sensory-memory stage of pitch analysis. Exposure and cortical plasticity seem to be involved in musical pitch encoding. For instance, after only one hour of laboratory training, the neural representations of pitch in the auditory cortex are altered (Study V). However, faulty brain mechanisms for attentive processing of fine-grained pitch steps lead to inborn deficits in music perception and recognition such as those encountered in congenital amusia (Study VI). These findings suggest that predispositions for exact pitch-step discrimination together with long-term exposure to music govern the acquisition of the automatized schematic knowledge of the music of a particular culture that even non-musicians possess.

Constructing skilled images

When experts construct mental images, they do not rely only on perceptual features; they also access domain-specific knowledge and skills in long-term memory, which enables them to exceed the capacity limitations of the short-term working memory system. The central question of the present dissertation was whether the facilitating effect of long-term memory knowledge on working memory imagery tasks is primarily based on perceptual chunking or whether it relies on higher-level conceptual knowledge. Three domains of expertise were studied: chess, music, and taxi driving. The effects of skill level, stimulus surface features, and the stimulus structure on incremental construction of mental images were investigated. A method was developed to capture the chunking mechanisms that experts use in constructing images: chess pieces, street names, and visual notes were presented in a piecemeal fashion for later recall. Over 150 experts and non-experts participated in a total of 13 experiments, as reported in five publications. The results showed skill effects in all of the studied domains when experts performed memory and problem solving tasks that required mental imagery. Furthermore, only experts' construction of mental images benefited from meaningful stimuli. Manipulation of the stimulus surface features, such as replacing chess pieces with dots, did not significantly affect experts' performance in the imagery tasks. In contrast, the structure of the stimuli had a significant effect on experts' performance in every task domain. For example, taxi drivers recalled more street names from lists that formed a spatially continuous route than from alphabetically organised lists. The results suggest that the mechanisms of conceptual chunking rather than automatic perceptual pattern matching underlie expert performance, even though the tasks of the present studies required perception-like mental representations. The results show that experts are able to construct skilled images that surpass working memory capacity, and that their images are conceptually organised and interpreted rather than merely depictive.

Feature integration in human vision

The earliest stages of human cortical visual processing can be conceived as extraction of local stimulus features. However, more complex visual functions, such as object recognition, require integration of multiple features. Recently, neural processes underlying feature integration in the visual system have been under intensive study. A specialized mid-level stage preceding the object recognition stage has been proposed to account for the processing of contours, surfaces and shapes as well as configuration. This thesis consists of four experimental, psychophysical studies on human visual feature integration. In two studies, classification image a recently developed psychophysical reverse correlation method was used. In this method visual noise is added to near-threshold stimuli. By investigating the relationship between random features in the noise and observer s perceptual decision in each trial, it is possible to estimate what features of the stimuli are critical for the task. The method allows visualizing the critical features that are used in a psychophysical task directly as a spatial correlation map, yielding an effective "behavioral receptive field". Visual context is known to modulate the perception of stimulus features. Some of these interactions are quite complex, and it is not known whether they reflect early or late stages of perceptual processing. The first study investigated the mechanisms of collinear facilitation, where nearby collinear Gabor flankers increase the detectability of a central Gabor. The behavioral receptive field of the mechanism mediating the detection of the central Gabor stimulus was measured by the classification image method. The results show that collinear flankers increase the extent of the behavioral receptive field for the central Gabor, in the direction of the flankers. The increased sensitivity at the ends of the receptive field suggests a low-level explanation for the facilitation. The second study investigated how visual features are integrated into percepts of surface brightness. A novel variant of the classification image method with brightness matching task was used. Many theories assume that perceived brightness is based on the analysis of luminance border features. Here, for the first time this assumption was directly tested. The classification images show that the perceived brightness of both an illusory Craik-O Brien-Cornsweet stimulus and a real uniform step stimulus depends solely on the border. Moreover, the spatial tuning of the features remains almost constant when the stimulus size is changed, suggesting that brightness perception is based on the output of a single spatial frequency channel. The third and fourth studies investigated global form integration in random-dot Glass patterns. In these patterns, a global form can be immediately perceived, if even a small proportion of random dots are paired to dipoles according to a geometrical rule. In the third study the discrimination of orientation structure in highly coherent concentric and Cartesian (straight) Glass patterns was measured. The results showed that the global form was more efficiently discriminated in concentric patterns. The fourth study investigated how form detectability depends on the global regularity of the Glass pattern. The local structure was either Cartesian or curved. It was shown that randomizing the local orientation deteriorated the performance only with the curved pattern. The results give support for the idea that curved and Cartesian patterns are processed in at least partially separate neural systems.

Essays in Philosophical Moral Psychology

The dissertation consists of four essays and a comprehensive introduction that discusses the topics, methods, and most prominent theories of philosophical moral psychology. I distinguish three main questions: What are the essential features of moral thinking? What are the psychological conditions of moral responsibility? And finally, what are the consequences of empirical facts about human nature to normative ethics? Each of the three last articles focuses on one of these issues. The first essay and part of the introduction are dedicated to methodological questions, in particular the relationship between empirical (social) psychology and philosophy. I reject recent attempts to understand the nature of morality on the basis of empirical research. One characteristic feature of moral thinking is its practical clout: if we regard an action as morally wrong, we either refrain from doing it even against our desires and interests, or else feel shame or guilt. Moral views seem to have a conceptual connection to motivation and emotions – roughly speaking, we can’t conceive of someone genuinely disapproving an action, but nonetheless doing it without any inner motivational conflict or regret. This conceptual thesis in moral psychology is called (judgment) internalism. It implies, among other things, that psychopaths cannot make moral judgments to the extent that they are incapable of corresponding motivation and emotion, even if they might say largely the words we would expect. Is internalism true? Recently, there has been an explosion of interest in so-called experimental philosophy, which is a methodological view according to which claims about conceptual truths that appeal to our intuitions should be tested by way of surveys presented to ordinary language users. One experimental result is that the majority of people are willing to grant that psychopaths make moral judgments, which challenges internalism. In the first article, ‘The Rise and Fall of Experimental Philosophy’, I argue that these results pose no real threat to internalism, since experimental philosophy is based on a too simple conception of the relationship between language use and concepts. Only the reactions of competent users in pragmatically neutral and otherwise conducive circumstances yield evidence about conceptual truths, and such robust intuitions remain inaccessible to surveys for reasons of principle. The epistemology of folk concepts must still be based on Socratic dialogue and critical reflection, whose character and authority I discuss at the end of the paper. The internal connection between moral judgment and motivation led many metaethicists in the past century to believe along Humean lines that judgment itself consists in a pro-attitude rather than a belief. This expressivist view, as it is called these days, has far-reaching consequences in metaethics. In the second essay I argue that perhaps the most sophisticated form of contemporary expressivism, Allan Gibbard’s norm-expressivism, according to which moral judgments are decisions or contingency plans, is implausible from the perspective of the theory of action. In certain circumstances it is possible to think that something is morally required of one without deciding to do so. Morality is not a matter of the will. Instead, I sketch on the basis of Robert Brandom’s inferentialist semantics a weak form of judgment internalism, according to which the content of moral judgment is determined by a commitment to a particular kind of practical reasoning. The last two essays in the dissertation emphasize the role of mutual recognition in the development and maintenance of responsible and autonomous moral agency. I defend a compatibilist view of autonomy, according to which agents who are unable to recognize right and wrong or act accordingly are not responsible for their actions – it is not fair to praise or blame them, since they lacked the relevant capacity to do otherwise. Conversely, autonomy demands an ability to recognize reasons and act on them. But as a long tradition in German moral philosophy whose best-known contemporary representative is Axel Honneth has it, both being aware of reasons and acting on them requires also the right sort of higher-order attitudes toward the self. Without self-respect and self-confidence we remain at the mercy of external pressures, even if we have the necessary normative competence. These attitudes toward the self, in turn, are formed through mutual recognition – we value ourselves when those who we value value us. Thus, standing in the right sort of relations of recognition is indirectly necessary for autonomy and moral responsibility. Recognition and valuing are concretely manifest in actions and institutions, whose practices make possible participation on an equal footing. Seeing this opens the way for a kind of normative social criticism that is grounded in the value of freedom and automomy, but is not limited to defending negative rights. It thus offers a new way to bridge the gap between liberalism and communitarianism.

Music in the recovering brain

Listening to music involves a widely distributed bilateral network of brain regions that controls many auditory perceptual, cognitive, emotional, and motor functions. Exposure to music can also temporarily improve mood, reduce stress, and enhance cognitive performance as well as promote neural plasticity. However, very little is currently known about the relationship between music perception and auditory and cognitive processes or about the potential therapeutic effects of listening to music after neural damage. This thesis explores the interplay of auditory, cognitive, and emotional factors related to music processing after a middle cerebral artery (MCA) stroke. In the acute recovery phase, 60 MCA stroke patients were randomly assigned to a music listening group, an audio book listening group, or a control group. All patients underwent neuropsychological assessments, magnetoencephalography (MEG) measurements, and magnetic resonance imaging (MRI) scans repeatedly during a six-month post-stroke period. The results revealed that amusia, a deficit of music perception, is a common and persistent deficit after a stroke, especially if the stroke affects the frontal and temporal brain areas in the right hemisphere. Amusia is clearly associated with deficits in both auditory encoding, as indicated by the magnetic mismatch negativity (MMNm) response, and domain-general cognitive processes, such as attention, working memory, and executive functions. Furthermore, both music and audio book listening increased the MMNm, whereas only music listening improved the recovery of verbal memory and focused attention as well as prevented a depressed and confused mood during the first post-stroke months. These findings indicate a close link between musical, auditory, and cognitive processes in the brain. Importantly, they also encourage the use of listening to music as a rehabilitative leisure activity after a stroke and suggest that the auditory environment can induce long-term plastic changes in the recovering brain.

Notions of Nation : A Critical Discursive Psychology perspective on the Isle of Man

At the the heart of this study can be seen the dual concern of how the nation is represented as a categorical entity and how this is put to use in everyday social interactions.This can be seen as a reaction to the general approach to categorisation and identity functions that tend to be reified and essentialized within the social sciences. The empirical focus of this study is the Isle of Man, a crown dependency situated geographically central within the British Isles while remaining political outside the United Kingdom. The choice of this site was chosen explicitly as ‘notions of nation’ expressed on the island can be seen as being contested and ephemerally unstable. To get at these ‘notions of nation’ is was necessary to choose specific theoretical tools that were able to capture the wider cultural and representational domain while being capable of addressing the nuanced and functional aspects of interaction. As such, the main theoretical perspective used within this study was that of critical discursive psychology which incorporates the specific theoretical tools interpretative repertoires, ideological dilemmas and subject positions. To supplement these tools, a discursive approach to place was taken in tandem to address the form and function of place attached to nationhood. Two methods of data collection were utilized, that of computer mediated communication and acquaintance interviews. From the data a number of interpretative repertoires were proposed, namely being, essential rights, economic worth, heritage claims, conflict orientation, people-as-nation and place-as-nation. Attached to such interpretative repertoires were the ideological dilemmas region vs. country, people vs. place and individualism vs. collectivism. The subject positions found are much more difficult to condense, but the most significant ones were gender, age and parentage. The final focus of the study, that of place, was shown to be more than just an unreflected on ‘container’ of people but was significant in terms of the rhetorical construction of such places for how people saw themselves and the discursive function of the particular interaction. As such, certain forms of place construction included size, community, temporal, economic, safety, political and recognition. A number of conclusions were drawn from the above which included, that when looking at nation categories we should take into account the specific meanings that people attach to such concepts and to be aware of the particular uses they are put to in interaction. Also, that it is impossible to separate concepts neatly, but it is necessary to be aware of the intersection where concepts cross, and clash, when looking at nationhood.

Linguistic and perceptual processing of communicative cues in Asperger Syndrome

Asperger Syndrome (AS) belongs to autism spectrum disorders where both verbal and non-verbal communication difficulties are at the core of the impairment. Social communication requires a complex use of affective, linguistic-cognitive and perceptual processes. In the four studies included in the current thesis, some of the linguistic and perceptual factors that are important for face-to-face communication were studied using behavioural methods. In all four studies the results obtained from individuals with AS were compared with typically developed age, gender and IQ matched controls. First, the language skills of school-aged children were characterized in detail with standardized tests that measured different aspects of receptive and expressive language (Study I). The children with AS were found to be worse than the controls in following complex verbal instructions. Next, the visual perception of facial expressions of emotion with varying degrees of visual detail was examined (Study II). Adults with AS were found to have impaired recognition of facial expressions on the basis of very low spatial frequencies which are important for processing global information. Following that, multisensory perception was investigated by looking at audiovisual speech perception (Studies III and IV). Adults with AS were found to perceive audiovisual speech qualitatively differently from typically developed adults, although both groups were equally accurate in recognizing auditory and visual speech presented alone. Finally, the effect of attention on audiovisual speech perception was studied by registering eye gaze behaviour (Study III) and by studying the voluntary control of visual attention (Study IV). The groups did not differ in eye gaze behaviour or in the voluntary control of visual attention. The results of the study series demonstrate that many factors underpinning face-to-face social communication are atypical in AS. In contrast with previous assumptions about intact language abilities, the current results show that children with AS have difficulties in understanding complex verbal instructions. Furthermore, the study makes clear that deviations in the perception of global features in faces expressing emotions as well as in the multisensory perception of speech are likely to harm face-to-face social communication.