Recognition of Human Action Using Moment-Based Feature

The performance of different classification approaches is evaluated using a view-based approach for motion representation. The view-based approach uses computer vision and image processing techniques to register and process the video sequence. Two motion representations called Motion Energy Images and Motion History Image are then constructed. These representations collapse the temporal component in a way that no explicit temporal analysis or sequence matching is needed. Statistical descriptions are then computed using moment-based features and dimensionality reduction techniques. For these tests, we used 7 Hu moments, which are invariant to scale and translation. Principal Components Analysis is used to reduce the dimensionality of this representation. The system is trained using different subjects performing a set of examples of every action to be recognized. Given these samples, K-nearest neighbor, Gaussian, and Gaussian mixture classifiers are used to recognize new actions. Experiments are conducted using instances of eight human actions (i.e., eight classes) performed by seven different subjects. Comparisons in the performance among these classifiers under different conditions are analyzed and reported. Our main goals are to test this dimensionality-reduced representation of actions, and more importantly to use this representation to compare the advantages of different classification approaches in this recognition task.

The role of areas MT+/V5 and SPOC in spatial and temporal control of manual interception: an rTMS study

Manual interception, such as catching or hitting an approaching ball, requires the hand to contact a moving object at the right location and at the right time. Many studies have examined the neural mechanisms underlying the spatial aspects of goal-directed reaching, but the neural basis of the spatial and temporal aspects of manual interception are largely unknown. Here, we used repetitive transcranial magnetic stimulation (rTMS) to investigate the role of the human middle temporal visual motion area (MT+/V5) and superior parieto-occipital cortex (SPOC) in the spatial and temporal control of manual interception. Participants were required to reach-to-intercept a downward moving visual target that followed an unpredictably curved trajectory, presented on a screen in the vertical plane. We found that rTMS to MT+/V5 influenced interceptive timing and positioning, whereas rTMS to SPOC only tended to increase the spatial variance in reach end points for selected target trajectories. These findings are consistent with theories arguing that distinct neural mechanisms contribute to spatial, temporal, and spatiotemporal control of manual interception.

Monocular-based 3-D seafloor reconstruction and ortho-mosaicing by piecewise planar representation

Photo-mosaicing techniques have become popular for seafloor mapping in various marine science applications. However, the common methods cannot accurately map regions with high relief and topographical variations. Ortho-mosaicing borrowed from photogrammetry is an alternative technique that enables taking into account the 3-D shape of the terrain. A serious bottleneck is the volume of elevation information that needs to be estimated from the video data, fused, and processed for the generation of a composite ortho-photo that covers a relatively large seafloor area. We present a framework that combines the advantages of dense depth-map and 3-D feature estimation techniques based on visual motion cues. The main goal is to identify and reconstruct certain key terrain feature points that adequately represent the surface with minimal complexity in the form of piecewise planar patches. The proposed implementation utilizes local depth maps for feature selection, while tracking over several views enables 3-D reconstruction by bundle adjustment. Experimental results with synthetic and real data validate the effectiveness of the proposed approach

View-based approaches to spatial representation in human vision

In an immersive virtual environment, observers fail to notice the expansion of a room around them and consequently make gross errors when comparing the size of objects. This result is difficult to explain if the visual system continuously generates a 3-D model of the scene based on known baseline information from interocular separation or proprioception as the observer walks. An alternative is that observers use view-based methods to guide their actions and to represent the spatial layout of the scene. In this case, they may have an expectation of the images they will receive but be insensitive to the rate at which images arrive as they walk. We describe the way in which the eye movement strategy of animals simplifies motion processing if their goal is to move towards a desired image and discuss dorsal and ventral stream processing of moving images in that context. Although many questions about view-based approaches to scene representation remain unanswered, the solutions are likely to be highly relevant to understanding biological 3-D vision.

A psychophysical and computational analysis of the spatio-temporal mechanisms underlying the flash-lag effect

Several accounts put forth to explain the flash-lag effect (FLE) rely mainly on either spatial or temporal mechanisms. Here we investigated the relationship between these mechanisms by psychophysical and theoretical approaches. In a first experiment we assessed the magnitudes of the FLE and temporal-order judgments performed under identical visual stimulation. The results were interpreted by means of simulations of an artificial neural network, that wits also employed to make predictions concerning the F LE. The model predicted that a spatio-temporal mislocalisation would emerge from two, continuous and abrupt-onset, moving stimuli. Additionally, a straightforward prediction of the model revealed that the magnitude of this mislocalisation should be task-dependent, increasing when the use of the abrupt-onset moving stimulus switches from a temporal marker only to both temporal and spatial markers. Our findings confirmed the model`s predictions and point to an indissoluble interplay between spatial facilitation and processing delays in the FLE.

Insect inspired behaviours for the autonomous control of mobile robots

Animals navigate through various uncontrolled environments with seemingly little effort. Flying insects, especially, are quite adept at manoeuvring in complex, unpredictable and possibly hostile environments. Through both simulation and real-world experiments, we demonstrate the feasibility of equipping a mobile robot with the ability to navigate a corridor environment, in real time, using principles based on insect-based visual guidance. In particular we have used the bees’ navigational strategy of measuring object range in terms of image velocity. We have also shown the viability and usefulness of various other insect behaviours: (i) keeping walls equidistant, (ii) slowing down when approaching an object, (iii) regulating speed according to tunnel width, and (iv) using visual motion as a measure of distance travelled.

Multichannel biomedical time series clustering via hierarchical probabilistic latent semantic analysis

Biomedical time series clustering that automatically groups a collection of time series according to their internal similarity is of importance for medical record management and inspection such as bio-signals archiving and retrieval. In this paper, a novel framework that automatically groups a set of unlabelled multichannel biomedical time series according to their internal structural similarity is proposed. Specifically, we treat a multichannel biomedical time series as a document and extract local segments from the time series as words. We extend a topic model, i.e., the Hierarchical probabilistic Latent Semantic Analysis (H-pLSA), which was originally developed for visual motion analysis to cluster a set of unlabelled multichannel time series. The H-pLSA models each channel of the multichannel time series using a local pLSA in the first layer. The topics learned in the local pLSA are then fed to a global pLSA in the second layer to discover the categories of multichannel time series. Experiments on a dataset extracted from multichannel Electrocardiography (ECG) signals demonstrate that the proposed method performs better than previous state-of-the-art approaches and is relatively robust to the variations of parameters including length of local segments and dictionary size. Although the experimental evaluation used the multichannel ECG signals in a biometric scenario, the proposed algorithm is a universal framework for multichannel biomedical time series clustering according to their structural similarity, which has many applications in biomedical time series management.

Locomoção visualmente guiada na transposição de obstáculos: efeitos de amostras visuais estáticas e dinâmica

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)

Video-based quantification of body movement during social interaction indicates the severity of negative symptoms in patients with schizophrenia

In schizophrenia, nonverbal behavior, including body movement, is of theoretical and clinical importance. Although reduced nonverbal expressiveness is a major component of the negative symptoms encountered in schizophrenia, few studies have objectively assessed body movement during social interaction. In the present study, 378 brief, videotaped role-play scenes involving 27 stabilized outpatients diagnosed with paranoid-type schizophrenia were analyzed using Motion Energy Analysis (MEA). This method enables the objective measuring of body movement in conjunction with ordinary video recordings. Correlations between movement parameters (percentage of time in movement, movement speed) and symptom ratings from independent PANSS interviews were calculated. Movement parameters proved to be highly reliable. In keeping with predictions, reduced movement and movement speed correlated with negative symptoms. Accordingly, in patients who exhibited noticeable movement for less than 20% of the observation time, prominent negative symptoms were highly probable. As a control measure, the percentage of movement exhibited by the patients during role-play scenes was compared to that of their normal interactants. Patients with negative symptoms differed from normal interactants by showing significantly reduced head and body movement. Two specific positive symptoms were possibly related to movement parameters: suspiciousness tended to correlate with reduced head movement, and the expression of unusual thought content tended to relate to increased movement. Overall, a close and theoretically meaningful association between the objective movement parameters and the symptom profiles was found. MEA appears to be an objective, reliable and valid method for quantifying nonverbal behavior, an aspect which may furnish new insights into the processes related to reduced expressiveness in schizophrenia.

Nonverbale Synchronisation in Dyaden: ein Vergleich von kooperativen und kompetitiven Interaktionen

Nonverbales Verhalten spielt in zwischenmenschlichen Interaktionen eine bedeutende Rolle. Es beeinflusst mutmaßlich den Verlauf eines Gesprächs, die Beziehung zwischen den Interaktionspartnern und die reziproke Sympathie. Trotz dieser immensen Bedeutung wird nonverbales Verhalten von den Beteiligten meist nicht bewusst wahrgenommen. Die vorliegende Arbeit widmet sich diesem wirkungsvollen Phänomen und vergleicht nonverbales Verhalten in kooperativen und kompetitiven Interaktionsbedingungen. Im Fokus der Experimentalstudie steht die nonverbale Synchronisation. Diese bezieht sich auf den dynamischen, quantitativen Aspekt der Koordination nonverbalen Verhaltens zwischen den Interaktionspartnern, konkret auf die Angleichung der Bewegungsenergie. Primäres Ziel ist es zu evaluieren, ob und inwiefern das nonverbale Verhalten, insbesondere das Ausmaß an Synchronisation, zwischen kooperativen und kompetitiven Bedingungen variiert. Hierzu wurden in einer Stichprobe von N = 168 gesunden Teilnehmern fünf standardisierte Interaktionen à fünf Minuten mit gleichgeschlechtlichen Dyaden realisiert. Die Probanden kannten sich vorher nicht. Die Interaktanten sahen sich aufgefordert, sowohl aktive Kooperationen zu etablieren, als auch in bestimmten Aufgaben zu konkurrieren. Kompromiss, Konsens, Konflikt, ungleicher Konflikt und Spiel waren die verschiedenen Bedingungen. Die resultierenden Synchronisationswerte basieren auf digitalen Videoaufnahmen von dyadischen Interaktionen, die durch das automatisierte, objektive Verfahren der Motion Energy Analysis (MEA) quantifiziert wurden. Neben den Synchronisationsprozessen wurden auch globale Bewegungscharakteristika wie Geschwindigkeit oder Maxima durch MEA erfasst. Des Weiteren wurden mittels Fragebögen individuelle Charakteristika (u.a. Empathie, Stimmung) sowie Einschätzungen (z.B. Sympathie) der Probanden erhoben und mit Synchronie in Zusammenhang gesetzt. Die Ergebnisse zeigen, dass sich nonverbale Synchronisation auf signifikant höherem Level manifestiert, als dies per Zufall erwartet werden könnte. Dabei war die Synchronisation in der spielerischen Kooperation am höchsten ausgeprägt - gefolgt von kompetitiven und (seriös-)kooperativen Interaktionen. Außerdem korrespondierten höhere Synchronisationswerte mit verstärkt positiven Affekten und verringerten negativen Emotionen. Darüber hinaus waren globale Bewegungsparameter wie Dauer und Komplexität oder der Prozentsatz von Bewegungen über dem Schwellenwert positiv mit Synchronisationsprozessen assoziiert. Das MEA-Verfahren sowie ein erstmals erprobtes Interaktionsparadigma konnten validiert werden.

EPA-1380 – Coordinated body-movement reflects psychotherapy outcome

Social interaction is a core aspect of human life that affects individuals’ physical and mental health. Social interaction usually leads to mutual engagement in diverse areas of mental, emotional, physiological and physical activity involving both interacting persons and subsequently impacting the outcome of interactions. A common approach to the analysis of social interaction is the study of the verbal content transmitted between sender and receiver. However, additional important processes and dynamics are occurring in other domains too, for example in the area of nonverbal behaviour: In a series of studies, we have looked at nonverbal synchrony – the coordination of two persons’ movement patterns – and it‘s association with relationship quality and with the outcome of interactions. Using a computer-based algorithm (Motion Energy Analysis, MEA: Ramseyer & Tschacher, 2011), which automatically quantifies a person‘s body-movement, we were able to objectively calculate nonverbal synchrony in a large number of dyads interacting in various settings. In a first step, we showed that the phenomenon of nonverbal synchrony exists at a level that is significantly higher than expected by chance. In a second step, we ascertained that across different settings – including patient-therapist dyads and healthy dyads – more synchronized movement was associated with better relationship quality and better interactional outcomes. The quality of a relationship is thus embodied by the synchronized movement patterns emerging between partners. Our studies suggest that embodied cognition is a valuable approach to research in social interaction, providing important clues for an improved understanding of interaction dynamics.

Kopf- und Körperbewegungen in sozialen Interaktionen als Indikatoren des Schweregrades schizophrener Störungen – Eine Replikation

Einleitung: Auffälligkeiten im nonverbalen Verhalten wurden bei schizophrenen Störungen seit jeher als diagnostisch relevant eingestuft. Sie weisen deutliche Beziehungen zu negativen Symptomen und zu Einschränkungen im sozialen Funktionsniveau auf, wurden jedoch bisher aufgrund methodischer Probleme nur selten objektiv erfasst. Wir konnten bereits zeigen, dass das objektiv erfasste Ausmaß an Bewegung von Patienten mit Schizophrenie in sozialen Rollenspielinteraktionen eng mit ihren Symptomprofilen verbunden sind (Kupper, Ramseyer, Hoffmann, & Tschacher, Sz Res, 2010). In der vorliegenden Studie wurde eine Replikation dieser Ergebnisse im Rahmen von halbstandardisierten PANSS-Interviews zur Psychopathologie versucht. Methode: In der vorliegenden Studie wurden bei 44 Patienten mit schizophrenen Störungen je eine 15-minütige Sequenz aus einem PANSS-Interview mittels Motion Energy Analyse (MEA) analysiert. MEA bezeichnet eine neuartige Methode durch die Körperbewegungen objektiv in Videoaufnahmen quantifiziert werden können. Bewegung wird dabei aufgrund der Frame-to-Frame-Veränderungen in den Grauwerten in vorher festgelegten Regionen erfasst. Ergebnisse: Es fanden sich deutliche und signifikante Zusammenhänge zwischen reduzierten Bewegungen der Patienten und dem Ausmaß an Negativsymptomatik. Auf der Ebene der einzelnen negativen Symptome waren wiederum besonders die affektiven Symptome mit reduzierte Bewegung verbunden. Damit konnten zentrale Ergebnisse der früheren Arbeit repliziert werden. Damit konnte ein zentrales Ergebnisse der früheren Arbeit repliziert werden. Bemerkenswert war, dass dies trotz der im Vergleich mit Rollenspielinteraktionen weniger stark strukturierten sozialen Interaktionen in den psychopathologischen Interviews gelang. Gesamthaft betrachtet unterstützen die hier und in früheren Arbeiteten berichten objektivierbaren Zusammenhänge zwischen Bewegung und dem Schweregrad schizophrener Störungen neue Zugänge sowohl in der grundlagenorientierten Forschung als auch bei klinischen Interventionen zur Verbesserung sozialer Fertigkeiten.

Head movement synchrony in social interactions of patients with schizophrenia indicates symptoms, cognition and social functioning

Background: Disturbed interpersonal communication is a core problem in schizophrenia. Patients with schizophrenia often appear disconnected and "out of sync" when interacting with others. This may involve perception, cognition, motor behavior, and nonverbal expressiveness. Although well-known from clinical observation, mainstream research has neglected this area. Corresponding theoretical concepts, statistical methods, and assessment were missing. In recent research, however, it has been shown that objective, video-based measures of nonverbal behavior can be used to reliably quantify nonverbal behavior in schizophrenia. Newly developed algorithms allow for a calculation of movement synchrony. We found that the objective amount of movement of patients with schizophrenia during social interactions was closely related to the symptom profiles of these patients (Kupper et al., 2010). In addition and above the mere amount of movement, the degree of synchrony between patients and healthy interactants may be indicative of various problems in the domain of interpersonal communication and social cognition. Methods: Based on our earlier study, head movement synchrony was assessed objectively (using Motion Energy Analysis, MEA) in 378 brief, videotaped role-play scenes involving 27 stabilized outpatients diagnosed with paranoid-type schizophrenia. Results: Lower head movement synchrony was indicative of symptoms (negative symptoms, but also of conceptual disorganization and lack of insight), verbal memory, patients’ self-evaluation of competence, and social functioning. Many of these relationships remained significant even when corrected for the amount of movement of the patients. Conclusion: The results suggest that nonverbal synchrony may be an objective and sensitive indicator of the severity of symptoms, cognition and social functioning.

Nonverbal synchrony in social interactions of patients with schizophrenia indicates socio-communicative deficits

Background Disordered interpersonal communication can be a serious problem in schizophrenia. Recent advances in computer-based measures allow reliable and objective quantification of nonverbal behavior. Research using these novel measures has shown that objective amounts of body and head movement in patients with schizophrenia during social interactions are closely related to the symptom profiles of these patients. In addition to and above mere amounts of movement, the degree of synchrony, or imitation, between patients and normal interactants may be indicative of core deficits underlying various problems in domains related to interpersonal communication, such as symptoms, social competence, and social functioning. Methods Nonverbal synchrony was assessed objectively using Motion Energy Analysis (MEA) in 378 brief, videotaped role-play scenes involving 27 stabilized outpatients diagnosed with paranoid-type schizophrenia. Results Low nonverbal synchrony was indicative of symptoms, low social competence, impaired social functioning, and low self-evaluation of competence. These relationships remained largely significant when correcting for the amounts of patients‘ movement. When patients showed reduced imitation of their interactants’ movements, negative symptoms were likely to be prominent. Conversely, positive symptoms were more prominent in patients when their interaction partners’ imitation of their movements was reduced. Conclusions Nonverbal synchrony can be an objective and sensitive indicator of the severity of patients’ problems. Furthermore, quantitative analysis of nonverbal synchrony may provide novel insights into specific relationships between symptoms, cognition, and core communicative problems in schizophrenia.

Video-based quantification of body movement indicates negative symptoms: a replication

Objective: In schizophrenia, abnormalities in nonverbal behaviors have always been considered as highly relevant. However, due to methodological limitations, nonverbal behavior was rarely quantified objectively. Recent methodological advances now allow a quantification of body movement from ordinary video recordings. We showed that patients’ objectively measured amount of movement in social role-play interactions was closely associated with their symptom profiles (Kupper, Ramseyer, Hoffmann, & Tschacher, Schizophrenia Research 2010). In the present study, a replication of these results in the context of semi-standardized PANSS (Positive and Negative Syndrome Scale) interviews was intended. Methods: 17 patients with schizophrenia were analyzed during the initial 15-min sequence of a videotaped PANSS interview using Motion Energy Analysis (MEA). The amount of patients’ movement was then correlated with their PANSS symptom scores. Results: Sizeable and significant correlations between negative symptoms and reduced movements (r = -.68, p<0.01) and reduced movement speed (r = -.80, p<0.001) were found. Moreover, cognitive symptoms were related to reduced movement speed (r = -.70, p<.01). Conclusion: Negative symptoms were reliably indicated by patients’ nonverbal behavior in psychopathology interviews. Hence, the main result of our earlier study, examining patients’ nonverbal behavior in role play tests, was replicated for the less structured interactions in psychopathological interviews. Results could encourage the use of MEA in a wide range of videotaped social interactions of patients with schizophrenia and other psychiatric disorders.