Content Classification of Multimedia Documents using Partitions of Low-Level Features

Audio-visual documents obtained from German TV news are classified according to the IPTC topic categorization scheme. To this end usual text classification techniques are adapted to speech, video, and non-speech audio. For each of the three modalities word analogues are generated: sequences of syllables for speech, “video words” based on low level color features (color moments, color correlogram and color wavelet), and “audio words” based on low-level spectral features (spectral envelope and spectral flatness) for non-speech audio. Such audio and video words provide a means to represent the different modalities in a uniform way. The frequencies of the word analogues represent audio-visual documents: the standard bag-of-words approach. Support vector machines are used for supervised classification in a 1 vs. n setting. Classification based on speech outperforms all other single modalities. Combining speech with non-speech audio improves classification. Classification is further improved by supplementing speech and non-speech audio with video words. Optimal F-scores range between 62% and 94% corresponding to 50% - 84% above chance. The optimal combination of modalities depends on the category to be recognized. The construction of audio and video words from low-level features provide a good basis for the integration of speech, non-speech audio and video.

Thematic and Situational Features of Chinese BBS Texts

This study adopts the framework of Systemic Functional Grammar (SFG; Halliday, 1994/2000; Halliday & Matthiessen, 2004) to investigate thematic features in messages sent to an electronic bulletin board system (BBS) in mainland China. As a concept derived from the Prague School, theme in SFG has been identified as “the point of departure of the message; it is that which locates and orients the clause within its context” (Halliday & Matthiessen, 2004, p. 64). Thematic features in the Chinese data are found to relate to the situational features of the BBS, the analysis of which is based on the frameworks of Biber (1988) and Herring (2007). The relevant situational features are further generalized into the three components of context: field, tenor, and mode (Halliday & Hasan, 1985) in order to examine the relation between thematic features and situational features. The study’s findings show that thematic features are more closely related to the field (nature of the activity) than to the mode, contrary to Halliday’s (1978/2001) claim that theme, as a realization of the textual meaning, is determined by the mode (medium). In concluding, this discrepancy is explored.