Novel approach to determining tempo and dramatic story sections in motion pictures

This paper presents an original computational approach to extraction of movie tempo for deriving story sections and events that convey high level semantics of stories portrayed in motion pictures, thus enabling better video annotation and interpretation systems. This approach, inspired by the existing cinematic conventions known as film grammar, uses the attributes of motion and shot length to define and compute a novel continuous measure of tempo of a movie. Tempo flow plots are derived for several full-length motion pictures and edge detection is performed to extract dramatic story sections and events occurring in the movie, underlined by their unique tempo. The results confirm reliable detection of actual distinct tempo changes and serve as useful index into the dramatic development and narration of the story in motion pictures.

Early jump-out corner detectors

Two corner detectors are presented, one of which works by testing similarity of image patches along the contour direction to detect curves in the image contour, and the other of which uses direct estimation image curvature along the contour direction. The operators are fast, robust to noise, and self-thresholding. An interpretation of the Kitchen-Rosenfeld corner operator is presented which shows that this operator can also be viewed as the second derivative of the image function along the edge direction.

Neighborhood coherence and edge based approaches to film scene extraction

In order to enable high-level semantics-based video annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from film production to determine when a scene change occurs in film. We examine different rules and conventions followed as part of Film Grammar to guide and shape our algorithmic solution for determining a scene boundary. Two different techniques are proposed as new solutions in this paper. Our experimental results on 10 full-length movies show that our technique based on shot sequence coherence performs well and reasonably better than the color edges-based approach.

Local energy, the pre-envelope, and filter resolution

We examine the construction of new filters for computing local energy, and compare these filters with the Gabor filters and the three-point-filter of Venkatesh [l]. Further, we demonstrate that the effect of convolution with complex Gabor filters is to band-pass (with some differentiating effect) and compute the local energy of the result. The magnitude of the resulting local energy is then used to detect features [2], [3] (step features, texture etc.), and the phase is used to classify the detected features [l], [4] or provide disparity information for stereo [5] and motion work [6], [7]. Each of these types of information can be obtained at multiple resolutions, enabling the use of course to fine strategies for computing disparity, and allowing the discrimination of image textures on the basis of which parts of the Fourier domain they dominate [8], [9].

Multiple-object tracking in cluttered and crowded public spaces

This paper addresses the problem of tracking moving objects of variable appearance in challenging scenes rich with features and texture. Reliable tracking is of pivotal importance in surveillance applications. It is made particularly difficult by the nature of objects encountered in such scenes: these too change in appearance and scale, and are often articulated (e.g. humans). We propose a method which uses fast motion detection and segmentation as a constraint for both building appearance models and their robust propagation (matching) in time. The appearance model is based on sets of local appearances automatically clustered using spatio-kinetic similarity, and is updated with each new appearance seen. This integration of all seen appearances of a tracked object makes it extremely resilient to errors caused by occlusion and the lack of permanence of due to low data quality, appearance change or background clutter. These theoretical strengths of our algorithm are empirically demonstrated on two hour long video footage of a busy city marketplace.

An integrative framework for the appraisal of coloration in nature

The world in color presents a dazzling dimension of phenotypic variation. Biological interest in this variation has burgeoned, due to both increased means for quantifying spectral information and heightened appreciation for how animals view the world differently than humans. Effective study of color traits is challenged by how to best quantify visual perception in nonhuman species. This requires consideration of at least visual physiology but ultimately also the neural processes underlying perception. Our knowledge of color perception is founded largely on the principles gained from human psychophysics that have proven generalizable based on comparative studies in select animal models. Appreciation of these principles, their empirical foundation, and the reasonable limits to their applicability is crucial to reaching informed conclusions in color research. In this article, we seek a common intellectual basis for the study of color in nature. We first discuss the key perceptual principles, namely, retinal photoreception, sensory channels, opponent processing, color constancy, and receptor noise. We then draw on this basis to inform an analytical framework driven by the research question in relation to identifiable viewers and visual tasks of interest. Consideration of the limits to perceptual inference guides two primary decisions: first, whether a sensory-based approach is necessary and justified and, second, whether the visual task refers to perceptual distance or discriminability. We outline informed approaches in each situation and discuss key challenges for future progress, focusing particularly on how animals perceive color. Given that animal behavior serves as both the basic unit of psychophysics and the ultimate driver of color ecology/evolution, behavioral data are critical to reconciling knowledge across the schools of color research.

Artificial selection for food colour preferences

Colour is an important factor in food detection and acquisition by animals using visually based foraging. Colour can be used to identify the suitability of a food source or improve the efficiency of food detection, and can even be linked to mate choice. Food colour preferences are known to exist, but whether these preferences are heritable and how these preferences evolve is unknown. Using the freshwater fish Poecilia reticulata, we artificially selected for chase behaviour towards two different-coloured moving stimuli: red and blue spots. A response to selection was only seen for chase behaviours towards the red, with realized heritabilities ranging from 0.25 to 0.30. Despite intense selection, no significant chase response was recorded for the blue-selected lines. This lack of response may be due to the motion-detection mechanism in the guppy visual system and may have novel implications for the evolvability of responses to colour-related signals. The behavioural response to several colours after five generations of selection suggests that the colour opponency system of the fish may regulate the response to selection.

Contextually Learnt Detection of Unusual Motion-Based Behaviour in Crowded Public Spaces

In this paper we are interested in analyzing behaviour in crowded publicplaces at the level of holistic motion. Our aim is to learn, without user input, strong scene priors or labelled data, the scope of ‘‘normal behaviour’’ for a particular scene and thus alert to novelty in unseen footage. The first contribution is a low-level motion model based on what we term tracklet primitives, which are scenespecific elementary motions. We propose a clustering-based algorithm for tracklet estimation from local approximations to tracks of appearance features. This is followed by two methods for motion novelty inference from tracklet primitives: (a) an approach based on a non-hierarchial ensemble of Markov chains as a means of capturing behavioural characteristics at different scales, and (b) a more flexible alternative which exhibits a higher generalizing power by accounting for constraints introduced by intentionality and goal-oriented planning of human motion in a particular scene. Evaluated on a 2 h long video of a busy city marketplace, both algorithms are shown to be successful at inferring unusual behaviour, the latter model achieving better performance for novelties at a larger spatial scale.

Detection of dynamic background due to swaying movements from motion features

Dynamically changing background (dynamic background) still presents a great challenge to many motion-based video surveillance systems. In the context of event detection, it is a major source of false alarms. There is a strong need from the security industry either to detect and suppress these false alarms, or dampen the effects of background changes, so as to increase the sensitivity to meaningful events of interest. In this paper, we restrict our focus to one of the most common causes of dynamic background changes: 1) that of swaying tree branches and 2) their shadows under windy conditions. Considering the ultimate goal in a video analytics pipeline, we formulate a new dynamic background detection problem as a signal processing alternative to the previously described but unreliable computer vision-based approaches. Within this new framework, we directly reduce the number of false alarms by testing if the detected events are due to characteristic background motions. In addition, we introduce a new data set suitable for the evaluation of dynamic background detection. It consists of real-world events detected by a commercial surveillance system from two static surveillance cameras. The research question we address is whether dynamic background can be detected reliably and efficiently using simple motion features and in the presence of similar but meaningful events, such as loitering. Inspired by the tree aerodynamics theory, we propose a novel method named local variation persistence (LVP), that captures the key characteristics of swaying motions. The method is posed as a convex optimization problem, whose variable is the local variation. We derive a computationally efficient algorithm for solving the optimization problem, the solution of which is then used to form a powerful detection statistic. On our newly collected data set, we demonstrate that the proposed LVP achieves excellent detection results and outperforms the best alternative adapted from existing art in the dynamic background literature.

Solution to robotic landmine detection through use of path planning and motor control

Low cost robotic detectors are a promising new approach to combat the disturbing landmine crisis. In this paper a low-cost robotic solution is proposed, we present several control techniques used to improve the precision of the robotic motion. A P and PD controller is applied, and it is concluded that a cascaded control system provides a more stable and accurate response. Two search patterns for landmine detection are considered, rectangular and spiral, these are used to analyse the effectiveness of the control system.

A video-based real-time vehicle detection method by classified background learning

A new two-level real-time vehicle detection method is proposed in order to meet the robustness and efficiency requirements of real world applications. At the high level, pixels of the background image are classified into three categories according to the characteristics of Red, Green, Blue (RGB) curves. The robustness of the classification is further enhanced by using
line detection and pattern connectivity. At the lower level, an exponential forgetting algorithm with adaptive parameters for different categories is utilised to calculate the background and reduce the distortion by the small motion of video cameras. Scene tests show that the proposed method is more robust and faster than previous methods, which is very suitable for real-time vehicle detection in outdoor environments, especially concerning locations where the level of illumination changes frequently and speed detection is important.

A probabilistic framework for extracting narrative act boundaries and semantics in motion pictures

This work constitutes the first attempt to extract the important narrative structure, the 3-Act storytelling paradigm in film. Widely prevalent in the domain of film, it forms the foundation and framework in which a film can be made to function as an effective tool for story telling, and its extraction is a vital step in automatic content management for film data. The identification of act boundaries allows for structuralizing film at a level far higher than existing segmentation frameworks, which include shot detection and scene identification, and provides a basis for inferences about the semantic content of dramatic events in film. A novel act boundary likelihood function for Act 1 and 2 is derived using a Bayesian formulation under guidance from film grammar, tested under many configurations and the results are reported for experiments involving 25 full-length movies. The result proves to be a useful tool in both the automatic and semi-interactive setting for semantic analysis of film, with potential application to analogues occuring in many other domains, including news, training video, sitcoms.

Scene extraction in motion pictures

This paper addresses the challenge of bridging the semantic gap between the rich meaning users desire when they query to locate and browse media and the shallowness of media descriptions that can be computed in today's content management systems. To facilitate high-level semantics-based content annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from fill production to determine when a scene change occurs. We then investigate different rules and conventions followed as part of Fill Grammar that would guide and shape an algorithmic solution for determining a scene. Two different techniques using intershot analysis are proposed as solutions in this paper. In addition, we present different refinement mechanisms, such as film-punctuation detection founded on Film Grammar, to further improve the results. These refinement techniques demonstrate significant improvements in overall performance. Furthermore, we analyze errors in the context of film-production techniques, which offer useful insights into the limitations of our method.

Towards automatic extraction of expressive elements from motion pictures : tempo

This paper proposes a unique computational approach to extraction of expressive elements of motion pictures for deriving high level semantics of stories portrayed, thus enabling better video annotation and interpretation systems. This approach, motivated and directed by the existing cinematic conventions known as film grammar, as a first step towards demonstrating its effectiveness, uses the attributes of motion and shot length to define and compute a novel measure of tempo of a movie. Tempo flow plots are defined and derived for four full-length movies and edge analysis is performed leading to the extraction of dramatic story sections and events signaled by their unique tempo. The results confirm tempo as a useful attribute in its own right and a promising component of semantic constructs such as tone or mood of a film.

New enhancements to cut, fade, and dissolve detection processes in video segmentation

We present improved algorithms for cut, fade, and dissolve detection which are fundamental steps in digital video analysis. In particular, we propose a new adaptive threshold determination method that is shown to reduce artifacts created by noise and motion in scene cut detection. We also describe new two-step algorithms for fade and dissolve detection, and introduce a method for eliminating false positives from a list of detected candidate transitions. In our detailed study of these gradual shot transitions, our objective has been to accurately classify the type of transitions (fade-in, fade-out, and dissolve) and to precisely locate the boundary of the transitions. This distinguishes our work from other early work in scene change detection which tends to focus primarily on identifying the existence of a transition rather than its precise temporal extent. We evaluate our improved algorithms against two other commonly used shot detection techniques on a comprehensive data set, and demonstrate the improved performance due to our enhancements.