Finding the optimal temporal partitioning of video sequences

The existing techniques for shot partitioning either process each shot boundary independently or proceed sequentially. The sequential process assumes the last shot boundary is correctly detected and utilizes the shot length distribution to adapt the threshold for detecting the next boundary. These techniques are only locally optimal and suffer from the strong assumption about the correct detection of the last boundary. Addressing these fundamental issues, in this paper, we aim to find the global optimal shot partitioning by utilizing Bayesian principles to model the probability of a particular video partition being the shot partition. A computationally efficient algorithm based on Dynamic Programming is then formulated. The experimental results on a large movie set show that our algorithm performs consistently better than the best adaptive-thresholding technique commonly used for the task.

Real-time lane detection on suburban streets using visual cue integration

The detection of lane boundaries on suburban streets using images obtained from video constitutes a challenging task. This is mainly due to the difficulties associated with estimating the complex geometric structure of lane boundaries, the quality of lane markings as a result of wear, occlusions by traffic, and shadows caused by road-side trees and structures. Most of the existing techniques for lane boundary detection employ a single visual cue and will only work under certain conditions and where there are clear lane markings. Also, better results are achieved when there are no other onroad objects present. This paper extends our previous work and discusses a novel lane boundary detection algorithm specifically addressing the abovementioned issues through the integration of two visual cues. The first visual cue is based on stripe-like features found on lane lines extracted using a two-dimensional symmetric Gabor filter. The second visual cue is based on a texture characteristic determined using the entropy measure of the predefined neighbourhood around a lane boundary line. The visual cues are then integrated using a rulebased classifier which incorporates a modified sequential covering algorithm to improve robustness. To separate lane boundary lines from other similar features, a road mask is generated using road chromaticity values estimated from CIE L*a*b* colour transformation. Extraneous points around lane boundary lines are then removed by an outlier removal procedure based on studentized residuals. The lane boundary lines are then modelled with Bezier spline curves. To validate the algorithm, extensive experimental evaluation was carried out on suburban streets and the results are presented. 

Improved fade and dissolve detection for reliable video segmentation

We present improved algorithms for automatic fade and dissolve detection in digital video analysis. We devise new two-step algorithms for fade and dissolve detection and introduce a method for eliminating false positives from a list of detected candidate transitions. In our detailed study of these gradual shot transitions, our objective has been to accurately classify the type of transitions (fade-in, fade-out, and dissolve) and to precisely locate the boundary of the transitions. This distinguishes our work from early work in scene change detection which focuses on identifying the existence of a transition rather than its precise temporal extent. We evaluate our algorithms against two other commonly used methods on a comprehensive data set, and demonstrate the improved performance due to our enhancements.

New enhancements to cut, fade, and dissolve detection processes in video segmentation

We present improved algorithms for cut, fade, and dissolve detection which are fundamental steps in digital video analysis. In particular, we propose a new adaptive threshold determination method that is shown to reduce artifacts created by noise and motion in scene cut detection. We also describe new two-step algorithms for fade and dissolve detection, and introduce a method for eliminating false positives from a list of detected candidate transitions. In our detailed study of these gradual shot transitions, our objective has been to accurately classify the type of transitions (fade-in, fade-out, and dissolve) and to precisely locate the boundary of the transitions. This distinguishes our work from other early work in scene change detection which tends to focus primarily on identifying the existence of a transition rather than its precise temporal extent. We evaluate our improved algorithms against two other commonly used shot detection techniques on a comprehensive data set, and demonstrate the improved performance due to our enhancements.

Fast scene change detection based histogram

This work presents a new approach to detecting the scene change in the successive capture of photographs of a place within equal time interval. This method is based on a gray level histogram of every image. In this method the histogram of an image is processed to modify it for matching with the processed histogram of a reference image. The coefficient of correlation is taken as the measure of matching. As the method does not do any heavy signal processing, and the images are taken successively with a multi-shot digital still camera, it can be applied for real-time processing of such pictures for detection of a scene change. A multi-camera in multi-position approach is also shown to evaluate the change in scene simultaneously from different angles. Both multi-camera and single-camera approaches are compared in detecting a scene change.

Spectral element modelling of wave propagation with boundary and structural discontinuity reflections

Spectral element method is very efficient in modelling high-frequency stress wave propagation because it works in the frequency domain. It does not need to use very fine meshes in order to capture high frequency wave energy as the time domain methods do, such as finite element method. However, the conventional spectral element method requires a throw-off element to be added to the structural boundaries to act as a conduit for energy to transmit out of the system. This makes the method difficult to model wave reflection at boundaries. To overcome this limitation, imaginary spectral elements are proposed in this study, which are combined with the real structural elements to model wave reflections at structural boundaries. The efficiency and accuracy of this proposed approach is verified by comparing the numerical simulation results with measured results of one dimensional stress wave propagation in a steel bar. The method is also applied to model wave propagation in a steel bar with not only boundary reflection, but also reflections from single and multiple cracks. The reflection and transmission coefficients, which are obtained from the discrete spring model, are adopted to quantify the discontinuities. Experimental tests of wave propagation in a steel bar with one crack of different depths are also carried out. Numerical simulations and experimental results show that the proposed method is effective and reliable in modelling wave propagation in one-dimensional waveguides with reflections from boundary and structural discontinuities. The proposed method can be applied to effectively model stress wave propagation for structural damage detection.

A probabilistic framework for extracting narrative act boundaries and semantics in motion pictures

This work constitutes the first attempt to extract the important narrative structure, the 3-Act storytelling paradigm in film. Widely prevalent in the domain of film, it forms the foundation and framework in which a film can be made to function as an effective tool for story telling, and its extraction is a vital step in automatic content management for film data. The identification of act boundaries allows for structuralizing film at a level far higher than existing segmentation frameworks, which include shot detection and scene identification, and provides a basis for inferences about the semantic content of dramatic events in film. A novel act boundary likelihood function for Act 1 and 2 is derived using a Bayesian formulation under guidance from film grammar, tested under many configurations and the results are reported for experiments involving 25 full-length movies. The result proves to be a useful tool in both the automatic and semi-interactive setting for semantic analysis of film, with potential application to analogues occuring in many other domains, including news, training video, sitcoms.

Detection of setting and subject information in documentary video

Interpretation of video information is a difficult task for computer vision and machine intelligence. In this paper we examine the utility of a non-image based source of information about video contents, namely the shot list, and study its use in aiding image interpretation. We show how the shot list may be analysed to produce a simple summary of the 'who and where' of a documentary or interview video. In order to detect the subject of a video we use the notion of a 'shot syntax' of a particular genre to isolate actual interview sections.

Neighborhood coherence and edge based approaches to film scene extraction

In order to enable high-level semantics-based video annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from film production to determine when a scene change occurs in film. We examine different rules and conventions followed as part of Film Grammar to guide and shape our algorithmic solution for determining a scene boundary. Two different techniques are proposed as new solutions in this paper. Our experimental results on 10 full-length movies show that our technique based on shot sequence coherence performs well and reasonably better than the color edges-based approach.

Topic transition detection using hierarchical hidden Markov and semi-Markov models

In this paper we introduce a probabilistic framework to exploit hierarchy, structure sharing and duration information for topic transition detection in videos. Our probabilistic detection framework is a combination of a shot classification step and a detection phase using hierarchical probabilistic models. We consider two models in this paper: the extended Hierarchical Hidden Markov Model (HHMM) and the Coxian Switching Hidden semi-Markov Model (S-HSMM) because they allow the natural decomposition of semantics in videos, including shared structures, to be modeled directly, and thus enabling efficient inference and reducing the sample complexity in learning. Additionally, the S-HSMM allows the duration information to be incorporated, consequently the modeling of long-term dependencies in videos is enriched through both hierarchical and duration modeling. Furthermore, the use of the Coxian distribution in the S-HSMM makes it tractable to deal with long sequences in video. Our experimentation of the proposed framework on twelve educational and training videos shows that both models outperform the baseline cases (flat HMM and HSMM) and performances reported in earlier work in topic detection. The superior performance of the S-HSMM over the HHMM verifies our belief that duration information is an important factor in video content modeling.

Object matching using boundary descriptors

The problem of object recognition is of immense practical importance and potential, and the last decade has witnessed a number of breakthroughs in the state of the art. Most of the past object recognition work focuses on textured objects and local appearance descriptors extracted around salient points in an image. These methods fail in the matching of smooth, untextured objects for which salient point detection does not produce robust results. The recently proposed bag of boundaries (BoB) method is the first to directly address this problem. Since the texture of smooth objects is largely uninformative, BoB focuses on describing and matching objects based on their post-segmentation boundaries. Herein we address three major weaknesses of this work. The first of these is the uniform treatment of all boundary segments. Instead, we describe a method for detecting the locations and scales of salient boundary segments. Secondly, while the BoB method uses an image based elementary descriptor (HoGs + occupancy matrix), we propose a more compact descriptor based on the local profile of boundary normals’ directions. Lastly, we conduct a far more systematic evaluation, both of the bag of boundaries method and the method proposed here. Using a large public database, we demonstrate that our method exhibits greater robustness while at the same time achieving a major computational saving – object representation is extracted from an image in only 6% of the time needed to extract a bag of boundaries, and the storage requirement is similarly reduced to less than 8%.

Fingertips detection for human interaction system

Fingertips of human hand play an important role in hand-based interaction with computers. Identification of fingertips' positions in hand images is vital for developing a human computer interaction system. This paper proposes a novel method for detecting fingertips of a hand image analyzing the concept of the geometrical structural information of fingers. The research is divided into three parts: First, hand image is segmented for detecting hand, Second, invariant features (curvature zero-crossing points) are extracted from the boundary of the hand, Third, fingertips are detected. Experimental results show that the proposed approach is promising.

Detection of fingertips based on the combination of color information and circle detection

Fingertips of human hand play an important role in hand-based interaction with computers. Therefore, identification of fingertips' positions on hand image is vital for developing a human computer interaction system. All most all of the research works for fingertips detection, initially isolate hand image from the background image. Most of these techniques develop color based segmentation methods because human skin color possess an exceptional characterises that can be used to isolate hand from the rest of the image quite easily. Sometimes color image segmentation becomes difficult due to illumination and background variations. To make it simple and reliable, this paper proposes a robust method for detecting fingertips of a hand image based on the combination of color segmentation and circle detection. Due to the characteristics of circularity of fingertips regions of hand boundary, any existing circle detection algorithms can be applied to detect circles at fingertips region. It is difficult to detect fingertips solely based on the circle detection method. For this reason, initially the proposed method detects all the circular regions on the image applying Circle Hough Transformation (CHT) then the fingertips are selected based on the color characteristics of the fingertips regions. Experimental results show that the proposed approach is promising.

Wireless sensor networks for heritage object deformation detection and tracking algorithm

Deformation is the direct cause of heritage object collapse. It is significant to monitor and signal the early warnings of the deformation of heritage objects. However, traditional heritage object monitoring methods only roughly monitor a simple-shaped heritage object as a whole, but cannot monitor complicated heritage objects, which may have a large number of surfaces inside and outside. Wireless sensor networks, comprising many small-sized, low-cost, low-power intelligent sensor nodes, are more useful to detect the deformation of every small part of the heritage objects. Wireless sensor networks need an effective mechanism to reduce both the communication costs and energy consumption in order to monitor the heritage objects in real time. In this paper, we provide an effective heritage object deformation detection and tracking method using wireless sensor networks (EffeHDDT). In EffeHDDT, we discover a connected core set of sensor nodes to reduce the communication cost for transmitting and collecting the data of the sensor networks. Particularly, we propose a heritage object boundary detecting and tracking mechanism. Both theoretical analysis and experimental results demonstrate that our EffeHDDT method outperforms the existing methods in terms of network traffic and the precision of the deformation detection.