Providing 3D video services: the challenge from 2D to 3DTV quality of experience

Recently, three-dimensional (3D) video has decisively burst onto the entertainment industry scene, and has arrived in households even before the standardization process has been completed. 3D television (3DTV) adoption and deployment can be seen as a major leap in television history, similar to previous transitions from black and white (B&W) to color, from analog to digital television (TV), and from standard definition to high definition. In this paper, we analyze current 3D video technology trends in order to define a taxonomy of the availability and possible introduction of 3D-based services. We also propose an audiovisual network services architecture which provides a smooth transition from two-dimensional (2D) to 3DTV in an Internet Protocol (IP)-based scenario. Based on subjective assessment tests, we also analyze those factors which will influence the quality of experience in those 3D video services, focusing on effects of both coding and transmission errors. In addition, examples of the application of the architecture and results of assessment tests are provided.

A framework for the analysis and optimization of encoding latency for multiview video

We present a novel framework for the analysis and optimization of encoding latency for multiview video. Firstly, we characterize the elements that have an influence in the encoding latency performance: (i) the multiview prediction structure and (ii) the hardware encoder model. Then, we provide algorithms to find the encoding latency of any arbitrary multiview prediction structure. The proposed framework relies on the directed acyclic graph encoder latency (DAGEL) model, which provides an abstraction of the processing capacity of the encoder by considering an unbounded number of processors. Using graph theoretic algorithms, the DAGEL model allows us to compute the encoding latency of a given prediction structure, and determine the contribution of the prediction dependencies to it. As an example of DAGEL application, we propose an algorithm to reduce the encoding latency of a given multiview prediction structure up to a target value. In our approach, a minimum number of frame dependencies are pruned, until the latency target value is achieved, thus minimizing the degradation of the rate-distortion performance due to the removal of the prediction dependencies. Finally, we analyze the latency performance of the DAGEL derived prediction structures in multiview encoders with limited processing capacity.

Minimally invasive surgical video analysis: a powerful tool for surgical training and navigation

Analysis of minimally invasive surgical videos is a powerful tool to drive new solutions for achieving reproducible training programs, objective and transparent assessment systems and navigation tools to assist surgeons and improve patient safety. This paper presents how video analysis contributes to the development of new cognitive and motor training and assessment programs as well as new paradigms for image-guided surgery.

Reconfigurable video code (RVC) : aplicación para automatizar la asignación de procesos RUC a los núcleos de un sistema multiprocesador

El esquema actual que existe en el ámbito de la normalización y el diseño de nuevos estándares de codificación de vídeo se está convirtiendo en una tarea difícil de satisfacer la evolución y dinamismo de la comunidad de codificación de vídeo. El problema estaba centrado principalmente en poder explotar todas las características y similitudes entre los diferentes códecs y estándares de codificación. Esto ha obligado a tener que rediseñar algunas partes comunes a varios estándares de codificación. Este problema originó la aparición de una nueva iniciativa de normalización dentro del comité ISO/IEC MPEG, llamado Reconfigurable Video Coding (RVC). Su principal idea era desarrollar un estándar de codificación de vídeo que actualizase e incrementase progresivamente una biblioteca de los componentes, aportando flexibilidad y la capacidad de tener un código reconfigurable mediante el uso de un nuevo lenguaje orientado a flujo de Actores/datos denominado CAL. Este lenguaje se usa para la especificación de la biblioteca estándar y para la creación de instancias del modelo del decodificador. Más tarde, se desarrolló un nuevo estándar de codificación de vídeo denominado High Efficiency Video Coding (HEVC), que actualmente se encuentra en continuo proceso de actualización y desarrollo, que mejorase la eficiencia y compresión de la codificación de vídeo. Obviamente se ha desarrollado una visión de HEVC empleando la metodología de RVC. En este PFC, se emplean diferentes implementaciones de estándares empleando RVC. Por ejemplo mediante los decodificadores Mpeg 4 Part 2 SP y Mpeg 4 Part 10 CBP y PHP así como del nuevo estándar de codificación HEVC, resaltando las características y utilidad de cada uno de ellos. En RVC los algoritmos se describen mediante una clase de actores que intercambian flujos de datos (tokens) para realizar diferentes acciones. El objetivo de este proyecto es desarrollar un programa que, partiendo de los decodificadores anteriormente mencionados, una serie de secuencia de vídeo en diferentes formatos de compresión y una distribución estándar de los actores (para cada uno de los decodificadores), sea capaz de generar diferentes distribuciones de los actores del decodificador sobre uno o varios procesadores del sistema sobre el que se ejecuta, para conseguir la mayor eficiencia en la codificación del vídeo. La finalidad del programa desarrollado en este proyecto es la de facilitar la realización de las distribuciones de los actores sobre los núcleos del sistema, y obtener las mejores configuraciones posibles de una manera automática y eficiente. ABSTRACT. The current scheme that exists in the field of standardization and the design of new video coding standards is becoming a difficult task to meet the evolving and dynamic community of video encoding. The problem was centered mainly in order to exploit all the features and similarities between different codecs and encoding standards. This has forced redesigning some parts common to several coding standards. This problem led to the emergence of a new initiative for standardization within the ISO / IEC MPEG committee, called Reconfigurable Video Coding (RVC). His main idea was to develop a video coding standard and gradually incrementase to update a library of components, providing flexibility and the ability to have a reconfigurable code using a new flow -oriented language Actors / data called CAL. This language is used for the specification of the standard library and to the instantiation model decoder. Later, a new video coding standard called High Efficiency Video Coding (HEVC), which currently is in continuous process of updating and development, which would improve the compression efficiency and video coding is developed. Obviously has developed a vision of using the methodology HEVC RVC. In this PFC, different implementations using RVC standard are used. For example, using decoders MPEG 4 Part 2 SP and MPEG 4 Part 10 CBP and PHP and the new coding standard HEVC, highlighting the features and usefulness of each. In RVC, the algorithms are described by a class of actors that exchange streams of data (tokens) to perform different actions. The objective of this project is to develop a program that, based on the aforementioned decoders, a series of video stream in different compression formats and a standard distribution of actors (for each of the decoders), is capable of generating different distributions decoder actors on one or more processors of the system on which it runs, to achieve greater efficiency in video coding. The purpose of the program developed in this project is to facilitate the realization of the distributions of the actors on the cores of the system, and get the best possible settings automatically and efficiently.

Innovations in the video game industry: changing global markets

The paper examines the video game industry in the perspective of being the paradigm of innovation in digital media and content. In particular, it analyses the response to two main factors that have impacted this industry over the last decade. First, it tracks the evolution of its global market and its emerging geography with the rise of Asia. Second, within this global landscape the paper explores how the changes derived from mobile and on-line gaming enabled major transformations of this industry. From here, some conclusions on the lessons from the evolution of this sector for the whole media and content industries are presented.

Perceptual distortion modeling for side-by-side 3D video delivery

A frame-level distortion model based on perceptual features of the human visual system is proposed to improve the performance of unequal error protection strategies and provide better quality of experience to users in Side-by-Side 3D video delivery systems.

dOTM: a mechanism for distributing centralized multi-party video conferencing in the cloud

One of the key factors for a given application to take advantage of cloud computing is the ability to scale in an efficient, fast and reliable way. In centralized multi-party video conferencing, dynamically scaling a running conversation is a complex problem. In this paper we propose a methodology to divide the Multipoint Control Unit (the video conferencing server) into more simple units, broadcasters. Each broadcaster receives the media from a participant, processes it and forwards it to the rest. These broadcasters can be distributed among a group of CPUs. By using this methodology, video conferencing systems can scale in a more granular way, improving the deployment.

Insertion of impairments in test video sequences for quality assessment based on psychovisual characteristics

Assessing video quality is a complex task. While most pixel-based metrics do not present enough correlation between objective and subjective results, algorithms need to correspond to human perception when analyzing quality in a video sequence. For analyzing the perceived quality derived from concrete video artifacts in determined region of interest we present a novel methodology for generating test sequences which allow the analysis of impact of each individual distortion. Through results obtained after subjective assessment it is possible to create psychovisual models based on weighting pixels belonging to different regions of interest distributed by color, position, motion or content. Interesting results are obtained in subjective assessment which demonstrates the necessity of new metrics adapted to human visual system.

3D simulation of complex shading affecting PV systems taking benefit from the power of graphics cards developed for the video game industry

Shading reduces the power output of a photovoltaic (PV) system. The design engineering of PV systems requires modeling and evaluating shading losses. Some PV systems are affected by complex shading scenes whose resulting PV energy losses are very difficult to evaluate with current modeling tools. Several specialized PV design and simulation software include the possibility to evaluate shading losses. They generally possess a Graphical User Interface (GUI) through which the user can draw a 3D shading scene, and then evaluate its corresponding PV energy losses. The complexity of the objects that these tools can handle is relatively limited. We have created a software solution, 3DPV, which allows evaluating the energy losses induced by complex 3D scenes on PV generators. The 3D objects can be imported from specialized 3D modeling software or from a 3D object library. The shadows cast by this 3D scene on the PV generator are then directly evaluated from the Graphics Processing Unit (GPU). Thanks to the recent development of GPUs for the video game industry, the shadows can be evaluated with a very high spatial resolution that reaches well beyond the PV cell level, in very short calculation times. A PV simulation model then translates the geometrical shading into PV energy output losses. 3DPV has been implemented using WebGL, which allows it to run directly from a Web browser, without requiring any local installation from the user. This also allows taken full benefits from the information already available from Internet, such as the 3D object libraries. This contribution describes, step by step, the method that allows 3DPV to evaluate the PV energy losses caused by complex shading. We then illustrate the results of this methodology to several application cases that are encountered in the world of PV systems design. Keywords: 3D, modeling, simulation, GPU, shading, losses, shadow mapping, solar, photovoltaic, PV, WebGL

3D video quality of experience - influence of scale and crosstalk

This paper gives an overview of three recent studies by the authors on the topic of 3D video Quality of Experience (QoE). Two of studies [1,2] investigated different psychological dimension that may be needed for describing 3D video QoE and the third the visibility and annoyance of crosstalk[3]. The results shows that the video quality scale could be sufficient for evaluating S3D video experience for coding and spatial resolution reduction distortions. It was also confirmed that with a more complex mixture of degradations more than one scale should be used to capture the QoE in these cases. The study found a linear relationship between the perceived crosstalk and the amount of crosstalk.

Quality of experience and HTTP adaptive streaming: a review of subjective studies

HTTP adaptive streaming technology has become widely spread in multimedia services because of its ability to provide adaptation to characteristics of various viewing devices and dynamic network conditions. There are various studies targeting the optimization of adaptation strategy. However, in order to provide an optimal viewing experience to the end-user, it is crucial to get knowledge about the Quality of Experience (QoE) of different adaptation schemes. This paper overviews the state of the art concerning subjective evaluation of adaptive streaming QoE and highlights the challenges and open research questions related to QoE assessment.

Systematic analysis of the decoding delay in multiview video

We present a framework for the analysis of the decoding delay in multiview video coding (MVC). We show that in real-time applications, an accurate estimation of the decoding delay is essential to achieve a minimum communication latency. As opposed to single-view codecs, the complexity of the multiview prediction structure and the parallel decoding of several views requires a systematic analysis of this decoding delay, which we solve using graph theory and a model of the decoder hardware architecture. Our framework assumes a decoder implementation in general purpose multi-core processors with multi-threading capabilities. For this hardware model, we show that frame processing times depend on the computational load of the decoder and we provide an iterative algorithm to compute jointly frame processing times and decoding delay. Finally, we show that decoding delay analysis can be applied to design decoders with the objective of minimizing the communication latency of the MVC system.

Desarrollo de un portal de monitorización y análisis de estadísticas de sesiones de comunicación en tiempo real

En este Trabajo de Fin de Grado se va a explicar el procedimiento seguido a la hora de estudiar, diseñar y desarrollar Ackuaria, un portal de monitorización y análisis de estadísticas de comunicaciones en tiempo real. Después, se mostrarán los resultados obtenidos y la interfaz gráfica desarrollada para una mejor experiencia de usuario. Ackuaria se apoyará en el uso de Licode, un proyecto de código libre desarrollado en la Universidad Politécnica de Madrid, más concretamente en el Grupo de Internet de Nueva Generación de la Escuela Técnica Superior de Ingenieros de Telecomunicación. Licode ofrece la posibilidad de crear un servicio de streaming y videoconferencia en la propia infraestructura del usuario. Está diseñado para ser totalmente escalable y su uso está orientado principalmente al Cloud, aunque es perfectamente utilizable en una infraestructura física. Licode a su vez se basa en WebRTC, un protocolo desarrollado por la W3C (World Wide Web Consortium) y el IETF (Internet Engineering Task Force) pensado para poder transmitir y recibir flujos de audio, video y datos a través del navegador. No necesita ninguna instalación adicional, por lo que establecer una sesión de videoconferencia Peer-to-Peer es realmente sencillo. Con Licode se usa una MCU (Multipoint Control Unit) para evitar que todas las conexiones entre los usuarios sean Peer-To-Peer. Actúa como un cliente WebRTC más por el que pasan todos los flujos, que se encarga de multiplexar y redirigir donde sea necesario. De esta forma se ahorra ancho de banda y recursos del dispositivo de una forma muy significativa. Existe la creciente necesidad de los usuarios de Licode y de cualquier servicio de videoconferencia en general de poder gestionar su infraestructura a partir de datos y estadísticas fiables. Sus objetivos son muy variados: desde estudiar el comportamiento de WebRTC en distintos escenarios hasta monitorizar el uso de los usuarios para poder contabilizar después el tiempo publicado por cada uno. En todos los casos era común la necesidad de disponer de una herramienta que permitiese conocer en todo momento qué está pasando en el servicio de Licode, así como de almacenar toda la información para poder ser analizada posteriormente. Para conseguir desarrollar Ackuaria se ha realizado un estudio de las comunicaciones en tiempo real con el objetivo de determinar qué parámetros era indispensable y útil monitorizar. A partir de este estudio se ha actualizado la arquitectura de Licode para que obtuviese todos los datos necesarios y los enviase de forma que pudiesen ser recogidos por Ackuaria. El portal de monitorización entonces tratará esa información y la mostrará de forma clara y ordenada, además de proporcionar una API REST al usuario.

Foreground segmentation in depth imagery using depth and spatial dynamic models for video surveillance applications

Low-cost systems that can obtain a high-quality foreground segmentation almostindependently of the existing illumination conditions for indoor environments are verydesirable, especially for security and surveillance applications. In this paper, a novelforeground segmentation algorithm that uses only a Kinect depth sensor is proposedto satisfy the aforementioned system characteristics. This is achieved by combininga mixture of Gaussians-based background subtraction algorithm with a new Bayesiannetwork that robustly predicts the foreground/background regions between consecutivetime steps. The Bayesian network explicitly exploits the intrinsic characteristics ofthe depth data by means of two dynamic models that estimate the spatial and depthevolution of the foreground/background regions. The most remarkable contribution is thedepth-based dynamic model that predicts the changes in the foreground depth distributionbetween consecutive time steps. This is a key difference with regard to visible imagery,where the color/gray distribution of the foreground is typically assumed to be constant.Experiments carried out on two different depth-based databases demonstrate that theproposed combination of algorithms is able to obtain a more accurate segmentation of theforeground/background than other state-of-the art approaches.

Optimized HOG for on-road video based vehicle verification

Vision-based object detection from a moving platform becomes particularly challenging in the field of advanced driver assistance systems (ADAS). In this context, onboard vision-based vehicle verification strategies become critical, facing challenges derived from the variability of vehicles appearance, illumination, and vehicle speed. In this paper, an optimized HOG configuration for onboard vehicle verification is proposed which not only considers its spatial and orientation resolution, but descriptor processing strategies and classification. An in-depth analysis of the optimal settings for HOG for onboard vehicle verification is presented, in the context of SVM classification with different kernels. In contrast to many existing approaches, the evaluation is realized in a public and heterogeneous database of vehicle and non-vehicle images in different areas of the road, rendering excellent verification rates that outperform other similar approaches in the literature.