Semi-automatic annotation and retrieval of visual content using the topic map technology

There are still major challenges in the area of automatic indexing and retrieval of multimedia content data for very large multimedia content corpora. Current indexing and retrieval applications still use keywords to index multimedia content and those keywords usually do not provide any knowledge about the semantic content of the data. With the increasing amount of multimedia content, it is inefficient to continue with this approach. In this paper, we describe the project DREAM, which addresses such challenges by proposing a new framework for semi-automatic annotation and retrieval of multimedia based on the semantic content. The framework uses the Topic Map Technology, as a tool to model the knowledge automatically extracted from the multimedia content using an Automatic Labelling Engine. We describe how we acquire knowledge from the content and represent this knowledge using the support of NLP to automatically generate Topic Maps. The framework is described in the context of film post-production.

An enhanced content selection mechanism for personalization of video news programmes

In this paper, we propose a content selection framework that improves the users` experience when they are enriching or authoring pieces of news. This framework combines a variety of techniques to retrieve semantically related videos, based on a set of criteria which are specified automatically depending on the media`s constraints. The combination of different content selection mechanisms can improve the quality of the retrieved scenes, because each technique`s limitations are minimized by other techniques` strengths. We present an evaluation based on a number of experiments, which show that the retrieved results are better when all criteria are used at time.

MoSAC: um módulo para seleção de configurações arquiteturais baseado em critérios de QoS no contexto de sistema distribuídos autoadaptativos

The process for choosing the best components to build systems has become increasingly complex. It becomes more critical if it was need to consider many combinations of components in the context of an architectural configuration. These circumstances occur, mainly, when we have to deal with systems involving critical requirements, such as the timing constraints in distributed multimedia systems, the network bandwidth in mobile applications or even the reliability in real-time systems. This work proposes a process of dynamic selection of architectural configurations based on non-functional requirements criteria of the system, which can be used during a dynamic adaptation. This proposal uses the MAUT theory (Multi-Attribute Utility Theory) for decision making from a finite set of possibilities, which involve multiple criteria to be analyzed. Additionally, it was proposed a metamodel which can be used to describe the application s requirements in terms of the non-functional requirements criteria and their expected values, to express them in order to make the selection of the desired configuration. As a proof of concept, it was implemented a module that performs the dynamic choice of configurations, the MoSAC. This module was implemented using a component-based development approach (CBD), performing a selection of architectural configurations based on the proposed selection process involving multiple criteria. This work also presents a case study where an application was developed in the context of Digital TV to evaluate the time spent on the module to return a valid configuration to be used in a middleware with autoadaptative features, the middleware AdaptTV

Cross-layer optimizations for multimedia distribution over Wireless Multimedia Sensor Networks and Flying Ad-Hoc Networks with quality of experience support

The proliferation of multimedia content and the demand for new audio or video services have fostered the development of a new era based on multimedia information, which allowed the evolution of Wireless Multimedia Sensor Networks (WMSNs) and also Flying Ad-Hoc Networks (FANETs). In this way, live multimedia services require realtime video transmissions with a low frame loss rate, tolerable end-to-end delay, and jitter to support video dissemination with Quality of Experience (QoE) support. Hence, a key principle in a QoE-aware approach is the transmission of high priority frames (protect them) with a minimum packet loss ratio, as well as network overhead. Moreover, multimedia content must be transmitted from a given source to the destination via intermediate nodes with high reliability in a large scale scenario. The routing service must cope with dynamic topologies caused by node failure or mobility, as well as wireless channel changes, in order to continue to operate despite dynamic topologies during multimedia transmission. Finally, understanding user satisfaction on watching a video sequence is becoming a key requirement for delivery of multimedia content with QoE support. With this goal in mind, solutions involving multimedia transmissions must take into account the video characteristics to improve video quality delivery. The main research contributions of this thesis are driven by the research question how to provide multimedia distribution with high energy-efficiency, reliability, robustness, scalability, and QoE support over wireless ad hoc networks. The thesis addresses several problem domains with contributions on different layers of the communication stack. At the application layer, we introduce a QoE-aware packet redundancy mechanism to reduce the impact of the unreliable and lossy nature of wireless environment to disseminate live multimedia content. At the network layer, we introduce two routing protocols, namely video-aware Multi-hop and multi-path hierarchical routing protocol for Efficient VIdeo transmission for static WMSN scenarios (MEVI), and cross-layer link quality and geographical-aware beaconless OR protocol for multimedia FANET scenarios (XLinGO). Both protocols enable multimedia dissemination with energy-efficiency, reliability and QoE support. This is achieved by combining multiple cross-layer metrics for routing decision in order to establish reliable routes.

Uniquitous internet middleware: architecture design and prototype evaluation

Technology advances in recent years have dramatically changed the way users exploit contents and services available on the Internet, by enforcing pervasive and mobile computing scenarios and enabling access to networked resources almost from everywhere, at anytime, and independently of the device in use. In addition, people increasingly require to customize their experience, by exploiting specific device capabilities and limitations, inherent features of the communication channel in use, and interaction paradigms that significantly differ from the traditional request/response one. So-called Ubiquitous Internet scenario calls for solutions that address many different challenges, such as device mobility, session management, content adaptation, context-awareness and the provisioning of multimodal interfaces. Moreover, new service opportunities demand simple and effective ways to integrate existing resources into new and value added applications, that can also undergo run-time modifications, according to ever-changing execution conditions. Despite service-oriented architectural models are gaining momentum to tame the increasing complexity of composing and orchestrating distributed and heterogeneous functionalities, existing solutions generally lack a unified approach and only provide support for specific Ubiquitous Internet aspects. Moreover, they usually target rather static scenarios and scarcely support the dynamic nature of pervasive access to Internet resources, that can make existing compositions soon become obsolete or inadequate, hence in need of reconfiguration. This thesis proposes a novel middleware approach to comprehensively deal with Ubiquitous Internet facets and assist in establishing innovative application scenarios. We claim that a truly viable ubiquity support infrastructure must neatly decouple distributed resources to integrate and push any kind of content-related logic outside its core layers, by keeping only management and coordination responsibilities. Furthermore, we promote an innovative, open, and dynamic resource composition model that allows to easily describe and enforce complex scenario requirements, and to suitably react to changes in the execution conditions.

Projektbericht: VITA

In traditionellen Übungsarrangements (z.B. handkorrigierte Hausaufgaben oder vorlesungsbegleitende Übungsgruppen) erhalten Studierende ihre Ergebnisse mit Zeitverzögerung, während der Kurs inzwischen fortschreitet. Dadurch besteht die Gefahr, dass Defizite nicht wahrgenommen oder nicht korrigiert werden. Traditionelle Übungsarrangements erfordern bei sinnvollem Einsatz (v. a. kleine Übungsgruppe) eine Personalkapazität, wie sie insbesondere an Fachhochschulen nicht existiert. Ein zweites Problem, das zum Teil infolge des mangelnden Übens auftritt, besteht darin, dass viele Studierende Lehrveranstaltungen nicht nachbereiten. Viele Studierende versuchen stattdessen, sich den Vorlesungsstoff in einem kurzen Zeitraum vor der Prüfung anzueignen, was, wenn es überhaupt gelingt, nicht zu einer nachhaltigen Verankerung der Inhalte führt. In der Folge fehlen in den Nachfolgeveranstaltungen die Grundlagen, die in den vorausgegangenen Veranstaltungen hätten gelegt werden sollen.

Cross-Layer Optimizations for Multimedia Distribution over Wireless Multimedia Sensor Networks and Flying Ad-Hoc Networks with Quality of Experience Support

The proliferation of multimedia content and the demand for new audio or video services have fostered the development of a new era based on multimedia information, which allowed the evolution of Wireless Multimedia Sensor Networks (WMSNs) and also Flying Ad-Hoc Networks (FANETs). In this way, live multimedia services require real-time video transmissions with a low frame loss rate, tolerable end-to-end delay, and jitter to support video dissemination with Quality of Experience (QoE) support. Hence, a key principle in a QoE-aware approach is the transmission of high priority frames (protect them) with a minimum packet loss ratio, as well as network overhead. Moreover, multimedia content must be transmitted from a given source to the destination via intermediate nodes with high reliability in a large scale scenario. The routing service must cope with dynamic topologies caused by node failure or mobility, as well as wireless channel changes, in order to continue to operate despite dynamic topologies during multimedia transmission. Finally, understanding user satisfaction on watching a video sequence is becoming a key requirement for delivery of multimedia content with QoE support. With this goal in mind, solutions involving multimedia transmissions must take into account the video characteristics to improve video quality delivery. The main research contributions of this thesis are driven by the research question how to provide multimedia distribution with high energy-efficiency, reliability, robustness, scalability, and QoE support over wireless ad hoc networks. The thesis addresses several problem domains with contributions on different layers of the communication stack. At the application layer, we introduce a QoE-aware packet redundancy mechanism to reduce the impact of the unreliable and lossy nature of wireless environment to disseminate live multimedia content. At the network layer, we introduce two routing protocols, namely video-aware Multi-hop and multi-path hierarchical routing protocol for Efficient VIdeo transmission for static WMSN scenarios (MEVI), and cross-layer link quality and geographical-aware beaconless OR protocol for multimedia FANET scenarios (XLinGO). Both protocols enable multimedia dissemination with energy-efficiency, reliability and QoE support. This is achieved by combining multiple cross-layer metrics for routing decision in order to establish reliable routes.

A Case for CDN-as-a-Service in the Cloud: A Mobile Cloud Networking Argument

Content Distribution Networks are mandatory components of modern web architectures, with plenty of vendors offering their services. Despite its maturity, new paradigms and architecture models are still being developed in this area. Cloud Computing, on the other hand, is a more recent concept which has expanded extremely quickly, with new services being regularly added to cloud management software suites such as OpenStack. The main contribution of this paper is the architecture and the development of an open source CDN that can be provisioned in an on-demand, pay-as-you-go model thereby enabling the CDN as a Service paradigm. We describe our experience with integration of CDNaaS framework in a cloud environment, as a service for enterprise users. We emphasize the flexibility and elasticity of such a model, with each CDN instance being delivered on-demand and associated to personalized caching policies as well as an optimized choice of Points of Presence based on exact requirements of an enterprise customer. Our development is based on the framework developed in the Mobile Cloud Networking EU FP7 project, which offers its enterprise users a common framework to instantiate and control services. CDNaaS is one of the core support components in this project as is tasked to deliver different type of multimedia content to several thousands of users geographically distributed. It integrates seamlessly in the MCN service life-cycle and as such enjoys all benefits of a common design environment, allowing for an improved interoperability with the rest of the services within the MCN ecosystem.

Adaptación de un reproductor de contenidos multimedia para la recepción de vídeo escalable sobre un protocolo de tiempo real

En los últimos años, debido al notable desarrollo de los terminales portátiles, que han pasado de ser “simples” teléfonos o reproductores a puros ordenadores, ha crecido el número de servicios que ofrecen cada vez mayor cantidad de contenido multimedia a través de internet. Además, la distinta evolución de estos terminales hace que nos encontremos en el mercado con una amplísima gama de productos de diferentes tamaños y capacidades de procesamiento, lo que hace necesario encontrar una fórmula que permita satisfacer la demanda de dichos servicios sea cual sea la naturaleza de nuestro dispositivo. Para poder ofrecer una solución adecuada se ha optado por la integración de un protocolo como RTP y un estándar de video como SVC. RTP (Real-time Transport Protocol), en contraposición a los protocolos de propósito general fue diseñado para aplicaciones de tiempo real por lo que es ideal para el streaming de contenido multimedia. Por su parte, SVC es un estándar de video escalable que permite transmitir en un mismo stream una capa base y múltiples capas de mejora, por lo que podremos adaptar la calidad y tamaño del contenido a la capacidad y tamaño de nuestro dispositivo. El objetivo de este proyecto consiste en integrar y modificar tanto el reproductor MPlayer como la librería RTP live555 de tal forma que sean capaces de soportar el formato SVC sobre el protocolo RTP y montar un sistema servidorcliente para comprobar su funcionamiento. Aunque este proceso esté orientado a llevarse a cabo en un dispositivo móvil, para este proyecto se ha optado por realizarlo en el escenario más sencillo posible, para lo cual, se emitirán secuencias a una máquina virtual alojada en el mismo ordenador que el servidor. ABSTRACT In recent years, due to the remarkable development of mobile devices, which have evolved from "simple" phones or players to computers, the amount of services that offer multimedia content over the internet have shot up. Furthermore, the different evolution of these terminals causes that we can find in the market a wide range of different sizes and processing capabilities, making necessary to find a formula that will satisfy the demand for such services regardless of the nature of our device. In order to provide a suitable solution we have chosen to integrate a protocol as RTP and a video standard as SVC. RTP (Real-time Transport Protocol), in opposition to general purpose protocols was designed for real-time applications making it ideal for media streaming. Meanwhile, SVC is a scalable video standard which can transmit a single stream in a base layer and multiple enhancement layers, so that we can adapt the quality and size of the content to the capacity and size of our device. The objective of this project is to integrate and modify both MPlayer and RTP library live555 so that they support the SVC format over RTP protocol and set up a client-server system to check its behavior. Although this process has been designed to be done on a mobile device, for this project we have chosen to do it in the simplest possible scenario so we will stream to a virtual machine hosted on the same computer where we have the server.

Arquitetura computacional para redes orgânicas e heterogêneas: plano de controle do sistema operacional swarm.

Computational Swarms (enxames computacionais), consistindo da integração de sensores e atuadores inteligentes no nosso mundo conectado, possibilitam uma extensão da info-esfera no mundo físico. Nós chamamos esta info-esfera extendida, cíber-física, de Swarm. Este trabalho propõe uma visão de Swarm onde dispositivos computacionais cooperam dinâmica e oportunisticamente, gerando redes orgânicas e heterogêneas. A tese apresenta uma arquitetura computacional do Plano de Controle do Sistema Operacional do Swarm, que é uma camada de software distribuída embarcada em todos os dispositivos que fazem parte do Swarm, responsável por gerenciar recursos, definindo atores, como descrever e utilizar serviços e recursos (como divulgá-los e descobrí-los, como realizar transações, adaptações de conteúdos e cooperação multiagentes). O projeto da arquitetura foi iniciado com uma revisão da caracterização do conceito de Swarm, revisitando a definição de termos e estabelecendo uma terminologia para ser utilizada. Requisitos e desafios foram identificados e uma visão operacional foi proposta. Esta visão operacional foi exercitada com casos de uso e os elementos arquiteturais foram extraídos dela e organizados em uma arquitetura. A arquitetura foi testada com os casos de uso, gerando revisões do sistema. Cada um dos elementos arquiteturais requereram revisões do estado da arte. Uma prova de conceito do Plano de Controle foi implementada e uma demonstração foi proposta e implementada. A demonstração selecionada foi o Smart Jukebox, que exercita os aspectos distribuídos e a dinamicidade do sistema proposto. Este trabalho apresenta a visão do Swarm computacional e apresenta uma plataforma aplicável na prática. A evolução desta arquitetura pode ser a base de uma rede global, heterogênea e orgânica de redes de dispositivos computacionais alavancando a integração de sistemas cíber-físicos na núvem permitindo a cooperação de sistemas escaláveis e flexíveis, interoperando para alcançar objetivos comuns.

Multimedia Standards

The aim of this paper is to review some of the standards, connected with multimedia and their metadata. We start with MPEG family. MPEG-21 provides an open framework for multimedia delivery and consumption. MPEG- 7 is a multimedia content description standard. With the Internet grow several format were proposed for media scenes description. Some of them are open standards such as: VRML1, X3D2, SMIL3, SVG4, MPEG-4 BIFS, MPEG-4, XMT, MPEG-4, LaSER, COLLADA5, published by ISO, W3C, etc. Television has become the most important mass medium. Standards such as MHEG, DAVIC, Java TV, MHP, GEM, OCAP and ACAP have been developed. Efficient video-streaming is presented. There exist a large number of standards for representing audiovisual metadata. We cover the Material Exchange Format (MXF), the Digital Picture Exchange (DPX), and the Digital Cinema Package (DCP).

Digital Libraries and Portals Saving National Cultural Heritage (IMI–BAS Experience)

The current research activities of the Institute of Mathematics and Informatics at the Bulgarian Academy of Sciences (IMI—BAS) include the study and application of knowledge-based methods for the creation, integration and development of multimedia digital libraries with applications in cultural heritage. This report presents IMI-BAS’s developments at the digital library management systems and portals, i.e. the Bulgarian Iconographical Digital Library, the Bulgarian Folklore Digital Library and the Bulgarian Folklore Artery, etc. developed during the several national and international projects: - "Digital Libraries with Multimedia Content and its Application in Bulgarian Cultural Heritage" (contract 8/21.07.2005 between the IMI–BAS, and the State Agency for Information Technologies and Communications; - FP6/IST/P-027451 PROJECT LOGOS "Knowledge-on-Demand for Ubiquitous Learning", EU FP6, IST, Priority 2.4.13 "Strengthening the Integration of the ICT research effort in an Enlarged Europe" - NSF project D-002-189 SINUS "Semantic Technologies for Web Services and Technology Enhanced Learning". - NSF project IO-03-03/2006 ―Development of Digital Libraries and Information Portal with Virtual Exposition "Bulgarian Folklore Heritage". The presented prototypes aims to provide flexible and effective access to the multimedia presentation of the cultural heritage artefacts and collections, maintaining different forms and format of the digitized information content and rich functionality for interaction. The developments are a result of long- standing interests and work in the technological developments in information systems, knowledge processing and content management systems. The current research activities aims at creating innovative solutions for assembling multimedia digital libraries for collaborative use in specific cultural heritage context, maintaining their semantic interoperability and creating new services for dynamic aggregation of their resources, access improvement, personification, intelligent curation of content, and content protection. The investigations are directed towards the development of distributed tools for aggregating heterogeneous content and ensuring semantic compatibility with the European digital library EUROPEANA, thus providing possibilities for pan- European access to rich digitalised collections of Bulgarian cultural heritage.

The Odrysian Kingdom – a Concept of an Interactive Multimedia Application for Educational Purposes

The presentation of cultural heritage is difficult comprehensive and constantly updated topic. Researchers often focus more on the different techniques to digitize artifacts of cultural heritage. This work focuses on the overall shape and structure of future multimedia application whose specificity is determined by the topic - Odrysian kingdom. Below is presented a concept for structure and content-based information available for individual kings from Odryssae dynasty. Special attention is paid to the presentation of preserved artifacts associated with the reign of specific rulers. The main concept of multimedia application dedicated to the Odrysian kingdom, it is to be used in teaching programs related to cultural heritage and history of antiquity in universities. The aim of designers is that it can be modified easy for use in museums also.

A multimedia indexing and retrieval framework for multimedia database systems

The main challenges of multimedia data retrieval lie in the effective mapping between low-level features and high-level concepts, and in the individual users' subjective perceptions of multimedia content. ^ The objectives of this dissertation are to develop an integrated multimedia indexing and retrieval framework with the aim to bridge the gap between semantic concepts and low-level features. To achieve this goal, a set of core techniques have been developed, including image segmentation, content-based image retrieval, object tracking, video indexing, and video event detection. These core techniques are integrated in a systematic way to enable the semantic search for images/videos, and can be tailored to solve the problems in other multimedia related domains. In image retrieval, two new methods of bridging the semantic gap are proposed: (1) for general content-based image retrieval, a stochastic mechanism is utilized to enable the long-term learning of high-level concepts from a set of training data, such as user access frequencies and access patterns of images. (2) In addition to whole-image retrieval, a novel multiple instance learning framework is proposed for object-based image retrieval, by which a user is allowed to more effectively search for images that contain multiple objects of interest. An enhanced image segmentation algorithm is developed to extract the object information from images. This segmentation algorithm is further used in video indexing and retrieval, by which a robust video shot/scene segmentation method is developed based on low-level visual feature comparison, object tracking, and audio analysis. Based on shot boundaries, a novel data mining framework is further proposed to detect events in soccer videos, while fully utilizing the multi-modality features and object information obtained through video shot/scene detection. ^ Another contribution of this dissertation is the potential of the above techniques to be tailored and applied to other multimedia applications. This is demonstrated by their utilization in traffic video surveillance applications. The enhanced image segmentation algorithm, coupled with an adaptive background learning algorithm, improves the performance of vehicle identification. A sophisticated object tracking algorithm is proposed to track individual vehicles, while the spatial and temporal relationships of vehicle objects are modeled by an abstract semantic model. ^

MUSEUM: Una aplicación de visionado de contenidos multimedia para Smart TV

Tradicionalmente, la televisión ha sido siempre el mejor medio (y hasta no hace mucho, el único) para visionar contenidos. Varios elementos, como áreas de visionado cada vez mayores, distintas posibilidades de conectividad y su presencia en la principal estancia del hogar lo siguen manteniendo como el dispositivo ideal para visionar contenidos. Gracias al acceso a internet que incorporan los últimos modelos se abre un nuevo abanico de posibilidades, permitiendo el acceso (en teoría, depende de las capacidades de cada aparato) a todo el contenido disponible en la red. Esto, junto a la gran área de visionado que ofrecen, lo hacen el elemento ideal para ver contenidos de forma grupal (tanto elementos educacionales como de ocio, para la familia, amigos o cualquier grupo de personas). Aunque cada día los fabricantes convergen hacia un estándar común que permita que una misma aplicación se ejecute en una televisión de cualquier marca sin tener que ser adaptada, lo cierto a día de hoy es que no existe una compatibilidad 100% entre aparatos, por lo que se elige para el desarrollo la plataforma más extendida hasta el momento, Smart TV de Samsung. No obstante, con pequeñas modificaciones en la aplicación se podría hacer compatible con aparatos de otras marcas. En este trabajo fin de grado se ha desarrollado un sistema que simula una visita virtual a un museo, donde se puede acceder a los contenidos disponibles en las distintas salas en forma de vídeos, fotografías o audios a la vez que se ofrece la opción de ver una descripción en texto sobre el contenido. El contenido multimedia se almacena en un repositorio externo (servidor) que envía a la televisión dicha información bajo petición. La estructura del contenido es adaptable de forma que, con pequeños cambios, un programador puede modificar la aplicación para ajustarla a otros casos de uso.De manera más específica:  Un usuario con capacidades de administración en el servidor puede: o Añadir datos de nuevos museos o Organizar la estructura del museo en distintas salas. o Añadir contenido multimedia a cada sala.  Un usuario que instala la aplicación en su TV puede: o Navegar por los distintos museos y salas definidos en la aplicación. o Visualizar el contenido multimedia (vídeo, fotos, audio, texto) que ofrece cada sala.