A scalable multimedia content processing framework with application to TV shopping

The advent of smart TVs has reshaped the TV-consumer interaction by combining TVs with mobile-like applications and access to the Internet. However, consumers are still unable to seamlessly interact with the contents being streamed. An example of such limitation is TV shopping, in which a consumer makes a purchase of a product or item displayed in the current TV show. Currently, consumers can only stop the current show and attempt to find a similar item in the Web or an actual store. It would be more convenient if the consumer could interact with the TV to purchase interesting items. ^ Towards the realization of TV shopping, this dissertation proposes a scalable multimedia content processing framework. Two main challenges in TV shopping are addressed: the efficient detection of products in the content stream, and the retrieval of similar products given a consumer-selected product. The proposed framework consists of three components. The first component performs computational and temporal aware multimedia abstraction to select a reduced number of frames that summarize the important information in the video stream. By both reducing the number of frames and taking into account the computational cost of the subsequent detection phase, this component component allows the efficient detection of products in the stream. The second component realizes the detection phase. It executes scalable product detection using multi-cue optimization. Additional information cues are formulated into an optimization problem that allows the detection of complex products, i.e., those that do not have a rigid form and can appear in various poses. After the second component identifies products in the video stream, the consumer can select an interesting one for which similar ones must be located in a product database. To this end, the third component of the framework consists of an efficient, multi-dimensional, tree-based indexing method for multimedia databases. The proposed index mechanism serves as the backbone of the search. Moreover, it is able to efficiently bridge the semantic gap and perception subjectivity issues during the retrieval process to provide more relevant results.^

Um framework baseado em modelos para desenvolvimento de sistemas multimídia distribuídos autoadaptativos

Distributed multimedia systems have highly variable characteristics, resulting in new requirements while new technologies become available or in the need for adequacy in accordance with the amount of available resources. So, these systems should provide support for dynamic adaptations in order to adjust their structures and behaviors at runtime. This paper presents an approach to adaptation model-based and proposes a reflective and component-based framework for construction and support of self-adaptive distributed multimedia systems, providing many facilities for the development and evolution of such systems, such as dynamic adaptation. The propose is to keep one or more models to represent the system at runtime, so some external entity can perform an analysis of these models by identifying problems and trying to solve them. These models integrate the reflective meta-level, acting as a system self-representation. The framework defines a meta-model for description of self-adaptive distributed multimedia applications, which can represent components and their relationships, policies for QoS specification and adaptation actions. Additionally, this paper proposes an ADL and architecture for model-based adaptation. As a case study, this paper presents some scenarios to demonstrate the application of the framework in practice, with and without the use of ADL, as well as check some characteristics related to dynamic adaptation

Personalised Advertising Supported by Agents

This paper reports the development of a B2B platform for the personalization of the publicity transmitted during the program intervals. The platform as a whole must ensure that the intervals are filled with ads compatible with the profile, context and expressed interests of the viewers. The platform acts as an electronic marketplace for advertising agencies (content producer companies) and multimedia content providers (content distribution companies). The companies, once registered at the platform, are represented by agents who negotiate automatically the price of the interval timeslots according to the specified price range and adaptation behaviour. The candidate ads for a given viewer interval are selected through a matching mechanism between ad, viewer and the current context (program being watched) profiles. The overall architecture of the platform consists of a multiagent system organized into three layers consisting of: (i) interface agents that interact with companies; (ii) enterprise agents that model the companies, and (iii) delegate agents that negotiate a specific ad or interval. The negotiation follows a variant of the Iterated Contract Net Interaction Protocol (ICNIP) and is based on the price/s offered by the advertising agencies to occupy the viewer’s interval.

Accessibility-by-design: a framework for delivery-context-aware personalised media content re-purposing

This paper describes a framework architecture for the automated re-purposing and efficient delivery of multimedia content stored in CMSs. It deploys specifically designed templates as well as adaptation rules based on a hierarchy of profiles to accommodate user, device and network requirements invoked as constraints in the adaptation process. The user profile provides information in accordance with the opt-in principle, while the device and network profiles provide the operational constraints such as for example resolution and bandwidth limitations. The profiles hierarchy ensures that the adaptation privileges the users' preferences. As part of the adaptation, we took into account the support for users' special needs, and therefore adopted a template-based approach that could simplify the adaptation process integrating accessibility-by-design in the template.

Stars in their eyes: what eye-tracking reveal about multimedia perceptual quality

Perceptual multimedia quality is of paramount importance to the continued take-up and proliferation of multimedia applications: users will not use and pay for applications if they are perceived to be of low quality. Whilst traditionally distributed multimedia quality has been characterised by Quality of Service (QoS) parameters, these neglect the user perspective of the issue of quality. In order to redress this shortcoming, we characterise the user multimedia perspective using the Quality of Perception (QoP) metric, which encompasses not only a user’s satisfaction with the quality of a multimedia presentation, but also his/her ability to analyse, synthesise and assimilate informational content of multimedia. In recognition of the fact that monitoring eye movements offers insights into visual perception, as well as the associated attention mechanisms and cognitive processes, this paper reports on the results of a study investigating the impact of differing multimedia presentation frame rates on user QoP and eye path data. Our results show that provision of higher frame rates, usually assumed to provide better multimedia presentation quality, do not significantly impact upon the median coordinate value of eye path data. Moreover, higher frame rates do not significantly increase level of participant information assimilation, although they do significantly improve overall user enjoyment and quality perception of the multimedia content being shown.

Adapting Web content for low-literacy readers by using lexical elaboration and named entities labeling

This paper presents an approach for assisting low-literacy readers in accessing Web online information. The oEducational FACILITAo tool is a Web content adaptation tool that provides innovative features and follows more intuitive interaction models regarding accessibility concerns. Especially, we propose an interaction model and a Web application that explore the natural language processing tasks of lexical elaboration and named entity labeling for improving Web accessibility. We report on the results obtained from a pilot study on usability analysis carried out with low-literacy users. The preliminary results show that oEducational FACILITAo improves the comprehension of text elements, although the assistance mechanisms might also confuse users when word sense ambiguity is introduced, by gathering, for a complex word, a list of synonyms with multiple meanings. This fact evokes a future solution in which the correct sense for a complex word in a sentence is identified, solving this pervasive characteristic of natural languages. The pilot study also identified that experienced computer users find the tool to be more useful than novice computer users do.

QoE-aware Multiple Path Video Transmission for Wireless Multimedia Sensor Networks

Wireless Multimedia Sensor Networks (WMSNs) promise a wide scope of emerging potential applications in both civilian and military areas, which require visual and audio information to enhance the level of collected information. The transmission of multimedia content requires a minimal video quality level from the user’s perspective. However, links in WMSN communi- cations are typically unreliable, as they often experience fluctuations in quality and weak connectivity, and thus, the routing protocol must evaluate the routes by using end-to-end link quality information to increase the packet delivery ratio. Moreover, the use multiple paths together with key video metrics can enhance the video quality level. In this paper, we propose a video-aware multiple path hierarchical routing protocol for efficient multimedia transmission over WMSN, called video-aware MMtransmission. This protocol finds node-disjoint multiple paths, and implements an end-to-end link quality estimation with minimal over- head to score the paths. Thus, our protocol assures multimedia transmission with Quality of Experience (QoE) and energy-efficiency support. The simula- tion results show the benefits of video-aware MMtransmission for disseminating video content by means of energy-efficiency and QoE analysis.

Meta level component-based framework for distributed computing systems

Adaptability for distributed object-oriented enterprise frameworks in multimedia technology is a critical mission for system evolution. Today, building adaptive services is a complex task due to lack of adequate framework support in the distributed computing systems. In this paper, we propose a Metalevel Component-Based Framework which uses distributed computing design patterns as components to develop an adaptable pattern-oriented framework for distributed computing applications. We describe our approach of combining a meta-architecture with a pattern-oriented framework, resulting in an adaptable framework which provides a mechanism to facilitate system evolution. This approach resolves the problem of dynamic adaptation in the framework, which is encountered in most distributed multimedia applications. The proposed architecture of the pattern-oriented framework has the abilities to dynamically adapt new design patterns to address issues in the domain of distributed computing and they can be woven together to shape the framework in future. © 2011 Springer Science+Business Media B.V.

A synchronous multimedia annotation system for secure collaboratories

In this paper, we describe the Vannotea system - an application designed to enable collaborating groups to discuss and annotate collections of high quality images, video, audio or 3D objects. The system has been designed specifically to capture and share scholarly discourse and annotations about multimedia research data by teams of trusted colleagues within a research or academic environment. As such, it provides: authenticated access to a web browser search interface for discovering and retrieving media objects; a media replay window that can incorporate a variety of embedded plug-ins to render different scientific media formats; an annotation authoring, editing, searching and browsing tool; and session logging and replay capabilities. Annotations are personal remarks, interpretations, questions or references that can be attached to whole files, segments or regions. Vannotea enables annotations to be attached either synchronously (using jabber message passing and audio/video conferencing) or asynchronously and stand-alone. The annotations are stored on an Annotea server, extended for multimedia content. Their access, retrieval and re-use is controlled via Shibboleth identity management and XACML access policies.

Dynamic QoS-Aware coalition formation

Users of wireless devices increasingly demand access to multimedia content with speci c quality of service requirements. Users might tolerate di erent levels of service, or could be satis ed with di erent quality combinations choices. However, multimedia processing introduces heavy resource requirements on the client side. Our work tries to address the growing demand on resources and performance requirements, by allowing wireless nodes to cooperate with each other to meet resource allocation requests and handle stringent constraints, opportunistically taking advantage of the local ad-hoc network that is created spontaneously, as nodes move in range of each other, forming a temporary coalition for service execution. Coalition formation is necessary when a single node cannot execute a speci c service, but it may also be bene cial when groups perform more e ciently when compared to a single s node performance.

Automatic musical instrument recognition for multimedia indexing

Trabalho apresentado no âmbito do Mestrado em Engenharia Informática, como requisito parcial para obtenção do grau de Mestre em Engenharia Informática

Tuning metadata for better movie content-based recommendation systems

The increasing number of television channels, on-demand services and online content, is expected to contribute to a better quality of experience for a costumer of such a service. However, the lack of efficient methods for finding the right content, adapted to personal interests, may lead to a progressive loss of clients. In such a scenario, recommendation systems are seen as a tool that can fill this gap and contribute to the loyalty of users. Multimedia content, namely films and television programmes are usually described using a set of metadata elements that include the title, a genre, the date of production, and the list of directors and actors. This paper provides a deep study on how the use of different metadata elements can contribute to increase the quality of the recommendations suggested. The analysis is conducted using Netflix and Movielens datasets and aspects such as the granularity of the descriptions, the accuracy metric used and the sparsity of the data are taken into account. Comparisons with collaborative approaches are also presented.

Relational navigation and archiving of multimedia information for contemporary dance

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Providing Adaptation for an Adaptive Hypermedia System

O ensino à distância cresceu consideravelmente nos últimos anos e a tendência é para que continue a crescer em anos vindouros. No entanto, enquanto que a maioria das plataformas de ensino à distância utilizam a mesma abordagem de ensino para todos os utilizadores, os estudantes que as usam são na realidade pessoas de diferentes culturas, locais, idades e géneros, e que possuem diferentes níveis de educação. Ao contrário do ensino à distância tradicional, os sistemas de hipermédia adaptativa educacional adaptam interface, apresentação de conteúdos e navegação, entre outros, às características, necessidades e interesses específicos de diferentes utilizadores. Apesar da investigação na área de sistemas de hipermédia adaptativa já estar bastante desenvolvida, é necessário efetuar mais desenvolvimento e experimentação de modo a determinar quais são os aspetos mais eficazes destes sistemas e avaliar o seu sucesso. A Plataforma de Aprendizagem Colaborativa da Matemática (PCMAT) é um sistema de hipermédia adaptativa educacional com uma abordagem construtivista, que foi desenvolvido com o objetivo de contribuir para a investigação na área de sistemas de hipermédia adaptativa. A plataforma avalia o conhecimento do utilizador e apresenta conteúdos e atividades adaptadas às características e estilo de aprendizagem dominante de estudantes de matemática do segundo ciclo. O desenvolvimento do PCMAT tem também o propósito de auxiliar os alunos Portugueses com a aprendizagem da matemática. De acordo com o estudo PISA 2012 da OCDE [OECD, 2014], o desempenho dos alunos Portugueses na área da matemática melhorou em relação à edição anterior do estudo, mas os resultados obtidos permanecem abaixo da média da OCDE. Por este motivo, uma das finalidades deste projeto é desenvolver um sistema de hipermédia adaptativa que, ao adequar o ensino da matemática às necessidades específicas de cada aluno, os assista com a aquisição de conhecimento. A adaptação é efetuada pelo sistema usando a informação constante no modelo do utilizador para definir um grafo de conceitos do domínio específico. Este grafo é adaptado do modelo do domínio e utilizado para dar resposta às necessidades particulares de cada aluno. Embora a trajetória inicial seja definida pelo professor, o percurso percorrido no grafo por cada aluno é determinado pela sua interação com o sistema, usando para o efeito a representação do conhecimento do aluno e outras características disponíveis no modelo do utilizador, assim como avaliação progressiva. A adaptação é conseguida através de alterações na apresentação de conteúdos e na estrutura e anotações das hiperligações. A apresentação de conteúdos é alterada mostrando ou ocultando cada um dos vários fragmentos que compõe as páginas dum curso. Estes fragmentos são compostos por diferentes objetos de aprendizagem, tais como exercícios, figuras, diagramas, etc. As mudanças efetuadas na estrutura e anotações das hiperligações têm o objetivo de guiar o estudante, apontando-o na direção do conhecimento mais relevante e mantendo-o afastado de informação inadequada. A escolha de objectos de aprendizagem adequados às características particulares de cada aluno é um aspecto essencial do modelo de adaptação do PCMAT. A plataforma inclui para esse propósito um módulo responsável pela recomendação de objectos de aprendizagem, e um módulo para a pesquisa e recuperação dos mesmos. O módulo de recomendação utiliza lógica Fuzzy para converter determinados atributos do aluno num conjunto de parâmetros que caracterizam o objecto de aprendizagem que idealmente deveria ser apresentado ao aluno. Uma vez que o objecto “ideal” poderá não existir no repositório de objectos de aprendizagem do sistema, esses parâmetros são utilizados pelo módulo de pesquisa e recuperação para procurar e devolver ao módulo de recomendação uma lista com os objectos que mais se assemelham ao objecto “ideal”. A pesquisa é feita numa árvore k-d usando o algoritmo k-vizinhos mais próximos. O modelo de recomendação utiliza a lista devolvida pelo módulo de pesquisa e recuperação para seleccionar o objecto de aprendizagem mais apropriado para o aluno e processa-o para inclusão numa das páginas Web do curso. O presente documento descreve o trabalho desenvolvido no âmbito do projeto PCMAT (PTDS/CED/108339/2008), dando relevância à adaptação de conteúdos.

Interfaces for television content sharing and annotation

Dissertação para obtenção do Grau de Mestre em Engenharia Informática