Middleware for Large-scale Distributed Systems

Nos últimos anos o aumento exponencial da utilização de dispositivos móveis e serviços disponibilizados na “Cloud” levou a que a forma como os sistemas são desenhados e implementados mudasse, numa perspectiva de tentar alcançar requisitos que até então não eram essenciais. Analisando esta evolução, com o enorme aumento dos dispositivos móveis, como os “smartphones” e “tablets” fez com que o desenho e implementação de sistemas distribuidos fossem ainda mais importantes nesta área, na tentativa de promover sistemas e aplicações que fossem mais flexíveis, robutos, escaláveis e acima de tudo interoperáveis. A menor capacidade de processamento ou armazenamento destes dispositivos tornou essencial o aparecimento e crescimento de tecnologias que prometem solucionar muitos dos problemas identificados. O aparecimento do conceito de Middleware visa solucionar estas lacunas nos sistemas distribuidos mais evoluídos, promovendo uma solução a nível de organização e desenho da arquitetura dos sistemas, ao memo tempo que fornece comunicações extremamente rápidas, seguras e de confiança. Uma arquitetura baseada em Middleware visa dotar os sistemas de um canal de comunicação que fornece uma forte interoperabilidade, escalabilidade, e segurança na troca de mensagens, entre outras vantagens. Nesta tese vários tipos e exemplos de sistemas distribuídos e são descritos e analisados, assim como uma descrição em detalhe de três protocolos (XMPP, AMQP e DDS) de comunicação, sendo dois deles (XMPP e AMQP) utilzados em projecto reais que serão descritos ao longo desta tese. O principal objetivo da escrita desta tese é demonstrar o estudo e o levantamento do estado da arte relativamente ao conceito de Middleware aplicado a sistemas distribuídos de larga escala, provando que a utilização de um Middleware pode facilitar e agilizar o desenho e desenvolvimento de um sistema distribuído e traz enormes vantagens num futuro próximo.

New visualization model for large scale biosignals analysis

Benefits of long-term monitoring have drawn considerable attention in healthcare. Since the acquired data provides an important source of information to clinicians and researchers, the choice for long-term monitoring studies has become frequent. However, long-term monitoring can result in massive datasets, which makes the analysis of the acquired biosignals a challenge. In this case, visualization, which is a key point in signal analysis, presents several limitations and the annotations handling in which some machine learning algorithms depend on, turn out to be a complex task. In order to overcome these problems a novel web-based application for biosignals visualization and annotation in a fast and user friendly way was developed. This was possible through the study and implementation of a visualization model. The main process of this model, the visualization process, comprised the constitution of the domain problem, the abstraction design, the development of a multilevel visualization and the study and choice of the visualization techniques that better communicate the information carried by the data. In a second process, the visual encoding variables were the study target. Finally, the improved interaction exploration techniques were implemented where the annotation handling stands out. Three case studies are presented and discussed and a usability study supports the reliability of the implemented work.

Analysis of panoramio photo tags in order to extract land use information

In the recent past, hardly anyone could predict this course of GIS development. GIS is moving from desktop to cloud. Web 2.0 enabled people to input data into web. These data are becoming increasingly geolocated. Big amounts of data formed something that is called "Big Data". Scientists still don't know how to deal with it completely. Different Data Mining tools are used for trying to extract some useful information from this Big Data. In our study, we also deal with one part of these data - User Generated Geographic Content (UGGC). The Panoramio initiative allows people to upload photos and describe them with tags. These photos are geolocated, which means that they have exact location on the Earth's surface according to a certain spatial reference system. By using Data Mining tools, we are trying to answer if it is possible to extract land use information from Panoramio photo tags. Also, we tried to answer to what extent this information could be accurate. At the end, we compared different Data Mining methods in order to distinguish which one has the most suited performances for this kind of data, which is text. Our answers are quite encouraging. With more than 70% of accuracy, we proved that extracting land use information is possible to some extent. Also, we found Memory Based Reasoning (MBR) method the most suitable method for this kind of data in all cases.

Text mining e twitter : o poder das redes sociais num mercado competitivo

Actualmente, com a massificação da utilização das redes sociais, as empresas passam a sua mensagem nos seus canais de comunicação, mas os consumidores dão a sua opinião sobre ela. Argumentam, opinam, criticam (Nardi, Schiano, Gumbrecht, & Swartz, 2004). Positiva ou negativamente. Neste contexto o Text Mining surge como uma abordagem interessante para a resposta à necessidade de obter conhecimento a partir dos dados existentes. Neste trabalho utilizámos um algoritmo de Clustering hierárquico com o objectivo de descobrir temas distintos num conjunto de tweets obtidos ao longo de um determinado período de tempo para as empresas Burger King e McDonald’s. Com o intuito de compreender o sentimento associado a estes temas foi feita uma análise de sentimentos a cada tema encontrado, utilizando um algoritmo Bag-of-Words. Concluiu-se que o algoritmo de Clustering foi capaz de encontrar temas através do tweets obtidos, essencialmente ligados a produtos e serviços comercializados pelas empresas. O algoritmo de Sentiment Analysis atribuiu um sentimento a esses temas, permitindo compreender de entre os produtos/serviços identificados quais os que obtiveram uma polaridade positiva ou negativa, e deste modo sinalizar potencias situações problemáticas na estratégia das empresas, e situações positivas passíveis de identificação de decisões operacionais bem-sucedidas.

O Twitter como agente facilitador de recolha e interpretação de sentimentos: Exemplo na escolha da palavra do ano

Desde 2009 que a Porto Editora elege em “infopédia.pt” a palavra que melhor representa os anos que terminam. Este trabalho apresenta uma forma alternativa a essa eleição, substituindo a votação dos cidadãos pela recolha de dados da rede social Twitter ao longo do ano, e procedendo à análise dos mesmos em substituição da votação. Assim sendo, foram recolhidos dados associados às dez palavras finalistas incluídas no conjunto da palavra do ano 2014, os quais foram armazenados em ambiente Hadoop para seguidamente e recorrendo a dois lexicons ser possível a classificação dos tweets. Os lexicons utilizados incluem, por um lado, a lista de palavras positivas e negativas e, por outro, as polaridades associadas às palavras em conjugação com o top vinte e cinco de emoticons utilizados no Twitter. Os resultados obtidos permitem identificar a palavra mais referida e o sentimento, positivo ou negativo associado à mesma.

Column-based databases: estudo exploratório no âmbito das bases de dados NoSQL

O aumento da quantidade de dados gerados que se tem verificado nos últimos anos e a que se tem vindo a dar o nome de Big Data levou a que a tecnologia relacional começasse a demonstrar algumas fragilidades no seu armazenamento e manuseamento o que levou ao aparecimento das bases de dados NoSQL. Estas estão divididas por quatro tipos distintos nomeadamente chave/valor, documentos, grafos e famílias de colunas. Este artigo é focado nas bases de dados do tipo column-based e nele serão analisados os dois sistemas deste tipo considerados mais relevantes: Cassandra e HBase.

The quantified-Self and wearable technologies in the workplace: implications and challenges for their implementations

Introduction of technologies in the workplace have led to a dramatic change. These changes have come with an increased capacity to gather data about one’s working performance (i.e. productivity), as well as the capacity to track one’s personal responses (i.e. emotional, physiological, etc.) to this changing workplace environment. This movement of self-monitoring or self-sensing using diverse types of wearable sensors combined with the use of computing has been identified as the Quantified-Self. Miniaturization of sensors, reduction in cost and a non-stop increase in the computer power capacity has led to a panacea of wearables and sensors to track and analyze all types of information. Utilized in the personal sphere to track information, a looming question remains, should employers use the information from the Quantified-Self to track their employees’ performance or well-being in the workplace and will this benefit employees? The aim of the present work is to layout the implications and challenges associated with the use of Quantified-Self information in the workplace. The Quantified-Self movement has enabled people to understand their personal life better by tracking multiple information and signals; such an approach could allow companies to gather knowledge on what drives productivity for their business and/or well-being of their employees. A discussion about the implications of this approach will cover 1) Monitoring health and well-being, 2) Oversight and safety, and 3) Mentoring and training. Challenges will address the question of 1) Privacy and Acceptability, 2) Scalability and 3) Creativity. Even though many questions remain regarding their use in the workplace, wearable technologies and Quantified-Self data in the workplace represent an exciting opportunity for the industry and health and safety practitioners who will be using them.

Column-based databases: estudo exploratório no âmbito das bases de dados NoSQL

Dissertação de mestrado integrado em Engenharia e Gestão de Sistemas de Informação

Document-based databases: estudo comparativo no âmbito das bases de dados NoSql

Dissertação de mestrado integrado em Engenharia e Gestão de Sistemas de Informação

DnaSP, DNA polymorphism analyses by the coalescent and other methods

DnaSP is a software package for the analysis of DNA polymorphism data. Present version introduces several new modules and features which, among other options allow: (1) handling big data sets (~5 Mb per sequence); (2) conducting a large number of coalescent-based tests by Monte Carlo computer simulations; (3) extensive analyses of the genetic differentiation and gene flow among populations; (4) analysing the evolutionary pattern of preferred and unpreferred codons; (5) generating graphical outputs for an easy visualization of results. Availability: The software package, including complete documentation and examples, is freely available to academic users from: http://www.ub.es/dnasp

A Framework for Building Hypercubes Using MapReduce

The European Space Agency's Gaia mission will create the largest and most precise three dimensional chart of our galaxy (the Milky Way), by providing unprecedented position, parallax, proper motion, and radial velocity measurements for about one billion stars. The resulting catalogue will be made available to the scientific community and will be analyzed in many different ways, including the production of a variety of statistics. The latter will often entail the generation of multidimensional histograms and hypercubes as part of the precomputed statistics for each data release, or for scientific analysis involving either the final data products or the raw data coming from the satellite instruments. In this paper we present and analyze a generic framework that allows the hypercube generation to be easily done within a MapReduce infrastructure, providing all the advantages of the new Big Data analysis paradigmbut without dealing with any specific interface to the lower level distributed system implementation (Hadoop). Furthermore, we show how executing the framework for different data storage model configurations (i.e. row or column oriented) and compression techniques can considerably improve the response time of this type of workload for the currently available simulated data of the mission. In addition, we put forward the advantages and shortcomings of the deployment of the framework on a public cloud provider, benchmark against other popular solutions available (that are not always the best for such ad-hoc applications), and describe some user experiences with the framework, which was employed for a number of dedicated astronomical data analysis techniques workshops.

A Framework for Building Hypercubes Using MapReduce

The European Space Agency's Gaia mission will create the largest and most precise three dimensional chart of our galaxy (the Milky Way), by providing unprecedented position, parallax, proper motion, and radial velocity measurements for about one billion stars. The resulting catalogue will be made available to the scientific community and will be analyzed in many different ways, including the production of a variety of statistics. The latter will often entail the generation of multidimensional histograms and hypercubes as part of the precomputed statistics for each data release, or for scientific analysis involving either the final data products or the raw data coming from the satellite instruments. In this paper we present and analyze a generic framework that allows the hypercube generation to be easily done within a MapReduce infrastructure, providing all the advantages of the new Big Data analysis paradigmbut without dealing with any specific interface to the lower level distributed system implementation (Hadoop). Furthermore, we show how executing the framework for different data storage model configurations (i.e. row or column oriented) and compression techniques can considerably improve the response time of this type of workload for the currently available simulated data of the mission. In addition, we put forward the advantages and shortcomings of the deployment of the framework on a public cloud provider, benchmark against other popular solutions available (that are not always the best for such ad-hoc applications), and describe some user experiences with the framework, which was employed for a number of dedicated astronomical data analysis techniques workshops.

Partial correlation : Network analysis

The final year project came to us as an opportunity to get involved in a topic which has appeared to be attractive during the learning process of majoring in economics: statistics and its application to the analysis of economic data, i.e. econometrics.Moreover, the combination of econometrics and computer science is a very hot topic nowadays, given the Information Technologies boom in the last decades and the consequent exponential increase in the amount of data collected and stored day by day. Data analysts able to deal with Big Data and to find useful results from it are verydemanded in these days and, according to our understanding, the work they do, although sometimes controversial in terms of ethics, is a clear source of value added both for private corporations and the public sector. For these reasons, the essence of this project is the study of a statistical instrument valid for the analysis of large datasets which is directly related to computer science: Partial Correlation Networks.The structure of the project has been determined by our objectives through the development of it. At first, the characteristics of the studied instrument are explained, from the basic ideas up to the features of the model behind it, with the final goal of presenting SPACE model as a tool for estimating interconnections in between elements in large data sets. Afterwards, an illustrated simulation is performed in order to show the power and efficiency of the model presented. And at last, the model is put into practice by analyzing a relatively large data set of real world data, with the objective of assessing whether the proposed statistical instrument is valid and useful when applied to a real multivariate time series. In short, our main goals are to present the model and evaluate if Partial Correlation Network Analysis is an effective, useful instrument and allows finding valuable results from Big Data.As a result, the findings all along this project suggest the Partial Correlation Estimation by Joint Sparse Regression Models approach presented by Peng et al. (2009) to work well under the assumption of sparsity of data. Moreover, partial correlation networks are shown to be a very valid tool to represent cross-sectional interconnections in between elements in large data sets.The scope of this project is however limited, as there are some sections in which deeper analysis would have been appropriate. Considering intertemporal connections in between elements, the choice of the tuning parameter lambda, or a deeper analysis of the results in the real data application are examples of aspects in which this project could be completed.To sum up, the analyzed statistical tool has been proved to be a very useful instrument to find relationships that connect the elements present in a large data set. And after all, partial correlation networks allow the owner of this set to observe and analyze the existing linkages that could have been omitted otherwise.

Análisis y creación de un sistema informático de soporte a entidades sanitarias para la detección precoz de brotes de gripe en épocas de máximo riesgo.

Hoy en día, las posibilidades del Big Data son incontables. Existe gran cantidad de información generada por la población general y disponible de forma pública.El reto consiste en poder trabajar con esta información y extraer conclusiones útiles y que generen valor.En este proyecto, queremos analizar en el tiempo el interés general de la población respecto a una enfermedad común como la gripe, y poder relacionarlos con brotes de gripe existentes en el pasado, para de esta manera, poder extrapolar y predecir futuros brotes.Esta información, en manos de las autoridades sanitarias, puede ser de gran ayuda para poder prevenir picos de solicitudes en los servicios de urgencias, anticipándose para gestionar de manera más eficaz los recursos disponibles, consiguiendo, de esta manera, un mejor servicio a la población en general.De esta manera, son los propios usuarios los que, sin saberlo, posibilitan una mayor y mejor respuesta en los servicios sanitarios mediante la información que ellos mismos distribuyen libremente, consiguiéndose de esta manera valiosos beneficios para la población general.

Análisis y creación de un sistema informático de soporte a entidades sanitarias para la detección precoz de brotes de gripe en épocas de máximo riesgo

En este proyecto queremos analizar en el tiempo el interés general de la población respecto a una enfermedad común como la gripe, y poder relacionarlos con brotes de gripe existentes en el pasado, para de esta manera, poder extrapolar y predecir futuros brotes. Esta información, en manos de las autoridades sanitarias, puede ser de gran ayuda para poder prevenir picos de solicitudes en los servicios de urgencias, anticipándose para gestionar de manera más eficaz los recursos disponibles, consiguiendo, de esta manera, un mejor servicio a la población en general.