Scaling up data mining techniques to large datasets using parallel and distributed processing

Advances in hardware and software technology enable us to collect, store and distribute large quantities of data on a very large scale. Automatically discovering and extracting hidden knowledge in the form of patterns from these large data volumes is known as data mining. Data mining technology is not only a part of business intelligence, but is also used in many other application areas such as research, marketing and financial analytics. For example medical scientists can use patterns extracted from historic patient data in order to determine if a new patient is likely to respond positively to a particular treatment or not; marketing analysts can use extracted patterns from customer data for future advertisement campaigns; finance experts have an interest in patterns that forecast the development of certain stock market shares for investment recommendations. However, extracting knowledge in the form of patterns from massive data volumes imposes a number of computational challenges in terms of processing time, memory, bandwidth and power consumption. These challenges have led to the development of parallel and distributed data analysis approaches and the utilisation of Grid and Cloud computing. This chapter gives an overview of parallel and distributed computing approaches and how they can be used to scale up data mining to large datasets.

Big data: il valore dei dati e applicazioni all'analisi strategica

Il presente lavoro di tesi tende a un duplice scopo: il primo è quello di fornire una accurata analisi tecnica, applicativa e culturale riguardante il vasto mondo dei big data e il secondo quello di trovare connessioni con l’analisi strategica verificando se e in quale modo i big data possano risultare una risorsa distintiva in campo aziendale. Nello specifico il primo capitolo presenta i big data nelle sue caratteristiche più importanti cercando di approfondire gli aspetti tecnici del fenomeno, le fonti di produzione dei dati, le metodologie principali di analisi e l’impatto sulla società. Il secondo capitolo descrive svariate applicazioni dei big data in campo aziendale concentrandosi sul rapporto tra questi e l’analisi strategica, non trascurando temi come il vantaggio competitivo e la business intelligence. Infine il terzo capitolo analizza la condizione attuale, il punto di vista italiano ed eventuali sviluppi futuri del fenomeno.

Progettazione di uno strumento di analisi dati per sistemi SCADA-ENERGY Management System

L’obiettivo di questa tesi è approfondire le competenze sulle funzionalità sviluppate nei sistemi SCADA/EMS presenti sul mercato, così da conoscerne le potenzialità offerte: tutte le conoscenze acquisite servono a progettare uno strumento di analisi dati flessibile e interattivo, con il quale è possibile svolgere analisi non proponibili con le altre soluzioni analizzate. La progettazione dello strumento di analisi dei dati è orientata a definire un modello multidimensionale per la rappresentazione delle informazioni: il percorso di progettazione richiede di individuare le informazioni d’interesse per l’utente, così da poterle reintrodurre in fase di progettazione della nuova base dati. L’infrastruttura finale di questa nuova funzionalità si concretizza in un data warehouse: tutte le informazioni di analisi sono memorizzare su una base dati diversa da quella di On.Energy, evitando di correlare le prestazione dei due diversi sottosistemi. L’utilizzo di un data warehouse pone le basi per realizzare analisi su lunghi periodi temporali: tutte le tipologie di interrogazione dati comprendono un enorme quantità d’informazioni, esattamente in linea con le caratteristiche delle interrogazioni OLAP

Progettazione di un sistema proattivo per l'identificazione di aggregati

Nel lavoro di tesi è stato studiato il problema del tuning di un data warehouse, in particolare la tecnica maggiormente utilizzata in ambito aziendale, ovvero la creazione degli aggregati. Inoltre, è stato progettato e implementato uno strumento che generi automaticamente l'insieme di viste che meglio risolve il carico di lavoro basato sulle analisi di business più frequenti su quella specifica base di dati.

Progettazione e sviluppo di una soluzione Hadoop per il calcolo di Big Data Analytics

Il presente elaborato ha come oggetto la progettazione e lo sviluppo di una soluzione Hadoop per il Calcolo di Big Data Analytics. Nell'ambito del progetto di monitoraggio dei bottle cooler, le necessità emerse dall'elaborazione di dati in continua crescita, ha richiesto lo sviluppo di una soluzione in grado di sostituire le tradizionali tecniche di ETL, non pi�ù su�fficienti per l'elaborazione di Big Data. L'obiettivo del presente elaborato consiste nel valutare e confrontare le perfomance di elaborazione ottenute, da un lato, dal flusso di ETL tradizionale, e dall'altro dalla soluzione Hadoop implementata sulla base del framework MapReduce.

An Exploratory Study of Data Quality Management Practices in the ERP Software Systems Context

Quality data are not only relevant for successful Data Warehousing or Business Intelligence applications; they are also a precondition for efficient and effective use of Enterprise Resource Planning (ERP) systems. ERP professionals in all kinds of businesses are concerned with data quality issues, as a survey, conducted by the Institute of Information Systems at the University of Bern, has shown. This paper demonstrates, by using results of this survey, why data quality problems in modern ERP systems can occur and suggests how ERP researchers and practitioners can handle issues around the quality of data in an ERP software Environment.

Monitorización inteligente de redes de comunicación

El avance tecnológico de los últimos años ha aumentado la necesidad de guardar enormes cantidades de datos de forma masiva, llegando a una situación de desorden en el proceso de almacenamiento de datos, a su desactualización y a complicar su análisis. Esta situación causó un gran interés para las organizaciones en la búsqueda de un enfoque para obtener información relevante de estos grandes almacenes de datos. Surge así lo que se define como inteligencia de negocio, un conjunto de herramientas, procedimientos y estrategias para llevar a cabo la “extracción de conocimiento”, término con el que se refiere comúnmente a la extracción de información útil para la propia organización. Concretamente en este proyecto, se ha utilizado el enfoque Knowledge Discovery in Databases (KDD), que permite lograr la identificación de patrones y un manejo eficiente de las anomalías que puedan aparecer en una red de comunicaciones. Este enfoque comprende desde la selección de los datos primarios hasta su análisis final para la determinación de patrones. El núcleo de todo el enfoque KDD es la minería de datos, que contiene la tecnología necesaria para la identificación de los patrones mencionados y la extracción de conocimiento. Para ello, se utilizará la herramienta RapidMiner en su versión libre y gratuita, debido a que es más completa y de manejo más sencillo que otras herramientas como KNIME o WEKA. La gestión de una red engloba todo el proceso de despliegue y mantenimiento. Es en este procedimiento donde se recogen y monitorizan todas las anomalías ocasionadas en la red, las cuales pueden almacenarse en un repositorio. El objetivo de este proyecto es realizar un planteamiento teórico y varios experimentos que permitan identificar patrones en registros de anomalías de red. Se ha estudiado el repositorio de MAWI Lab, en el que se han almacenado anomalías diarias. Se trata de buscar indicios característicos anuales detectando patrones. Los diferentes experimentos y procedimientos de este estudio pretenden demostrar la utilidad de la inteligencia de negocio a la hora de extraer información a partir de un almacén de datos masivo, para su posterior análisis o futuros estudios. ABSTRACT. The technological progresses in the recent years required to store a big amount of information in repositories. This information is often in disorder, outdated and needs a complex analysis. This situation has caused a relevant interest in investigating methodologies to obtain important information from these huge data stores. Business intelligence was born as a set of tools, procedures and strategies to implement the "knowledge extraction". Specifically in this project, Knowledge Discovery in Databases (KDD) approach has been used. KDD is one of the most important processes of business intelligence to achieve the identification of patterns and the efficient management of the anomalies in a communications network. This approach includes all necessary stages from the selection of the raw data until the analysis to determine the patterns. The core process of the whole KDD approach is the Data Mining process, which analyzes the information needed to identify the patterns and to extract the knowledge. In this project we use the RapidMiner tool to carry out the Data Mining process, because this tool has more features and is easier to use than other tools like WEKA or KNIME. Network management includes the deployment, supervision and maintenance tasks. Network management process is where all anomalies are collected, monitored, and can be stored in a repository. The goal of this project is to construct a theoretical approach, to implement a prototype and to carry out several experiments that allow identifying patterns in some anomalies records. MAWI Lab repository has been selected to be studied, which contains daily anomalies. The different experiments show the utility of the business intelligence to extract information from big data warehouse.

Novas tendências, o big data: deteção de fraudes e transações no comércio local: SAF-T com tecnologia big data

Dissertação apresentada à Escola Superior de Tecnologia do Instituto Politécnico de Castelo Branco para cumprimento dos requisitos necessários à obtenção do grau de Mestre em Desenvolvimento de Software e Sistemas Interactivos, realizada sob a orientação científica da categoria profissional do orientador Doutor Eurico Ribeiro Lopes, do Instituto Politécnico de Castelo Branco.

Decisions, decisions everywhere (in the open data era).

Abstract: Decision support systems have been widely used for years in companies to gain insights from internal data, thus making successful decisions. Lately, thanks to the increasing availability of open data, these systems are also integrating open data to enrich decision making process with external data. On the other hand, within an open-data scenario, decision support systems can be also useful to decide which data should be opened, not only by considering technical or legal constraints, but other requirements, such as "reusing potential" of data. In this talk, we focus on both issues: (i) open data for decision making, and (ii) decision making for opening data. We will first briefly comment some research problems regarding using open data for decision making. Then, we will give an outline of a novel decision-making approach (based on how open data is being actually used in open-source projects hosted in Github) for supporting open data publication. Bio of the speaker: Jose-Norberto Mazón holds a PhD from the University of Alicante (Spain). He is head of the "Cátedra Telefónica" on Big Data and coordinator of the Computing degree at the University of Alicante. He is also member of the WaKe research group at the University of Alicante. His research work focuses on open data management, data integration and business intelligence within "big data" scenarios, and their application to the tourism domain (smart tourism destinations). He has published his research in international journals, such as Decision Support Systems, Information Sciences, Data & Knowledge Engineering or ACM Transaction on the Web. Finally, he is involved in the open data project in the University of Alicante, including its open data portal at http://datos.ua.es

Context-awareness privacy in data communications

Internet users consume online targeted advertising based on information collected about them and voluntarily share personal information in social networks. Sensor information and data from smart-phones is collected and used by applications, sometimes in unclear ways. As it happens today with smartphones, in the near future sensors will be shipped in all types of connected devices, enabling ubiquitous information gathering from the physical environment, enabling the vision of Ambient Intelligence. The value of gathered data, if not obvious, can be harnessed through data mining techniques and put to use by enabling personalized and tailored services as well as business intelligence practices, fueling the digital economy. However, the ever-expanding information gathering and use undermines the privacy conceptions of the past. Natural social practices of managing privacy in daily relations are overridden by socially-awkward communication tools, service providers struggle with security issues resulting in harmful data leaks, governments use mass surveillance techniques, the incentives of the digital economy threaten consumer privacy, and the advancement of consumergrade data-gathering technology enables new inter-personal abuses. A wide range of fields attempts to address technology-related privacy problems, however they vary immensely in terms of assumptions, scope and approach. Privacy of future use cases is typically handled vertically, instead of building upon previous work that can be re-contextualized, while current privacy problems are typically addressed per type in a more focused way. Because significant effort was required to make sense of the relations and structure of privacy-related work, this thesis attempts to transmit a structured view of it. It is multi-disciplinary - from cryptography to economics, including distributed systems and information theory - and addresses privacy issues of different natures. As existing work is framed and discussed, the contributions to the state-of-theart done in the scope of this thesis are presented. The contributions add to five distinct areas: 1) identity in distributed systems; 2) future context-aware services; 3) event-based context management; 4) low-latency information flow control; 5) high-dimensional dataset anonymity. Finally, having laid out such landscape of the privacy-preserving work, the current and future privacy challenges are discussed, considering not only technical but also socio-economic perspectives.

Hotel Vila Galé Lagos

Dissertação de Mestrado, Direção e Gestão Hoteleira, Escola Superior de Gestão, Hotelaria e Turismo, Universidade do Algarve, 2016

Desenvolvimento de um sistema de Data Warehousing para a consolidação da informação num grupo empresarial

Pretende-se desenvolver um Data Warehouse para um grupo empresarial constituído por quatro empresas, tendo como objectivo primordial a consolidação de informação. A consolidação da informação é de extrema utilidade, uma vez que as empresas podem ter dados comuns, tais como, produtos ou clientes. O principal objectivo dos sistemas analíticos é permitir analisar os dados dos sistemas transacionais da organização, fazendo com que os utilizadores que nada percebem destes sistemas consigam ter apoio nas tomadas decisão de uma forma simples e eficaz. A utilização do Data Warehouse é útil no apoio a decisões, uma vez que torna os utilizadores autónomos na realização de análises. Os utilizadores deixam de estar dependentes de especialistas em informática para efectuar as suas consultas e passam a ser eles próprios a realizá-las. Por conseguinte, o tempo de execução de uma consulta através do Data Warehouse é de poucos segundos, ao contrário das consultas criadas anteriormente pelos especialistas que por vezes demoravam horas a ser executadas. __ ABSTRACT: lt is intended to develop a Data Warehouse for a business related group of four companies, having by main goal the information consolidation. This information consolidation is of extreme usefulness since the companies can have common data, such as products or customers. The main goal of the analytical systems is to allow analyze data from the organization transactional systems, making that the users that do not understand anything of these systems may have support in a simple and effective way in every process of taking decisions. Using the Data Warehouse is useful to support decisions, once it will allow users to become autonomous in carrying out analysis. Users will no longer depend on computer experts to make their own queries and they can do it themselves. Therefore, the time of a query through the Data Warehouse takes only a few seconds, unlike the earlier queries created previously by experts that sometimes took hours to run.

The impact of spatial data redundancy on SOLAP query performance

Geographic Data Warehouses (GDW) are one of the main technologies used in decision-making processes and spatial analysis, and the literature proposes several conceptual and logical data models for GDW. However, little effort has been focused on studying how spatial data redundancy affects SOLAP (Spatial On-Line Analytical Processing) query performance over GDW. In this paper, we investigate this issue. Firstly, we compare redundant and non-redundant GDW schemas and conclude that redundancy is related to high performance losses. We also analyze the issue of indexing, aiming at improving SOLAP query performance on a redundant GDW. Comparisons of the SB-index approach, the star-join aided by R-tree and the star-join aided by GiST indicate that the SB-index significantly improves the elapsed time in query processing from 25% up to 99% with regard to SOLAP queries defined over the spatial predicates of intersection, enclosure and containment and applied to roll-up and drill-down operations. We also investigate the impact of the increase in data volume on the performance. The increase did not impair the performance of the SB-index, which highly improved the elapsed time in query processing. Performance tests also show that the SB-index is far more compact than the star-join, requiring only a small fraction of at most 0.20% of the volume. Moreover, we propose a specific enhancement of the SB-index to deal with spatial data redundancy. This enhancement improved performance from 80 to 91% for redundant GDW schemas.

Remote control of CNC machines using the CyberOPC communication system over public networks

During the last few years, the evolution of fieldbus and computers networks allowed the integration of different communication systems involving both production single cells and production cells, as well as other systems for business intelligence, supervision and control. Several well-adopted communication technologies exist today for public and non-public networks. Since most of the industrial applications are time-critical, the requirements of communication systems for remote control differ from common applications for computer networks accessing the Internet, such as Web, e-mail and file transfer. The solution proposed and outlined in this work is called CyberOPC. It includes the study and the implementation of a new open communication system for remote control of industrial CNC machines, making the transmission delay for time-critical control data shorter than other OPC-based solutions, and fulfilling cyber security requirements.