Meter data management for smart monitoring power networks

The electrical power distribution and commercialization scenario is evolving worldwide, and electricity companies, faced with the challenge of new information requirements, are demanding IT solutions to deal with the smart monitoring of power networks. Two main challenges arise from data management and smart monitoring of power networks: real-time data acquisition and big data processing over short time periods. We present a solution in the form of a system architecture that conveys real time issues and has the capacity for big data management.

HDTourist: exploring urban data on Android

The Web of Data currently comprises ? 62 billion triples from more than 2,000 different datasets covering many fields of knowledge3. This volume of structured Linked Data can be seen as a particular case of Big Data, referred to as Big Semantic Data [4]. Obviously, powerful computational configurations are tradi- tionally required to deal with the scalability problems arising to Big Semantic Data. It is not surprising that this ?data revolution? has competed in parallel with the growth of mobile computing. Smartphones and tablets are massively used at the expense of traditional computers but, to date, mobile devices have more limited computation resources. Therefore, one question that we may ask ourselves would be: can (potentially large) semantic datasets be consumed natively on mobile devices? Currently, only a few mobile apps (e.g., [1, 9, 2, 8]) make use of semantic data that they store in the mobile devices, while many others access existing SPARQL endpoints or Linked Data directly. Two main reasons can be considered for this fact. On the one hand, in spite of some initial approaches [6, 3], there are no well-established triplestores for mobile devices. This is an important limitation because any po- tential app must assume both RDF storage and SPARQL resolution. On the other hand, the particular features of these devices (little storage space, less computational power or more limited bandwidths) limit the adoption of seman- tic data for different uses and purposes. This paper introduces our HDTourist mobile application prototype. It con- sumes urban data from DBpedia4 to help tourists visiting a foreign city. Although it is a simple app, its functionality allows illustrating how semantic data can be stored and queried with limited resources. Our prototype is implemented for An- droid, but its foundations, explained in Section 2, can be deployed in any other platform. The app is described in Section 3, and Section 4 concludes about our current achievements and devises the future work.

Telemetría de vehículos FSAE con microcontrolador PIC32 y transceptor de radio CC1101

Quizás el campo de las telecomunicaciones sea uno de los campos en el que más se ha progresado en este último siglo y medio, con la ayuda de otros campos de la ciencia y la técnica tales como la computación, la física electrónica, y un gran número de disciplinas, que se han utilizado estos últimos 150 años en conjunción para mejorarse unas con la ayuda de otras. Por ejemplo, la química ayuda a comprender y mejorar campos como la medicina, que también a su vez se ve mejorada por los progresos en la electrónica creados por los físicos y químicos, que poseen herramientas más potentes para calcular y simular debido a los progresos computacionales. Otro de los campos que ha sufrido un gran avance en este último siglo es el de la automoción, aunque estancados en el motor de combustión, los vehículos han sufrido enormes cambios debido a la irrupción de los avances en la electrónica del automóvil con multitud de sistemas ya ampliamente integrados en los vehículos actuales. La Formula SAE® o Formula Student es una competición de diseño, organizada por la SAE International (Society of Automotive Engineers) para estudiantes de universidades de todo el mundo que promueve la ingeniería a través de una competición donde los miembros del equipo diseñan, construyen, desarrollan y compiten en un pequeño y potente monoplaza. En el ámbito educativo, evitando el sistema tradicional de clases magistrales, se introducen cambios en las metodologías de enseñanza y surge el proyecto de la Fórmula Student para lograr una mejora en las acciones formativas, que permitan ir incorporando nuevos objetivos y diseñar nuevas situaciones de aprendizaje que supongan una oportunidad para el desarrollo de competencias de los alumnos, mejorar su formación como ingenieros y contrastar sus progresos compitiendo con las mejores universidades del mundo. En este proyecto se pretende dotar a los alumnos de las escuelas de ingeniería de la UPM que desarrollan el vehículo de FSAE de una herramienta de telemetría con la que evaluar y probar comportamiento del vehículo de FSAE junto con sus subsistemas que ellos mismos diseñan, con el objetivo de evaluar el comportamiento, introducir mejoras, analizar resultados de una manera más rápida y cómoda, con el objetivo de poder progresar más rápidamente en su desarrollo, recibiendo y almacenando una realimentación directa e instantánea del funcionamiento mediante la lectura de los datos que circulan por el bus CAN del vehículo. También ofrece la posibilidad de inyectar datos a los sistemas conectados al bus CAN de manera remota. Se engloba en el conjunto de proyectos de la FSAE, más concretamente en los basados en la plataforma PIC32 y propone una solución conjunta con otros proyectos o también por sí sola. Para la ejecución del proyecto se fabricó una placa compuesta de dos placas de circuito impreso, la de la estación base que envía comandos, instrucciones y datos para inyectar en el bus CAN del vehículo mediante radiofrecuencia y la placa que incorpora el vehículo que envía las tramas que circulan por el bus CAN del vehículo con los identificadores deseados, ejecuta los comandos recibidos por radiofrecuencia y salva las tramas CAN en una memoria USB o SD Card. Las dos PCBs constituyen el hardware del proyecto. El software se compone de dos programas. Un programa para la PCB del vehículo que emite los datos a la estación base, codificado en lenguaje C con ayuda del entorno de desarrollo MPLAB de Microchip. El otro programa hecho con LabView para la PCB de la estación base que recibe los datos provenientes del vehículo y los interpreta. Se propone un hardware y una capa o funciones de software para los microcontroladores PIC32 (similar al de otros proyectos del FSAE) para la transmisión de las tramas del bus CAN del vehículo de manera inalámbrica a una estación base, capaz de insertar tramas en el bus CAN del vehículo enviadas desde la estación base. También almacena estas tramas CAN en un dispositivo USB o SD Card situado en el vehículo. Para la transmisión de los datos se hizo un estudio de las frecuencias de transmisión, la legislación aplicable y los tipos de transceptores. Se optó por utilizar la banda de radiofrecuencia de uso común ISM de 433MHz mediante el transceptor integrado CC110L de Texas Instruments altamente configurable y con interfaz SPI. Se adquirieron dos parejas de módulos compatibles, con amplificador de potencia o sin él. LabView controla la estación que recoge las tramas CAN vía RF y está dotada del mismo transceptor de radio junto con un puente de comunicaciones SPI-USB, al que se puede acceder de dos diferentes maneras, mediante librerías dll, o mediante NI-VISA con transferencias RAW-USB. La aplicación desarrollada posee una interfaz configurable por el usuario para la muestra de los futuros sensores o actuadores que se incorporen en el vehículo y es capaz de interpretar las tramas CAN, mostrarlas, gráfica, numéricamente y almacenar esta información, como si fuera el cuadro de instrumentos del vehículo. Existe una limitación de la velocidad global del sistema en forma de cuello de botella que se crea debido a las limitaciones del transceptor CC110L por lo que si no se desea filtrar los datos que se crean necesarios, sería necesario aumentar el número de canales de radio para altas ocupaciones del bus CAN. Debido a la pérdida de relaciones con el INSIA, no se pudo probar de manera real en el propio vehículo, pero se hicieron pruebas satisfactorias (hasta 1,6 km) con una configuración de tramas CAN estándar a una velocidad de transmisión de 1 Mbit/s y un tiempo de bit de 1 microsegundo. El periférico CAN del PIC32 se programará para cumplir con estas especificaciones de la ECU del vehículo, que se presupone que es la MS3 Sport de Bosch, de la que LabView interpretará las tramas CAN recibidas de manera inalámbrica. Para poder probar el sistema, ha sido necesario reutilizar el hardware y adaptar el software del primer prototipo creado, que emite tramas CAN preprogramadas con una latencia también programable y que simulará al bus CAN proporcionando los datos a transmitir por el sistema que incorpora el vehículo. Durante el desarrollo de este proyecto, en las etapas finales, el fabricante del puente de comunicaciones SPI-USB MCP2210 liberó una librería (dll) compatible y sin errores, por lo que se nos ofrecía una oportunidad interesante para la comparación de las velocidades de acceso al transceptor de radio, que se presuponía y se comprobó más eficiente que la solución ya hecha mediante NI-VISA. ABSTRACT. The Formula SAE competition is an international university applied to technological innovation in vehicles racing type formula, in which each team, made up of students, should design, construct and test a prototype each year within certain rules. The challenge of FSAE is that it is an educational project farther away than a master class. The goal of the present project is to make a tool for other students to use it in his projects related to FSAE to test and improve the vehicle, and, the improvements that can be provided by the electronics could be materialized in a victory and win the competition with this competitive advantage. A telemetry system was developed. It sends the data provided by the car’s CAN bus through a radio frequency transceiver and receive commands to execute on the system, it provides by a base station on the ground. Moreover, constant verification in real time of the status of the car or data parameters like the revolutions per minute, pressure from collectors, water temperature, and so on, can be accessed from the base station on the ground, so that, it could be possible to study the behaviour of the vehicle in early phases of the car development. A printed circuit board, composed of two boards, and two software programs in two different languages, have been developed, and built for the project implementation. The software utilized to design the PCB is Orcad10.5/Layout. The base station PCB on a PC receives data from the PCB connected to the vehicle’s CAN bus and sends commands like set CAN filters or masks, activate data logger or inject CAN frames. This PCB is connected to a PC via USB and contains a bridge USB-SPI to communicate with a similar transceiver on the vehicle PCB. LabView controls this part of the system. A special virtual Instrument (VI) had been created in order to add future new elements to the vehicle, is a dashboard, which reads the data passed from the main VI and represents them graphically to studying the behaviour of the car on track. In this special VI other alums can make modifications to accommodate the data provided from the vehicle CAN’s bus to new elements on the vehicle, show or save the CAN frames in the form or format they want. Two methods to access to SPI bus of CC110l RF transceiver over LabView have been developed with minimum changes between them. Access through NI-VISA (Virtual Instrument Software Architecture) which is a standard for configuring, programming, USB interfaces or other devices in National Instruments LabView. And access through DLL (dynamic link library) supplied by the manufacturer of the bridge USB-SPI, Microchip. Then the work is done in two forms, but the dll solution developed shows better behaviour, and increase the speed of the system because has less overload of the USB bus due to a better efficiency of the dll solution versus VISA solution. The PCB connected to the vehicle’s CAN bus receives commands from the base station PCB on a PC, and, acts in function of the command or execute actions like to inject packets into CAN bus or activate data logger. Also sends over RF the CAN frames present on the bus, which can be filtered, to avoid unnecessary radio emissions or overflowing the RF transceiver. This PCB consists of two basic pieces: A microcontroller with 32 bit architecture PIC32MX795F512L from Microchip and the radio transceiver integrated circuit CC110l from Texas Instruments. The PIC32MX795F512L has an integrated CAN and several peripherals like SPI controllers that are utilized to communicate with RF transceiver and SD Card. The USB controller on the PIC32 is utilized to store CAN data on a USB memory, and change notification peripheral is utilized like an external interrupt. Hardware for other peripherals is accessible. The software part of this PCB is coded in C with MPLAB from Microchip, and programming over PICkit 3 Programmer, also from Microchip. Some of his libraries have been modified to work properly with this project and other was created specifically for this project. In the phase for RF selection and design is made a study to clarify the general aspects of regulations for the this project in order to understand it and select the proper band, frequency, and radio transceiver for the activities developed in the project. From the different options available it selects a common use band ICM, with less regulation and free to emit with restrictions and disadvantages like high occupation. The transceiver utilized to transmit and receive the data CC110l is an integrated circuit which needs fewer components from Texas Instruments and it can be accessed through SPI bus. Basically is a state machine which changes his state whit commands received over an SPI bus or internal events. The transceiver has several programmable general purpose Inputs and outputs. These GPIOs are connected to PIC32 change notification input to generate an interrupt or connected to GPIO to MCP2210 USB-SPI bridge to inform to the base station for a packet received. A two pair of modules of CC110l radio module kit from different output power has been purchased which includes an antenna. This is to keep away from fabrication mistakes in RF hardware part or designs, although reference design and gerbers files are available on the webpage of the chip manufacturer. A neck bottle is present on the complete system, because the maximum data rate of CC110l transceiver is a half than CAN bus data rate, hence for high occupation of CAN bus is recommendable to filter the data or add more radio channels, because the buffers can’t sustain this load along the time. Unfortunately, during the development of the project, the relations with the INSIA, who develops the vehicle, was lost, for this reason, will be made impossible to test the final phases of the project like integration on the car, final test of integration, place of the antenna, enclosure of the electronics, connectors selection, etc. To test or evaluate the system, it was necessary to simulate the CAN bus with a hardware to feed the system with entry data. An early hardware prototype was adapted his software to send programed CAN frames at a fixed data rate and certain timing who simulate several levels of occupation of the CAN Bus. This CAN frames emulates the Bosch ECU MS3 Sport.

Design and implementation of an adaptive dashboard and visualization interface for the dynamic optimization and control of Data Centers

Over the last few years, the Data Center market has increased exponentially and this tendency continues today. As a direct consequence of this trend, the industry is pushing the development and implementation of different new technologies that would improve the energy consumption efficiency of data centers. An adaptive dashboard would allow the user to monitor the most important parameters of a data center in real time. For that reason, monitoring companies work with IoT big data filtering tools and cloud computing systems to handle the amounts of data obtained from the sensors placed in a data center.Analyzing the market trends in this field we can affirm that the study of predictive algorithms has become an essential area for competitive IT companies. Complex algorithms are used to forecast risk situations based on historical data and warn the user in case of danger. Considering that several different users will interact with this dashboard from IT experts or maintenance staff to accounting managers, it is vital to personalize it automatically. Following that line of though, the dashboard should only show relevant metrics to the user in different formats like overlapped maps or representative graphs among others. These maps will show all the information needed in a visual and easy-to-evaluate way. To sum up, this dashboard will allow the user to visualize and control a wide range of variables. Monitoring essential factors such as average temperature, gradients or hotspots as well as energy and power consumption and savings by rack or building would allow the client to understand how his equipment is behaving, helping him to optimize the energy consumption and efficiency of the racks. It also would help him to prevent possible damages in the equipment with predictive high-tech algorithms.

Lightweight Internet Traffic Classification - A Subject Based Solution with Word Embeddings

Internet traffic classification is a relevant and mature research field, anyway of growing importance and with still open technical challenges, also due to the pervasive presence of Internet-connected devices into everyday life. We claim the need for innovative traffic classification solutions capable of being lightweight, of adopting a domain-based approach, of not only concentrating on application-level protocol categorization but also classifying Internet traffic by subject. To this purpose, this paper originally proposes a classification solution that leverages domain name information extracted from IPFIX summaries, DNS logs, and DHCP leases, with the possibility to be applied to any kind of traffic. Our proposed solution is based on an extension of Word2vec unsupervised learning techniques running on a specialized Apache Spark cluster. In particular, learning techniques are leveraged to generate word-embeddings from a mixed dataset composed by domain names and natural language corpuses in a lightweight way and with general applicability. The paper also reports lessons learnt from our implementation and deployment experience that demonstrates that our solution can process 5500 IPFIX summaries per second on an Apache Spark cluster with 1 slave instance in Amazon EC2 at a cost of $ 3860 year. Reported experimental results about Precision, Recall, F-Measure, Accuracy, and Cohen's Kappa show the feasibility and effectiveness of the proposal. The experiments prove that words contained in domain names do have a relation with the kind of traffic directed towards them, therefore using specifically trained word embeddings we are able to classify them in customizable categories. We also show that training word embeddings on larger natural language corpuses leads improvements in terms of precision up to 180%.

L'implementazione delle piattaforme di E-Procurement: il caso FAAC Group

Questa tesi concerne quella che è una generalizzata tendenza verso la trasformazione digitale dei processi di business. Questa evoluzione, che implica l’utilizzo delle moderne tecnologie informatiche tra cui il Cloud Computing, le Big Data Analytics e gli strumenti Mobile, non è priva di insidie che vanno di volta in volta individuate ed affrontate opportunamente. In particolare si farà riferimento ad un caso aziendale, quello della nota azienda bolognese FAAC spa, ed alla funzione acquisti. Nell'ambito degli approvvigionamenti l'azienda sente la necessità di ristrutturare e digitalizzare il processo di richiesta di offerta (RdO) ai propri fornitori, al fine di consentire alla funzione di acquisti di concentrarsi sull'implementazione della strategia aziendale più che sull'operatività quotidiana. Si procede quindi in questo elaborato all'implementazione di un progetto di implementazione di una piattaforma specifica di e-procurement per la gestione delle RdO. Preliminarmente vengono analizzati alcuni esempi di project management presenti in letteratura e quindi viene definito un modello per la gestione del progetto specifico. Lo svolgimento comprende quindi: una fase di definizione degli obiettivi di continuità dell'azienda, un'analisi As-Is dei processi, la definizione degli obiettivi specifici di progetto e dei KPI di valutazione delle performance, la progettazione della piattaforma software ed infine alcune valutazioni relative ai rischi ed alle alternative dell'implementazione.

A World that Counts: Mobilising the Data Revolution for Sustainable Development

In September 2015, the UN Member States are expected to commit to an ambitious new set of global goals for a new era of sustainable development. Achieving them will require an unprecedented joint effort on the part of governments at every level, civil society and the private sector, and millions of individual choices and actions. To be realised, the SDGs will require a monitoring and accountability framework and a plan for implementation. A commitment to realise the opportunities of the data revolution should be firmly embedded into the action plan for the SDGs, to support those countries most in need of resources, and to set the world on track for an unprecedented push towards a new world of data for change.

Approaches and Strategy for Cancer Research and Surveillance Data: Integration, Information Pipeline, Data Models, and Informatics Opportunities

Thesis (Ph.D.)--University of Washington, 2016-04

Submodular Optimization and Data Processing

Thesis (Ph.D.)--University of Washington, 2016-06

Understanding U.S. regional linguistic variation with Twitter data analysis

We analyze a Big Data set of geo-tagged tweets for a year (Oct. 2013–Oct. 2014) to understand the regional linguistic variation in the U.S. Prior work on regional linguistic variations usually took a long time to collect data and focused on either rural or urban areas. Geo-tagged Twitter data offers an unprecedented database with rich linguistic representation of fine spatiotemporal resolution and continuity. From the one-year Twitter corpus, we extract lexical characteristics for twitter users by summarizing the frequencies of a set of lexical alternations that each user has used. We spatially aggregate and smooth each lexical characteristic to derive county-based linguistic variables, from which orthogonal dimensions are extracted using the principal component analysis (PCA). Finally a regionalization method is used to discover hierarchical dialect regions using the PCA components. The regionalization results reveal interesting linguistic regional variations in the U.S. The discovered regions not only confirm past research findings in the literature but also provide new insights and a more detailed understanding of very recent linguistic patterns in the U.S.

Precompetitive data sharing as a catalyst to address unmet needs in Parkinson's disease

Parkinson's disease is a complex heterogeneous disorder with urgent need for disease-modifying therapies. Progress in successful therapeutic approaches for PD will require an unprecedented level of collaboration. At a workshop hosted by Parkinson's UK and co-organized by Critical Path Institute's (C-Path) Coalition Against Major Diseases (CAMD) Consortiums, investigators from industry, academia, government and regulatory agencies agreed on the need for sharing of data to enable future success. Government agencies included EMA, FDA, NINDS/NIH and IMI (Innovative Medicines Initiative). Emerging discoveries in new biomarkers and genetic endophenotypes are contributing to our understanding of the underlying pathophysiology of PD. In parallel there is growing recognition that early intervention will be key for successful treatments aimed at disease modification. At present, there is a lack of a comprehensive understanding of disease progression and the many factors that contribute to disease progression heterogeneity. Novel therapeutic targets and trial designs that incorporate existing and new biomarkers to evaluate drug effects independently and in combination are required. The integration of robust clinical data sets is viewed as a powerful approach to hasten medical discovery and therapies, as is being realized across diverse disease conditions employing big data analytics for healthcare. The application of lessons learned from parallel efforts is critical to identify barriers and enable a viable path forward. A roadmap is presented for a regulatory, academic, industry and advocacy driven integrated initiative that aims to facilitate and streamline new drug trials and registrations in Parkinson's disease.

Open Research Data

Open Research Data - A step by step guide through the research data lifecycle, data set creation, big data vs long-tail, metadata, data centres/data repositories, open access for data, data sharing, data citation and publication.

A Comparative Analysis of Predictive Learning Algorithms on High-Dimensional Microarray Cancer Data

This research evaluates pattern recognition techniques on a subclass of big data where the dimensionality of the input space (p) is much larger than the number of observations (n). Specifically, we evaluate massive gene expression microarray cancer data where the ratio κ is less than one. We explore the statistical and computational challenges inherent in these high dimensional low sample size (HDLSS) problems and present statistical machine learning methods used to tackle and circumvent these difficulties. Regularization and kernel algorithms were explored in this research using seven datasets where κ < 1. These techniques require special attention to tuning necessitating several extensions of cross-validation to be investigated to support better predictive performance. While no single algorithm was universally the best predictor, the regularization technique produced lower test errors in five of the seven datasets studied.

Labeling large scale social media data using budget-driven One-class SVM classification

The social media classification problems draw more and more attention in the past few years. With the rapid development of Internet and the popularity of computers, there is astronomical amount of information in the social network (social media platforms). The datasets are generally large scale and are often corrupted by noise. The presence of noise in training set has strong impact on the performance of supervised learning (classification) techniques. A budget-driven One-class SVM approach is presented in this thesis that is suitable for large scale social media data classification. Our approach is based on an existing online One-class SVM learning algorithm, referred as STOCS (Self-Tuning One-Class SVM) algorithm. To justify our choice, we first analyze the noise-resilient ability of STOCS using synthetic data. The experiments suggest that STOCS is more robust against label noise than several other existing approaches. Next, to handle big data classification problem for social media data, we introduce several budget driven features, which allow the algorithm to be trained within limited time and under limited memory requirement. Besides, the resulting algorithm can be easily adapted to changes in dynamic data with minimal computational cost. Compared with two state-of-the-art approaches, Lib-Linear and kNN, our approach is shown to be competitive with lower requirements of memory and time.

Progettazione e prototipazione di un sistema di Data Stream Processing basato su Apache Storm

Con l’avvento di Internet, il numero di utenti con un effettivo accesso alla rete e la possibilità di condividere informazioni con tutto il mondo è, negli anni, in continua crescita. Con l’introduzione dei social media, in aggiunta, gli utenti sono portati a trasferire sul web una grande quantità di informazioni personali mettendoli a disposizione delle varie aziende. Inoltre, il mondo dell’Internet Of Things, grazie al quale i sensori e le macchine risultano essere agenti sulla rete, permette di avere, per ogni utente, un numero maggiore di dispositivi, direttamente collegati tra loro e alla rete globale. Proporzionalmente a questi fattori anche la mole di dati che vengono generati e immagazzinati sta aumentando in maniera vertiginosa dando luogo alla nascita di un nuovo concetto: i Big Data. Nasce, di conseguenza, la necessità di far ricorso a nuovi strumenti che possano sfruttare la potenza di calcolo oggi offerta dalle architetture più complesse che comprendono, sotto un unico sistema, un insieme di host utili per l’analisi. A tal merito, una quantità di dati così vasta, routine se si parla di Big Data, aggiunta ad una velocità di trasmissione e trasferimento altrettanto alta, rende la memorizzazione dei dati malagevole, tanto meno se le tecniche di storage risultano essere i tradizionali DBMS. Una soluzione relazionale classica, infatti, permetterebbe di processare dati solo su richiesta, producendo ritardi, significative latenze e inevitabile perdita di frazioni di dataset. Occorre, perciò, far ricorso a nuove tecnologie e strumenti consoni a esigenze diverse dalla classica analisi batch. In particolare, è stato preso in considerazione, come argomento di questa tesi, il Data Stream Processing progettando e prototipando un sistema bastato su Apache Storm scegliendo, come campo di applicazione, la cyber security.