Concise server-wide causality management for eventually consistent data stores

Large scale distributed data stores rely on optimistic replication to scale and remain highly available in the face of net work partitions. Managing data without coordination results in eventually consistent data stores that allow for concurrent data updates. These systems often use anti-entropy mechanisms (like Merkle Trees) to detect and repair divergent data versions across nodes. However, in practice hash-based data structures are too expensive for large amounts of data and create too many false conflicts. Another aspect of eventual consistency is detecting write conflicts. Logical clocks are often used to track data causality, necessary to detect causally concurrent writes on the same key. However, there is a nonnegligible metadata overhead per key, which also keeps growing with time, proportional with the node churn rate. Another challenge is deleting keys while respecting causality: while the values can be deleted, perkey metadata cannot be permanently removed without coordination. Weintroduceanewcausalitymanagementframeworkforeventuallyconsistentdatastores,thatleveragesnodelogicalclocks(BitmappedVersion Vectors) and a new key logical clock (Dotted Causal Container) to provides advantages on multiple fronts: 1) a new efficient and lightweight anti-entropy mechanism; 2) greatly reduced per-key causality metadata size; 3) accurate key deletes without permanent metadata.

Reliability of operational data from pig herds and performance ratings by veterinarians and pig farmers collected during telephone interviews for the evaluation of a PCV2 piglet vaccination

BackgroundThe aim of the present study was to evaluate the feasibility of using a telephone survey in gaining an understanding of the possible herd and management factors influencing the performance (i.e. safety and efficacy) of a vaccine against porcine circovirus type 2 (PCV2) in a large number of herds and to estimate customers¿ satisfaction.ResultsDatasets from 227 pig herds that currently applied or have applied a PCV2 vaccine were analysed. Since 1-, 2- and 3-site production systems were surveyed, the herds were allocated in one of two subsets, where only applicable variables out of 180 were analysed. Group 1 was comprised of herds with sows, suckling pigs and nursery pigs, whereas herds in Group 2 in all cases kept fattening pigs. Overall 14 variables evaluating the subjective satisfaction with one particular PCV2 vaccine were comingled to an abstract dependent variable for further models, which was characterized by a binary outcome from a cluster analysis: good/excellent satisfaction (green cluster) and moderate satisfaction (red cluster). The other 166 variables comprised information about diagnostics, vaccination, housing, management, were considered as independent variables. In Group 1, herds using the vaccine due to recognised PCV2 related health problems (wasting, mortality or porcine dermatitis and nephropathy syndrome) had a 2.4-fold increased chance (1/OR) of belonging to the green cluster. In the final model for Group 1, the diagnosis of diseases other than PCV2, the reason for vaccine administration being other than PCV2-associated diseases and using a single injection of iron had significant influence on allocating into the green cluster (P¿<¿0.05). In Group 2, only unchanged time or delay of time of vaccination influenced the satisfaction (P¿<¿0.05).ConclusionThe methodology and statistical approach used in this study were feasible to scientifically assess ¿satisfaction¿, and to determine factors influencing farmers¿ and vets¿ opinion about the safety and efficacy of a new vaccine.

Composting at Johnson City; final report on Joint USEPA-TVA Composting Project with operational data, 1967-1971 /

Prepared for U. S. Environmental Protection Agency, Office of Solid Waste Management Programs.

Brooms : operational data /

Cover title.

Building Data Warehouses Using Numbered Information Spaces

An approach for organizing the information in the data warehouses is presented in the paper. The possibilities of the numbered information spaces for building data warehouses are discussed. An application is outlined in the paper.

Estimating data divergence in cloud computing storage systems

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Data management for the ocean sciences - perspectives for the next decade

There is remarkable agreement in expectations today for vastly improved ocean data management a decade from now -- capabilities that will help to bring significant benefits to ocean research and to society. Advancing data management to such a degree, however, will require cultural and policy changes that are slow to effect. The technological foundations upon which data management systems are built are certain to continue advancing rapidly in parallel. These considerations argue for adopting attitudes of pragmatism and realism when planning data management strategies. In this paper we adopt those attitudes as we outline opportunities for progress in ocean data management. We begin with a synopsis of expectations for integrated ocean data management a decade from now. We discuss factors that should be considered by those evaluating candidate “standards”. We highlight challenges and opportunities in a number of technical areas, including “Web 2.0” applications, data modeling, data discovery and metadata, real-time operational data, archival of data, biological data management and satellite data management. We discuss the importance of investments in the development of software toolkits to accelerate progress. We conclude the paper by recommending a few specific, short term targets for implementation, that we believe to be both significant and achievable, and calling for action by community leadership to effect these advancements.

Communicating thematic data quality with web map services

Geospatial information of many kinds, from topographic maps to scientific data, is increasingly being made available through web mapping services. These allow georeferenced map images to be served from data stores and displayed in websites and geographic information systems, where they can be integrated with other geographic information. The Open Geospatial Consortium’s Web Map Service (WMS) standard has been widely adopted in diverse communities for sharing data in this way. However, current services typically provide little or no information about the quality or accuracy of the data they serve. In this paper we will describe the design and implementation of a new “quality-enabled” profile of WMS, which we call “WMS-Q”. This describes how information about data quality can be transmitted to the user through WMS. Such information can exist at many levels, from entire datasets to individual measurements, and includes the many different ways in which data uncertainty can be expressed. We also describe proposed extensions to the Symbology Encoding specification, which include provision for visualizing uncertainty in raster data in a number of different ways, including contours, shading and bivariate colour maps. We shall also describe new open-source implementations of the new specifications, which include both clients and servers.

Cluster identification and characterization in the riser of a circulating fluidized bed from numerical simulation results

A methodology of identification and characterization of coherent structures mostly known as clusters is applied to hydrodynamic results of numerical simulation generated for the riser of a circulating fluidized bed. The numerical simulation is performed using the MICEFLOW code, which includes the two-fluids IIT`s hydrodynamic model B. The methodology for cluster characterization that is used is based in the determination of four characteristics, related to average life time, average volumetric fraction of solid, existing time fraction and frequency of occurrence. The identification of clusters is performed by applying a criterion related to the time average value of the volumetric solid fraction. A qualitative rather than quantitative analysis is performed mainly owing to the unavailability of operational data used in the considered experiments. Concerning qualitative analysis, the simulation results are in good agreement with literature. Some quantitative comparisons between predictions and experiment were also presented to emphasize the capability of the modeling procedure regarding the analysis of macroscopic scale coherent structures. (c) 2007 Elsevier Inc. All rights reserved.

Modelagem de dados de entrada para simulação estocástica de lavra

Mine simulation depends on data that is both coherent and representative of the mining operation. This paper describes a methodology for modeling operational data which has been developed for mine simulation. The methodology has been applied to a case study of an open-pit mine, where the cycle times of the truck fleet have been modeled for mine simulation purposes. The results obtained have shown that once the operational data has been treated using the proposed methodology, the system variables have proven to be adherent to theoretical distributions. The research indicated the need jar tracking the origin of data inconsistencies through the development of a process to manage inconsistent data from the mining operation.

Technical and economical evaluation of a harvester, working under different spacing and planting arrangement conditions in eucalypts plantations

Plantation spacing selection has the primary objective of assigning each tree enough space for maximum growth and best quality to be attained with a minimum cost. From the harvest standpoint, an increase in stand density directly implies a decrease of individual tree volume, reducing also harvester productive capacity. The objective of this research is to assess the effects of several initial spacings and arrangements in eucalyptus plantations on production capacity, operational capacity and costs of forest harvester. Real operational data were collected from two eucalypt plantations at different initial spacing of 6.0, 7.5, 9.0, 12 and 18 m(2) per tree. Simulation data were obtained from a forest harvester simulator. Using spacing (E), mean tree volume (MV), diameter at breast height (DBH) and height (H) values, a stepwise regression test procedure was run, and correlations computed in order to measure their participation in operational capacity. Operational costs were computed with an accounting method proposed by FAO. Mean tree volume (MV) explained 88% of forest harvester operational capacity. Spacing (E) affected 8.5% of harvester operational capacity; wider spacings were related to higher individual tree volumes. Harvesting operation costs were lower in wider spaced treatments.

A comparative benchmarking analysis of main Iberian container terminals: a DEA approach

Benchmarking is an important tool to organisations to improve their productivity, product quality, process efficiency or services. From Benchmarking the organisations could compare their performance with competitors and identify their strengths and weaknesses. This study intends to do a benchmarking analysis on the main Iberian Sea ports with a special focus on their container terminals efficiency. To attain this, the DEA (data envelopment analysis) is used since it is considered by several researchers as the most effective method to quantify a set of key performance indicators. In order to reach a more reliable diagnosis tool the DEA is used together with the data mining in comparing the sea ports operational data of container terminals during 2007.Taking into account that sea ports are global logistics networks the performance evaluation is essential to an effective decision making in order to improve their efficiency and, therefore, their competitiveness.

Desenvolvimento de uma metodologia baseada em redes neuronais para o cálculo da energia produzida por aerogeradores

O uso da energia eólica para a produção de eletricidade apresenta na última década um crescimento apreciável. Monitorizar o desempenho dos aerogeradores torna-se um processo incontornável, quer por motivos financeiros, quer por questões operacionais. Os investimentos despendidos na construção de parques eólicos são muito consideráveis, pelo que é essencial a análise constante dos aspetos preponderantes no retorno do investimento. A maximização da energia produzida por cada aerogerador é o objetivo principal da monitorização dos parques eólicos. Os sistemas Supervisory Control and Data Acquisition (SCADAs) instalados nos parques eólicos permitem uma supervisão em tempo real relativamente ao estado e funcionamento dos aerogeradores, adquirindo uma elevada importância na avaliação dos rendimentos energéticos e anomalias de funcionamento, garantido desta forma melhorias de produtividade. O objetivo deste trabalho é estimar a energia produzida pelos aerogeradores quando ocorrem falhas de comunicação com o seu contador interno ou avaria do mesmo. A ocorrência destas situações não permite a monitorização da energia produzida durante esse período. Foram analisados dados operacionais dos aerogeradores relativos a um parque eólico localizado na zona Norte de Portugal, sendo usados os dados recolhidos pelo sistema SCADA sobre a forma de médias de 10 min referentes ao período de janeiro de 2011 a agosto 2011. O desempenho da rede neuronal depende da qualidade e quantidade do conjunto de dados usados para o treino da rede. Os dados usados devem representar de forma fiel o estado que se pretende para o equipamento. Para a obtenção do objetivo proposto foi fundamental a identificação das grandezas disponíveis a utilizar no método de cálculo da energia produzida. Os resultados obtidos com aplicação das redes neuronais no método de cálculo da energia produzida por aerogeradores demonstram que independentemente do período de indisponibilidade da informação referente à energia produzida é possível estimar o valor da mesma.

Armazéns de dados em bases de dados NoSQL

Na atualidade, existe uma quantidade de dados criados diariamente que ultrapassam em muito as mais otimistas espectativas estabelecidas na década anterior. Estes dados têm origens bastante diversas e apresentam-se sobre várias formas. Este novo conceito que dá pelo nome de Big Data está a colocar novos e rebuscados desafios ao seu armazenamento, tratamento e manipulação. Os tradicionais sistemas de armazenamento não se apresentam como a solução indicada para este problema. Estes desafios são alguns dos mais analisados e dissertados temas informáticos do momento. Várias tecnologias têm emergido com esta nova era, das quais se salienta um novo paradigma de armazenamento, o movimento NoSQL. Esta nova filosofia de armazenamento visa responder às necessidades de armazenamento e processamento destes volumosos e heterogéneos dados. Os armazéns de dados são um dos componentes mais importantes do âmbito Business Intelligence e são, maioritariamente, utilizados como uma ferramenta de apoio aos processos de tomada decisão, levados a cabo no dia-a-dia de uma organização. A sua componente histórica implica que grandes volumes de dados sejam armazenados, tratados e analisados tendo por base os seus repositórios. Algumas organizações começam a ter problemas para gerir e armazenar estes grandes volumes de informação. Esse facto deve-se, em grande parte, à estrutura de armazenamento que lhes serve de base. Os sistemas de gestão de bases de dados relacionais são, há algumas décadas, considerados como o método primordial de armazenamento de informação num armazém de dados. De facto, estes sistemas começam a não se mostrar capazes de armazenar e gerir os dados operacionais das organizações, sendo consequentemente cada vez menos recomendada a sua utilização em armazéns de dados. É intrinsecamente interessante o pensamento de que as bases de dados relacionais começam a perder a luta contra o volume de dados, numa altura em que um novo paradigma de armazenamento surge, exatamente com o intuito de dominar o grande volume inerente aos dados Big Data. Ainda é mais interessante o pensamento de que, possivelmente, estes novos sistemas NoSQL podem trazer vantagens para o mundo dos armazéns de dados. Assim, neste trabalho de mestrado, irá ser estudada a viabilidade e as implicações da adoção de bases de dados NoSQL, no contexto de armazéns de dados, em comparação com a abordagem tradicional, implementada sobre sistemas relacionais. Para alcançar esta tarefa, vários estudos foram operados tendo por base o sistema relacional SQL Server 2014 e os sistemas NoSQL, MongoDB e Cassandra. Várias etapas do processo de desenho e implementação de um armazém de dados foram comparadas entre os três sistemas, sendo que três armazéns de dados distintos foram criados tendo por base cada um dos sistemas. Toda a investigação realizada neste trabalho culmina no confronto da performance de consultas, realizadas nos três sistemas.

Building anonymised database samples

Dissertação para obtenção do Grau de Mestre em Engenharia Informática