Improving Storage Performance Through Layout Optimizations

Disk drives are the bottleneck in the processing of large amounts of data used in almost all common applications. File systems attempt to reduce this by storing data sequentially on the disk drives, thereby reducing the access latencies. Although this strategy is useful when data is retrieved sequentially, the access patterns in real world workloads is not necessarily sequential and this mismatch results in storage I/O performance degradation. This thesis demonstrates that one way to improve the storage performance is to reorganize data on disk drives in the same way in which it is mostly accessed. We identify two classes of accesses: static, where access patterns do not change over the lifetime of the data and dynamic, where access patterns frequently change over short durations of time, and propose, implement and evaluate layout strategies for each of these. Our strategies are implemented in a way that they can be seamlessly integrated or removed from the system as desired. We evaluate our layout strategies for static policies using tree-structured XML data where accesses to the storage device are mostly of two kinds - parent-tochild or child-to-sibling. Our results show that for a specific class of deep-focused queries, the existing file system layout policy performs better by 5-54X. For the non-deep-focused queries, our native layout mechanism shows an improvement of 3-127X. To improve performance of the dynamic access patterns, we implement a self-optimizing storage system that performs rearranges popular block accesses on a dedicated partition based on the observed workload characteristics. Our evaluation shows an improvement of over 80% in the disk busy times over a range of workloads. These results show that applying the knowledge of data access patterns for allocation decisions can substantially improve the I/O performance.

Geochemical data and radiocarbon ages of organic matter fractions and bacteriohopanepolyol data of sediment core GeoB6518-1 from the Congo River basin

The age of organic material discharged by rivers provides information about its sources and carbon cycling processes within watersheds. While elevated ages in fluvially-transported organic matter are usually explained by erosion of soils and sediments, it is commonly assumed that mainly young organic material is discharged from flat tropical watersheds due to their extensive plant cover and high carbon turnover. Here we present compound-specific radiocarbon data of terrigenous organic fractions from a sedimentary archive offshore the Congo River in conjunction with molecular markers for methane-producing land cover reflecting wetland extent in the watershed. We find that the Congo River has been discharging aged organic matter for several thousand years with increasing ages from the mid- to the Late Holocene. This suggests that aged organic matter in modern samples is concealed by radiocarbon from nuclear weapons testing. By comparison to indicators for past rainfall changes we detect a systematic control of organic matter sequestration and release by continental hydrology mediating temporary carbon storage in wetlands. As aridification also leads to exposure and rapid remineralization of large amounts of previously stored labile organic matter we infer that this process may cause a profound direct climate feedback currently underestimated in carbon cycle assessments.

Progettazione e prototipazione di un sistema di Data Stream Processing basato su Apache Storm

Con l’avvento di Internet, il numero di utenti con un effettivo accesso alla rete e la possibilità di condividere informazioni con tutto il mondo è, negli anni, in continua crescita. Con l’introduzione dei social media, in aggiunta, gli utenti sono portati a trasferire sul web una grande quantità di informazioni personali mettendoli a disposizione delle varie aziende. Inoltre, il mondo dell’Internet Of Things, grazie al quale i sensori e le macchine risultano essere agenti sulla rete, permette di avere, per ogni utente, un numero maggiore di dispositivi, direttamente collegati tra loro e alla rete globale. Proporzionalmente a questi fattori anche la mole di dati che vengono generati e immagazzinati sta aumentando in maniera vertiginosa dando luogo alla nascita di un nuovo concetto: i Big Data. Nasce, di conseguenza, la necessità di far ricorso a nuovi strumenti che possano sfruttare la potenza di calcolo oggi offerta dalle architetture più complesse che comprendono, sotto un unico sistema, un insieme di host utili per l’analisi. A tal merito, una quantità di dati così vasta, routine se si parla di Big Data, aggiunta ad una velocità di trasmissione e trasferimento altrettanto alta, rende la memorizzazione dei dati malagevole, tanto meno se le tecniche di storage risultano essere i tradizionali DBMS. Una soluzione relazionale classica, infatti, permetterebbe di processare dati solo su richiesta, producendo ritardi, significative latenze e inevitabile perdita di frazioni di dataset. Occorre, perciò, far ricorso a nuove tecnologie e strumenti consoni a esigenze diverse dalla classica analisi batch. In particolare, è stato preso in considerazione, come argomento di questa tesi, il Data Stream Processing progettando e prototipando un sistema bastato su Apache Storm scegliendo, come campo di applicazione, la cyber security.

Modelling storage-driven connectivity between landscapes and riverscapes : towards a simple framework for long-term ecohydrological assessment

Acknowledgements: We thank Iain Malcolm of Marine Scotland Science for access to data from the Girnock and the Scottish Environment Protection Agency for historical stage-discharge relationships. CS contributions on this paper were in part supported by the NERC/JPI SIWA project (NE/M019896/1).

Resistance and resilience to droughts : hydropedological controls on catchment storage and run-off response

Acknowledgements The authors would like to thank Jonathan Dick, Maria Blumstock, Claire Tunaley and Jason Lessels for assistance with the field work and Audrey Innes for lab sample preparation. Climatic data were provided by Iain Malcolm and Marine Scotland Fisheries at the Freshwater Lab, Pitlochry. Additional precipitation and temperature data were provided by the UK Meteorological Office and the British Atmospheric Data Centre (BADC). We are grateful for the careful and constructive comments of two anonymous reviewers that helped to improve an earlier version of this manuscript. The European Research Council ERC (project GA 335910) is thanked for funding.

Resistance and resilience to droughts : hydropedological controls on catchment storage and run-off response

Acknowledgements The authors would like to thank Jonathan Dick, Maria Blumstock, Claire Tunaley and Jason Lessels for assistance with the field work and Audrey Innes for lab sample preparation. Climatic data were provided by Iain Malcolm and Marine Scotland Fisheries at the Freshwater Lab, Pitlochry. Additional precipitation and temperature data were provided by the UK Meteorological Office and the British Atmospheric Data Centre (BADC). We are grateful for the careful and constructive comments of two anonymous reviewers that helped to improve an earlier version of this manuscript. The European Research Council ERC (project GA 335910) is thanked for funding.

Comparing Remote Data Transfer Rates of Compact Muon Solenoid Jobs with Xrootd and Lustre

To explore the feasibility of processing Compact Muon Solenoid (CMS) analysis jobs across the wide area network, the FIU CMS Tier-3 center and the Florida CMS Tier-2 center designed a remote data access strategy. A Kerberized Lustre test bed was installed at the Tier-2 with the design to provide storage resources to private-facing worker nodes at the Tier-3. However, the Kerberos security layer is not capable of authenticating resources behind a private network. As a remedy, an xrootd server on a public-facing node at the Tier-3 was installed to export the file system to the private-facing worker nodes. We report the performance of CMS analysis jobs processed by the Tier-3 worker nodes accessing data from a Kerberized Lustre file. The processing performance of this configuration is benchmarked against a direct connection to the Lustre file system, and separately, where the xrootd server is near the Lustre file system.

Data S1. Results of the analyses considering developmental time, supplementary figures and tables & Video S1

Our ability to project the impact of global change on marine ecosystem is limited by our poor understanding on how to predict species sensitivity. For example, the impact of ocean acidification is highly species-specific, even in closely related taxa. The aim of this study was to test the hypothesis that the tolerance range of a given species to decreased pH corresponds to their natural range of exposure. Larvae of the green sea urchin Strongylocentrotus droebachiensis were cultured from fertilization to metamorphic competence (29 days) under a wide range of pH (from pHT = 8.0/pCO2 ~ 480 ?atm to pHT = 6.5/pCO2 ~ 20 000 ?atm) covering present (from pHT 8.7 to 7.6), projected near-future variability (from pHT 8.3 to 7.2) and beyond. Decreasing pH impacted all tested parameters (mortality, symmetry, growth, morphometry and respiration). Development of normal, although showing morphological plasticity, swimming larvae was possible as low as pHT >= 7.0. Within that range, decreasing pH increased mortality and asymmetry and decreased body length (BL) growth rate. Larvae raised at lowered pH and with similar BL had shorter arms and a wider body. Relative to a given BL, respiration rates and stomach volume both increased with decreasing pH suggesting changes in energy budget. At the lowest pHs (pHT <= 6.5), all the tested parameters were strongly negatively affected and no larva survived past 13 days post fertilization. In conclusion, sea urchin larvae appeared to be highly plastic when exposed to decreased pH until a physiological tipping point at pHT = 7.0. However, this plasticity was associated with direct (increased mortality) and indirect (decreased growth) consequences for fitness.

Soil organic carbon storage and soil properties for 50 soil profiles in the Lena River Delta including land form description and map

To project the future development of the soil organic carbon (SOC) storage in permafrost environments, the spatial and vertical distribution of key soil properties and their landscape controls needs to be understood. This article reports findings from the Arctic Lena River Delta where we sampled 50 soil pedons. These were classified according to the U.S.D.A. Soil Taxonomy and fall mostly into the Gelisol soil order used for permafrost-affected soils. Soil profiles have been sampled for the active layer (mean depth 58±10 cm) and the upper permafrost to one meter depth. We analyze SOC stocks and key soil properties, i.e. C%, N%, C/N, bulk density, visible ice and water content. These are compared for different landscape groupings of pedons according to geomorphology, soil and land cover and for different vertical depth increments. High vertical resolution plots are used to understand soil development. These show that SOC storage can be highly variable with depth. We recommend the treatment of permafrost-affected soils according to subdivisions into: the surface organic layer, mineral subsoil in the active layer, organic enriched cryoturbated or buried horizons and the mineral subsoil in the permafrost. The major geomorphological units of a subregion of the Lena River Delta were mapped with a land form classification using a data-fusion approach of optical satellite imagery and digital elevation data to upscale SOC storage. Landscape mean SOC storage is estimated to 19.2±2.0 kg C/m**2. Our results show that the geomorphological setting explains more soil variability than soil taxonomy classes or vegetation cover. The soils from the oldest, Pleistocene aged, unit of the delta store the highest amount of SOC per m**2 followed by the Holocene river terrace. The Pleistocene terrace affected by thermal-degradation, the recent floodplain and bare alluvial sediments store considerably less SOC in descending order.

Sharing in the Rain: Secure and Efficient Data Sharing for the Cloud

Cloud storage has rapidly become a cornerstone of many businesses and has moved from an early adopters stage to an early majority, where we typically see explosive deployments. As companies rush to join the cloud revolution, it has become vital to create the necessary tools that will effectively protect users' data from unauthorized access. Nevertheless, sharing data between multiple users' under the same domain in a secure and efficient way is not trivial. In this paper, we propose Sharing in the Rain – a protocol that allows cloud users' to securely share their data based on predefined policies. The proposed protocol is based on Attribute-Based Encryption (ABE) and allows users' to encrypt data based on certain policies and attributes. Moreover, we use a Key-Policy Attribute-Based technique through which access revocation is optimized. More precisely, we show how to securely and efficiently remove access to a file, for a certain user that is misbehaving or is no longer part of a user group, without having to decrypt and re-encrypt the original data with a new key or a new policy.

Big Data Analytics for Earth Sciences: the EarthServer approach

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains.

Big Data Analytics for Earth Sciences: the EarthServer approach

Big Data Analytics is an emerging field since massive storage and computing capabilities have been made available by advanced e-infrastructures. Earth and Environmental sciences are likely to benefit from Big Data Analytics techniques supporting the processing of the large number of Earth Observation datasets currently acquired and generated through observations and simulations. However, Earth Science data and applications present specificities in terms of relevance of the geospatial information, wide heterogeneity of data models and formats, and complexity of processing. Therefore, Big Earth Data Analytics requires specifically tailored techniques and tools. The EarthServer Big Earth Data Analytics engine offers a solution for coverage-type datasets, built around a high performance array database technology, and the adoption and enhancement of standards for service interaction (OGC WCS and WCPS). The EarthServer solution, led by the collection of requirements from scientific communities and international initiatives, provides a holistic approach that ranges from query languages and scalability up to mobile access and visualization. The result is demonstrated and validated through the development of lighthouse applications in the Marine, Geology, Atmospheric, Planetary and Cryospheric science domains.

Local Storage Meets Local Demand: A Technical Solution to Future Power Distribution System

Future power systems are expected to integrate large-scale stochastic and intermittent generation and load due to reduced use of fossil fuel resources, including renewable energy sources (RES) and electric vehicles (EV). Inclusion of such resources poses challenges for the dynamic stability of synchronous transmission and distribution networks, not least in terms of generation where system inertia may not be wholly governed by large-scale generation but displaced by small-scale and localised generation. Energy storage systems (ESS) can limit the impact of dispersed and distributed generation by offering supporting reserve while accommodating large-scale EV connection; the latter (load) also participating in storage provision. In this paper, a local energy storage system (LESS) is proposed. The structure, requirement and optimal sizing of the LESS are discussed. Three operating modes are detailed, including: 1) storage pack management; 2) normal operation; and 3) contingency operation. The proposed LESS scheme is evaluated using simulation studies based on data obtained from the Northern Ireland regional and residential network.

Data-Adaptive Modeling using Convex Regression

Thesis (Ph.D.)--University of Washington, 2016-08