Data analytics in the cloud with flexible mapreduced workflows

Data analytic applications are characterized by large data sets that are subject to a series of processing phases. Some of these phases are executed sequentially but others can be executed concurrently or in parallel on clusters, grids or clouds. The MapReduce programming model has been applied to process large data sets in cluster and cloud environments. For developing an application using MapReduce there is a need to install/configure/access specific frameworks such as Apache Hadoop or Elastic MapReduce in Amazon Cloud. It would be desirable to provide more flexibility in adjusting such configurations according to the application characteristics. Furthermore the composition of the multiple phases of a data analytic application requires the specification of all the phases and their orchestration. The original MapReduce model and environment lacks flexible support for such configuration and composition. Recognizing that scientific workflows have been successfully applied to modeling complex applications, this paper describes our experiments on implementing MapReduce as subworkflows in the AWARD framework (Autonomic Workflow Activities Reconfigurable and Dynamic). A text mining data analytic application is modeled as a complex workflow with multiple phases, where individual workflow nodes support MapReduce computations. As in typical MapReduce environments, the end user only needs to define the application algorithms for input data processing and for the map and reduce functions. In the paper we present experimental results when using the AWARD framework to execute MapReduce workflows deployed over multiple Amazon EC2 (Elastic Compute Cloud) instances.

Tertiary evolution of the São Vicente and Setúbal submarine canyons, Southwest Portugal: insights from seismic stratigraphy

The Setúbal and São Vicente canyons are two major modern submarine canyons located in the southwest Iberian margin of Portugal. Although recognised as Pliocene to Quaternary features, their development during the Tertiary has not been fully understood up to date. A grid of 2D seismic data has been used to characterise the sedimentary deposits of the adjacent flanks to the submarine canyons. The relationship between the geological structure of the margin and the canyon's present location has been investigated. The interpretation of the main seismic units allowed the recognition of three generations of ravinements probably originated after middle Oligocene. Six units grouped in two distinctive seismic sequences have been identified and correlated with offshore stratigraphic data. Seismic Sequence 2 (SS2), the oldest, overlies Mesozoic and upper Eocene deformed units. Seismic Sequence I (SS1) is composed of four different seismic packages separated from SS2 by an erosional surface. The base of the studied sediment ridges is marked by an extensive erosional surface derived from a early/middle Oligocene relative sea-level fall. Deposition in the adjacent area to the actual canyons was reinitiated in late Oligocene in the form of transgressive and channel-fill deposits. A new depositional hiatus is recorded onshore during the Burdigalian, coincident with the unconformity separating SS1 and SS2. This can be correlated with the Arrábida unconformity and with the paroxysmal Burdigalian phase of the Betic domain. Presently, the Setúbal and São Vicente submarine canyons locally cut SS1 and SS2, forming distinctive channels from those recognised on the seismic data. On the upper shelf both dissect highly deformed areas subject to important erosion.

Estimation of signal subspace on hyperspectral data

Dimensionality reduction plays a crucial role in many hyperspectral data processing and analysis algorithms. This paper proposes a new mean squared error based approach to determine the signal subspace in hyperspectral imagery. The method first estimates the signal and noise correlations matrices, then it selects the subset of eigenvalues that best represents the signal subspace in the least square sense. The effectiveness of the proposed method is illustrated using simulated and real hyperspectral images.

A microscope for the data center

Nowadays, data centers are large energy consumers and the trend for next years is expected to increase further, considering the growth in the order of cloud services. A large portion of this power consumption is due to the control of physical parameters of the data center (such as temperature and humidity). However, these physical parameters are tightly coupled with computations, and even more so in upcoming data centers, where the location of workloads can vary substantially due, for example, to workloads being moved in the cloud infrastructure hosted in the data center. Therefore, managing the physical and compute infrastructure of a large data center is an embodiment of a Cyber-Physical System (CPS). In this paper, we describe a data collection and distribution architecture that enables gathering physical parameters of a large data center at a very high temporal and spatial resolution of the sensor measurements. We think this is an important characteristic to enable more accurate heat-flow models of the data center and with them, find opportunities to optimize energy consumptions. Having a high-resolution picture of the data center conditions, also enables minimizing local hot-spots, perform more accurate predictive maintenance (failures in all infrastructure equipments can be more promptly detected) and more accurate billing. We detail this architecture and define the structure of the underlying messaging system that is used to collect and distribute the data. Finally, we show the results of a preliminary study of a typical data center radio environment.

Integer and Fractional Order Entropy Analysis of Earthquake Data-series

This paper studies the statistical distributions of worldwide earthquakes from year 1963 up to year 2012. A Cartesian grid, dividing Earth into geographic regions, is considered. Entropy and the Jensen–Shannon divergence are used to analyze and compare real-world data. Hierarchical clustering and multi-dimensional scaling techniques are adopted for data visualization. Entropy-based indices have the advantage of leading to a single parameter expressing the relationships between the seismic data. Classical and generalized (fractional) entropy and Jensen–Shannon divergence are tested. The generalized measures lead to a clear identification of patterns embedded in the data and contribute to better understand earthquake distributions.

Optimization of fMRI Processing Parameters for Simutaneous Acquisition of EEG/fMRI in Focal Epilepsy

In the context of focal epilepsy, the simultaneous combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) holds a great promise as a technique by which the hemodynamic correlates of interictal spikes detected on scalp EEG can be identified. The fact that traditional EEG recordings have not been able to overcome the difficulty in correlating the ictal clinical symptoms to the onset in particular areas of the lobes, brings the need of mapping with more precision the epileptogenic cortical regions. On the other hand, fMRI suggested localizations more consistent with the ictal clinical manifestations detected. This study was developed in order to improve the knowledge about the way parameters involved in the physical and mathematical data, produced by the EEG/fMRI technique processing, would influence the final results. The evaluation of the accuracy was made by comparing the BOLD results with: the high resolution EEG maps; the malformative lesions detected in the T1 weighted MR images; and the anatomical localizations of the diagnosed symptomatology of each studied patient. The optimization of the set of parameters used, will provide an important contribution to the diagnosis of epileptogenic focuses, in patients included on an epilepsy surgery evaluation program. The results obtained allowed us to conclude that: by associating the BOLD effect with interictal spikes, the epileptogenic areas are mapped to localizations different from those obtained by the EEG maps representing the electrical potential distribution across the scalp (EEG); there is an important and solid bond between the variation of particular parameters (manipulated during the fMRI data processing) and the optimization of the final results, from which smoothing, deleted volumes, HRF (used to convolve with the activation design), and the shape of the Gamma function can be certainly emphasized.

Data fusion with computational intelligence techniques: a case study of fuzzy inference for terrain assessment

Dissertação apresentada para obtenção do Grau de Mestre em Engenharia Electrotécnica e de Computadores, pela Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia

Pervasive adaptive data acquisition gateway for critical healthcare

The data acquisition process in real-time is fundamental to provide appropriate services and improve health professionals decision. In this paper a pervasive adaptive data acquisition architecture of medical devices (e.g. vital signs, ventilators and sensors) is presented. The architecture was deployed in a real context in an Intensive Care Unit. It is providing clinical data in real-time to the INTCare system. The gateway is composed by several agents able to collect a set of patients’ variables (vital signs, ventilation) across the network. The paper shows as example the ventilation acquisition process. The clients are installed in a machine near the patient bed. Then they are connected to the ventilators and the data monitored is sent to a multithreading server which using Health Level Seven protocols records the data in the database. The agents associated to gateway are able to collect, analyse, interpret and store the data in the repository. This gateway is composed by a fault tolerant system that ensures a data store in the database even if the agents are disconnected. The gateway is pervasive, universal, and interoperable and it is able to adapt to any service using streaming data.

Diseño y Desarrollo de un cliente Web Processing Services (WPS) para gvSIG

Configuració d'un entorn de desenvolupament en el IDE Eclipse. Introducció als SIG. Usos, utilitats i exemples. Conèixer la eina gvSIG. Conèixer els estàndards més estesos de l'Open Geospatial Consortium (OGC) i en especial del Web Processing Services. Analitzar, dissenyar i desenvolupar un client capaç de consumir serveis wps.

Diseño y Desarrollo de un cliente Web Processing Service (WPS) para gvSIG

Estudi dels estàndards definits per l'Open Geospatial Consortium, i més concretament en l'estàndard Web Processing Service (wps). Així mateix, ha tingut una component pràctica que ha consistit en el disseny i desenvolupament d'un client capaç de consumir serveis Web creats segons wps i integrat a la plataforma gvSIG.

Fracture imaging within a granitic rock aquifer using multiple-offset single-hole and cross-hole GPR reflection data

The sparsely spaced highly permeable fractures of the granitic rock aquifer at Stang-er-Brune (Brittany, France) form a well-connected fracture network of high permeability but unknown geometry. Previous work based on optical and acoustic logging together with single-hole and cross-hole flowmeter data acquired in 3 neighbouring boreholes (70-100 m deep) has identified the most important permeable fractures crossing the boreholes and their hydraulic connections. To constrain possible flow paths by estimating the geometries of known and previously unknown fractures, we have acquired, processed and interpreted multifold, single- and cross-hole GPR data using 100 and 250 MHz antennas. The GPR data processing scheme consisting of timezero corrections, scaling, bandpass filtering and F-X deconvolution, eigenvector filtering, muting, pre-stack Kirchhoff depth migration and stacking was used to differentiate fluid-filled fracture reflections from source generated noise. The final stacked and pre-stack depth-migrated GPR sections provide high-resolution images of individual fractures (dipping 30-90°) in the surroundings (2-20 m for the 100 MHz antennas; 2-12 m for the 250 MHz antennas) of each borehole in a 2D plane projection that are of superior quality to those obtained from single-offset sections. Most fractures previously identified from hydraulic testing can be correlated to reflections in the single-hole data. Several previously unknown major near vertical fractures have also been identified away from the boreholes.

Probabilistic base calling of Solexa sequencing data.

BACKGROUND: Solexa/Illumina short-read ultra-high throughput DNA sequencing technology produces millions of short tags (up to 36 bases) by parallel sequencing-by-synthesis of DNA colonies. The processing and statistical analysis of such high-throughput data poses new challenges; currently a fair proportion of the tags are routinely discarded due to an inability to match them to a reference sequence, thereby reducing the effective throughput of the technology. RESULTS: We propose a novel base calling algorithm using model-based clustering and probability theory to identify ambiguous bases and code them with IUPAC symbols. We also select optimal sub-tags using a score based on information content to remove uncertain bases towards the ends of the reads. CONCLUSION: We show that the method improves genome coverage and number of usable tags as compared with Solexa's data processing pipeline by an average of 15%. An R package is provided which allows fast and accurate base calling of Solexa's fluorescence intensity files and the production of informative diagnostic plots.

Seismic attenuation and velocity dispersion in heterogeneous partially saturated porous rocks

Using a numerical approach, we explore wave-induced fluid flow effects in partially saturated porous rocks in which the gas-water saturation patterns are governed by mesoscopic heterogeneities associated with the dry frame properties. The link between the dry frame properties and the gas saturation is defined by the assumption of capillary pressure equilibrium, which in the presence of heterogeneity implies that neighbouring regions can exhibit different levels of saturation. To determine the equivalent attenuation and phase velocity of the synthetic rock samples considered in this study, we apply a numerical upscaling procedure, which permits to take into account mesoscopic heterogeneities associated with the dry frame properties as well as spatially continuous variations of the pore fluid properties. The multiscale nature of the fluid saturation is taken into account by locally computing the physical properties of an effective fluid, which are then used for the larger-scale simulations. We consider two sets of numerical experiments to analyse such effects in heterogeneous partially saturated porous media, where the saturation field is determined by variations in porosity and clay content, respectively. In both cases we also evaluate the seismic responses of corresponding binary, patchy-type saturation patterns. Our results indicate that significant attenuation and modest velocity dispersion effects take place in this kind of media for both binary patchy-type and spatially continuous gas saturation patterns and in particular in the presence of relatively small amounts of gas. The numerical experiments also show that the nature of the gas distribution patterns is a critical parameter controlling the seismic responses of these environments, since attenuation and velocity dispersion effects are much more significant and occur over a broader saturation range for binary patchy-type gas-water distributions. This analysis therefore suggests that the physical mechanisms governing partial saturation should be accounted for when analysing seismic data in a poroelastic framework. In this context, heterogeneities associated with the dry frame properties, which do not play important roles in wave-induced fluid flow processes per se, should be taken into account since they may determine the kind of gas distribution pattern taking place in the porous rock.

High-resolution seismic images of potentially seismogenic structures beneath the northwest Canterbury Plains, New Zealand

The transpressional boundary between the Australian and Pacific plates in the central South Island of New Zealand comprises the Alpine Fault and a broad region of distributed strain concentrated in the Southern Alps but encompassing regions further to the east, including the northwest Canterbury Plains. Low to moderate levels of seismicity (e. g., 2 > M 5 events since 1974 and 2 > M 4.0 in 2009) and Holocene sediments offset or disrupted along rare exposed active fault segments are evidence for ongoing tectonism in the northwest plains, the surface topography of which is remarkably flat and even. Because the geology underlying the late Quaternary alluvial fan deposits that carpet most of the plains is not established, the detailed tectonic evolution of this region and the potential for larger earthquakes is only poorly understood. To address these issues, we have processed and interpreted high-resolution (2.5 m subsurface sampling interval) seismic data acquired along lines strategically located relative to extensive rock exposures to the north, west, and southwest and rare exposures to the east. Geological information provided by these rock exposures offer important constraints on the interpretation of the seismic data. The processed seismic reflection sections image a variably thick layer of generally undisturbed younger (i.e., < 24 ka) Quaternary alluvial sediments unconformably overlying an older (> 59 ka) Quaternary sedimentary sequence that shows evidence of moderate faulting and folding during and subsequent to deposition. These Quaternary units are in unconformable contact with Late Cretaceous-Tertiary interbedded sedimentary and volcanic rocks that are highly faulted, folded, and tilted. The lowest imaged unit is largely reflection-free Permian Triassic basement rocks. Quaternary-age deformation has affected all the rocks underlying the younger alluvial sediments, and there is evidence for ongoing deformation. Eight primary and numerous secondary faults as well as a major anticlinal fold are revealed on the seismic sections. Folded sedimentary and volcanic units are observed in the hanging walls and footwalls of most faults. Five of the primary faults represent plausible extensions of mapped faults, three of which are active. The major anticlinal fold is the probable continuation of known active structure. A magnitude 7.1 earthquake occurred on 4 September 2010 near the southeastern edge of our study area. This predominantly right-lateral strike-slip event and numerous aftershocks (ten with magnitudes >= 5 within one week of the main event) highlight the primary message of our paper: that the generally flat and topographically featureless Canterbury Plains is underlain by a network of active faults that have the potential to generate significant earthquakes.

Seismic attenuation in heterogeneous partially saturated rocks

We study wave-induced fluid flow effects in porous rocks partially saturated with gas and water, where the saturation patterns are governed by mesoscopic heterogeneities associated with the dry frame properties. The link between the dry frame properties and the gas saturation is defined by the assumption of capillary pressure equilibrium, which in the presence of heterogeneity implies that neighboring regions can exhibit different levels of saturation. In order to determine the equivalent attenuation and phase velocity of the synthetic rock samples considered in this study, we apply a numerical upscaling procedure, which permits to take into account mesoscopic heterogeneities associated with the dry frame properties as well as spatially continuous variations of the pore fluid properties. We consider numerical experiments to analyze such effects in heterogeneous partially saturated porous media, where the saturation field is determined by realistic variations in porosity. Our results indicate that the spatially continuous nature of gas saturation inherent to this study is a critical parameter controlling the seismic response of these environments, which in turn suggests that the physical mechanisms governing partial saturation should be accounted for when analyzing seismic data in a poro-elastic context.