Design and Implementation of a Cloud Infrastructure for Distributed Scientific Calculation

Cloud computing enables independent end users and applications to share data and pooled resources, possibly located in geographically distributed Data Centers, in a fully transparent way. This need is particularly felt by scientific applications to exploit distributed resources in efficient and scalable way for the processing of big amount of data. This paper proposes an open so- lution to deploy a Platform as a service (PaaS) over a set of multi- site data centers by applying open source virtualization tools to facilitate operation among virtual machines while optimizing the usage of distributed resources. An experimental testbed is set up in Openstack environment to obtain evaluations with different types of TCP sample connections to demonstrate the functionality of the proposed solution and to obtain throughput measurements in relation to relevant design parameters.

Using high resolution tracer data to constrain water storage, flux and age estimates in a spatially distributed rainfall-runoff model

Acknowledgements The authors would like to thank Jonathan Dick, Josie Geris, Jason Lessels, and Claire Tunaley for data collection and Audrey Innes for lab sample preparation. We also thank Christian Birkel for discussions about the model structure and comments on an earlier draft of the paper. Climatic data were provided by Iain Malcolm and Marine Scotland Fisheries at the Freshwater Lab, Pitlochry. Additional precipitation data were provided by the UK Meteorological Office and the British Atmospheric Data Centre (BADC).We thank the European Research Council ERC (project GA 335910 VEWA) for funding the VeWa project.

DiCER: A distributed consumer experience research method for use in public spaces

Date of Acceptance: 06/03/2015 Acknowledgement: This research was funded by NCR and the Northern Research Partnership

Distributed Power balance and damping control for high power multiport LCL DC hub

This project is funded by European Research Council in FP7; grant no 259328, 2010 and EPSRC grant no EP/K006428/1, 2013.

DiCER: A distributed consumer experience research method for use in public spaces

Date of Acceptance: 06/03/2015 Acknowledgement: This research was funded by NCR and the Northern Research Partnership

Distributed Power balance and damping control for high power multiport LCL DC hub

This project is funded by European Research Council in FP7; grant no 259328, 2010 and EPSRC grant no EP/K006428/1, 2013.

Distributed Learning: An Agent-Based Approach to Data-Mining

Peer reviewed

DiCER: A distributed consumer experience research method for use in public spaces

Date of Acceptance: 06/03/2015 Acknowledgement: This research was funded by NCR and the Northern Research Partnership

Performance improvement of broadband distributed Raman amplifier using bidirectional pumping with first and dual order forward pumps

In this paper, a new bidirectional pumping scheme with dual order forward pumps is proposed. Performance is compared numerically with conventional bidirectional and backward only pumping schemes for a 70 nm bandwidth, 61.5 km distributed Raman amplifier. We demonstrate that it is possible to design a flat gain spectrum with improved noise figure and OSNR, as well as a low gain ripple (<1 dB).

CW lasing in a telecom fibre due to the random distributed feedback via Rayleigh scattering

We demonstrate a fibre laser with a mirrorless cavity that operates via Rayleigh scattering amplified through the Raman effect. The properties of such random distributed feedback laser appear different from those of both traditional random lasers and conventional fibre lasers. ©2010 IEEE.

Relative intensity noise transfer in higher-order distributed amplification through ultra-long fiber cavities

Among the different possible amplification solutions offered by Raman scattering in optical fibers, ultra-long Raman lasers are particularly promising as they can provide quasi-losless second order amplification with reduced complexity, displaying excellent potential in the design of low-noise long-distance communication systems. Still, some of their advantages can be partially offset by the transfer of relative intensity noise from the pump sources and cavity-generated Stokes to the transmitted signal. In this paper we study the effect of ultra-long cavity design (length, pumping, grating reflectivity) on the transfer of RIN to the signal, demonstrating how the impact of noise can be greatly reduced by carefully choosing appropriate cavity parameters depending on the intended application of the system. © 2010 Copyright SPIE - The International Society for Optical Engineering.

High spatial and temporal resolution interrogation of fully distributed chirped fiber Bragg grating sensors

A novel interrogation technique for fully distributed linearly chirped fiber Bragg grating (LCFBG) strain sensors with simultaneous high temporal and spatial resolution based on optical time-stretch frequency-domain reflectometry (OTS-FDR) is proposed and experimentally demonstrated. LCFBGs is a promising candidate for fully distributed sensors thanks to its longer grating length and broader reflection bandwidth compared to normal uniform FBGs. In the proposed system, two identical LCFBGs are employed in a Michelson interferometer setup with one grating serving as the reference grating whereas the other serving as the sensing element. Broadband spectral interferogram is formed and the strain information is encoded into the wavelength-dependent free spectral range (FSR). Ultrafast interrogation is achieved based on dispersion-induced time stretch such that the target spectral interferogram is mapped to a temporal interference waveform that can be captured in real-Time using a single-pixel photodector. The distributed strain along the sensing grating can be reconstructed from the instantaneous RF frequency of the captured waveform. High-spatial resolution is also obtained due to high-speed data acquisition. In a proof-of-concept experiment, ultrafast real-Time interrogation of fully-distributed grating sensors with various strain distributions is experimentally demonstrated. An ultrarapid measurement speed of 50 MHz with a high spatial resolution of 31.5 μm over a gauge length of 25 mm and a strain resolution of 9.1 μϵ have been achieved.

Studying Recommender Systems to Enhance Distributed Computing Schedulers

Distributed Computing frameworks belong to a class of programming models that allow developers to

launch workloads on large clusters of machines. Due to the dramatic increase in the volume of

data gathered by ubiquitous computing devices, data analytic workloads have become a common

case among distributed computing applications, making Data Science an entire field of

Computer Science. We argue that Data Scientist's concern lays in three main components: a dataset,

a sequence of operations they wish to apply on this dataset, and some constraint they may have

related to their work (performances, QoS, budget, etc). However, it is actually extremely

difficult, without domain expertise, to perform data science. One need to select the right amount

and type of resources, pick up a framework, and configure it. Also, users are often running their

application in shared environments, ruled by schedulers expecting them to specify precisely their resource

needs. Inherent to the distributed and concurrent nature of the cited frameworks, monitoring and

profiling are hard, high dimensional problems that block users from making the right

configuration choices and determining the right amount of resources they need. Paradoxically, the

system is gathering a large amount of monitoring data at runtime, which remains unused.

In the ideal abstraction we envision for data scientists, the system is adaptive, able to exploit

monitoring data to learn about workloads, and process user requests into a tailored execution

context. In this work, we study different techniques that have been used to make steps toward

such system awareness, and explore a new way to do so by implementing machine learning

techniques to recommend a specific subset of system configurations for Apache Spark applications.

Furthermore, we present an in depth study of Apache Spark executors configuration, which highlight

the complexity in choosing the best one for a given workload.

Distributed Feature Selection in Large n and Large p Regression Problems

Fitting statistical models is computationally challenging when the sample size or the dimension of the dataset is huge. An attractive approach for down-scaling the problem size is to first partition the dataset into subsets and then fit using distributed algorithms. The dataset can be partitioned either horizontally (in the sample space) or vertically (in the feature space), and the challenge arise in defining an algorithm with low communication, theoretical guarantees and excellent practical performance in general settings. For sample space partitioning, I propose a MEdian Selection Subset AGgregation Estimator ({\em message}) algorithm for solving these issues. The algorithm applies feature selection in parallel for each subset using regularized regression or Bayesian variable selection method, calculates the `median' feature inclusion index, estimates coefficients for the selected features in parallel for each subset, and then averages these estimates. The algorithm is simple, involves very minimal communication, scales efficiently in sample size, and has theoretical guarantees. I provide extensive experiments to show excellent performance in feature selection, estimation, prediction, and computation time relative to usual competitors.

While sample space partitioning is useful in handling datasets with large sample size, feature space partitioning is more effective when the data dimension is high. Existing methods for partitioning features, however, are either vulnerable to high correlations or inefficient in reducing the model dimension. In the thesis, I propose a new embarrassingly parallel framework named {\em DECO} for distributed variable selection and parameter estimation. In {\em DECO}, variables are first partitioned and allocated to m distributed workers. The decorrelated subset data within each worker are then fitted via any algorithm designed for high-dimensional problems. We show that by incorporating the decorrelation step, DECO can achieve consistent variable selection and parameter estimation on each subset with (almost) no assumptions. In addition, the convergence rate is nearly minimax optimal for both sparse and weakly sparse models and does NOT depend on the partition number m. Extensive numerical experiments are provided to illustrate the performance of the new framework.

For datasets with both large sample sizes and high dimensionality, I propose a new "divided-and-conquer" framework {\em DEME} (DECO-message) by leveraging both the {\em DECO} and the {\em message} algorithm. The new framework first partitions the dataset in the sample space into row cubes using {\em message} and then partition the feature space of the cubes using {\em DECO}. This procedure is equivalent to partitioning the original data matrix into multiple small blocks, each with a feasible size that can be stored and fitted in a computer in parallel. The results are then synthezied via the {\em DECO} and {\em message} algorithm in a reverse order to produce the final output. The whole framework is extremely scalable.

Reduced resilience of a globally distributed coccolithophore to ocean acidification: Confirmed up to 2000 generations

Ocean acidification (OA), induced by rapid anthropogenic CO2 rise and its dissolution in seawater, is known to have consequences for marine organisms. However, knowledge on the evolutionary responses of phytoplankton to OA has been poorly studied. Here we examined the coccolithophore Gephyrocapsa oceanica, while growing it for 2000 generations under ambient and elevated CO2 levels. While OA stimulated growth in the earlier selection period (from generations 700 to 1550), it reduced it in the later selection period up to 2000 generations. Similarly, stimulated production of particulate organic carbon and nitrogen reduced with increasing selection period and decreased under OA up to 2000 generations. The specific adaptation of growth to OA disappeared in generations 1700 to 2000 when compared with that at 1000 generations. Both phenotypic plasticity and fitness decreased within selection time, suggesting that the species' resilience to OA decreased after 2000 generations under high CO2 selection.