HPC management and engineering in the hybrid cloud

The evolution and maturation of Cloud Computing created an opportunity for the emergence of new Cloud applications. High-performance Computing, a complex problem solving class, arises as a new business consumer by taking advantage of the Cloud premises and leaving the expensive datacenter management and difficult grid development. Standing on an advanced maturing phase, today’s Cloud discarded many of its drawbacks, becoming more and more efficient and widespread. Performance enhancements, prices drops due to massification and customizable services on demand triggered an emphasized attention from other markets. HPC, regardless of being a very well established field, traditionally has a narrow frontier concerning its deployment and runs on dedicated datacenters or large grid computing. The problem with common placement is mainly the initial cost and the inability to fully use resources which not all research labs can afford. The main objective of this work was to investigate new technical solutions to allow the deployment of HPC applications on the Cloud, with particular emphasis on the private on-premise resources – the lower end of the chain which reduces costs. The work includes many experiments and analysis to identify obstacles and technology limitations. The feasibility of the objective was tested with new modeling, architecture and several applications migration. The final application integrates a simplified incorporation of both public and private Cloud resources, as well as HPC applications scheduling, deployment and management. It uses a well-defined user role strategy, based on federated authentication and a seamless procedure to daily usage with balanced low cost and performance.

Data analytics in the cloud with flexible mapreduced workflows

Data analytic applications are characterized by large data sets that are subject to a series of processing phases. Some of these phases are executed sequentially but others can be executed concurrently or in parallel on clusters, grids or clouds. The MapReduce programming model has been applied to process large data sets in cluster and cloud environments. For developing an application using MapReduce there is a need to install/configure/access specific frameworks such as Apache Hadoop or Elastic MapReduce in Amazon Cloud. It would be desirable to provide more flexibility in adjusting such configurations according to the application characteristics. Furthermore the composition of the multiple phases of a data analytic application requires the specification of all the phases and their orchestration. The original MapReduce model and environment lacks flexible support for such configuration and composition. Recognizing that scientific workflows have been successfully applied to modeling complex applications, this paper describes our experiments on implementing MapReduce as subworkflows in the AWARD framework (Autonomic Workflow Activities Reconfigurable and Dynamic). A text mining data analytic application is modeled as a complex workflow with multiple phases, where individual workflow nodes support MapReduce computations. As in typical MapReduce environments, the end user only needs to define the application algorithms for input data processing and for the map and reduce functions. In the paper we present experimental results when using the AWARD framework to execute MapReduce workflows deployed over multiple Amazon EC2 (Elastic Compute Cloud) instances.

Towards high-available and energy-efficient virtual computing environments in the cloud

Empowered by virtualisation technology, cloud infrastructures enable the construction of flexi- ble and elastic computing environments, providing an opportunity for energy and resource cost optimisation while enhancing system availability and achieving high performance. A crucial re- quirement for effective consolidation is the ability to efficiently utilise system resources for high- availability computing and energy-efficiency optimisation to reduce operational costs and carbon footprints in the environment. Additionally, failures in highly networked computing systems can negatively impact system performance substantially, prohibiting the system from achieving its initial objectives. In this paper, we propose algorithms to dynamically construct and readjust vir- tual clusters to enable the execution of users’ jobs. Allied with an energy optimising mechanism to detect and mitigate energy inefficiencies, our decision-making algorithms leverage virtuali- sation tools to provide proactive fault-tolerance and energy-efficiency to virtual clusters. We conducted simulations by injecting random synthetic jobs and jobs using the latest version of the Google cloud tracelogs. The results indicate that our strategy improves the work per Joule ratio by approximately 12.9% and the working efficiency by almost 15.9% compared with other state-of-the-art algorithms.

Phylogenetic Code in the Cloud - Can it Meet the Expectations?

Cloud computing has recently become very popular, and several bioinformatics applications exist already in that domain. The aim of this article is to analyse a current cloud system with respect to usability, benchmark its performance and compare its user friendliness with a conventional cluster job submission system. Given the current hype on the theme, user expectations are rather high, but current results show that neither the price/performance ratio nor the usage model is very satisfactory for large-scale embarrassingly parallel applications. However, for small to medium scale applications that require CPU time at certain peak times the cloud is a suitable alternative.

Research Data Management in the Cloud

Workshop at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Big Data Processing in the Cloud: a Hydra/Sufia Experience

Presentation at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

The cumulus-capped boundary layer. I: Modelling transports in the cloud layer

Scalar-flux budgets have been obtained from large-eddy simulations (LESs) of the cumulus-capped boundary layer. Parametrizations of the terms in the budgets are discussed, and two parametrizations for the transport term in the cloud layer are proposed. It is shown that these lead to two models for scalar transports by shallow cumulus convection. One is equivalent to the subsidence detrainment form of convective tendencies obtained from mass-flux parametrizations of cumulus convection. The second is a flux-gradient relationship that is similar in form to the non-local parametrizations of turbulent transports in the dry-convective boundary layer. Using the fluxes of liquid-water potential temperature and total water content from the LES, it is shown that both models are reasonable diagnostic relations between fluxes and the vertical gradients of the mean fields. The LESs used in this study are for steady-state convection and it is possible to treat the fluxes of conserved thermodynamic variables as independent, and ignore the effects of condensation. It is argued that a parametrization of cumulus transports in a model of the cumulus-capped boundary layer should also include an explicit representation of condensation. A simple parametrization of the liquid-water flux in terms of conserved variables is also derived.

GIS in the cloud: implementing a web map service on Google App Engine

Many producers of geographic information are now disseminating their data using open web service protocols, notably those published by the Open Geospatial Consortium. There are many challenges inherent in running robust and reliable services at reasonable cost. Cloud computing provides a new kind of scalable infrastructure that could address many of these challenges. In this study we implement a Web Map Service for raster imagery within the Google App Engine environment. We discuss the challenges of developing GIS applications within this framework and the performance characteristics of the implementation. Results show that the application scales well to multiple simultaneous users and performance will be adequate for many applications, although concerns remain over issues such as latency spikes. We discuss the feasibility of implementing services within the free usage quotas of Google App Engine and the possibility of extending the approaches in this paper to other GIS applications.

Quantifying sources of inter-model diversity in the cloud albedo effect

There is large diversity in simulated aerosol forcing among models that participated in the fifth Coupled Model Intercomparison Project (CMIP5), particularly related to aerosol interactions with clouds. Here we use the reported model data and fitted aerosol-cloud relations to separate the main sources of inter-model diversity in the magnitude of the cloud albedo effect. There is large diversity in the global load and spatial distribution of sulfate aerosol, as well as in global-mean cloud-top effective radius. The use of different parameterizations of aerosol-cloud interactions makes the largest contribution to diversity in modeled radiative forcing (up to -39%, +48% about the mean estimate). Uncertainty in pre-industrial sulfate load also makes a substantial contribution (-15%, +61% about the mean estimate), with smaller contributions from inter-model differences in the historical change in sulfate load and in mean cloud fraction.

Coordination as a Service (CaaS) in the Cloud

Nel primo capitolo si è studiata la nuova tecnologia del Cloud Computing, fornendo una semplice analisi di tutte le caratteristiche principali, gli attori coinvolti e i relativi metodi di distribuzione e servizi offerti. Nel secondo capitolo si è introdotta la nozione di coordination as a service, discutendone le relative astrazioni che compongono l'architettura logica. Successivamente si è preso in considerazione il modello di coordinazione TuCSoN definendo cosa si intende per nodo, agente, centro di tuple e agent coordination context ed è stato analizzato il relativo linguaggio di coordinazione attraverso il quale essi interagiscono. Nel terzo capitolo sono state riviste ed estese le nozioni di TuCSoN, precedentemente acquisite, nell'ambito del Cloud Computing ed è stato fornito un modello astratto ed una possibile architettura di TuCSoN in the Cloud. Sono stati analizzati anche gli aspetti di un possibile servizio di tale genere nello scenario di servizio pay-per-use. Infine nel quarto ed ultimo capitolo si è sviluppato un caso di studio in cui si è implementata un'interfaccia per l'attuale CLI di TuCSoN sottoforma di applet, che è stata poi inserita nel Cloud attraverso la piattaforma PaaS Cloudify.

Automatic Deployment of Applications in the Cloud

In distributed systems like clouds or service oriented frameworks, applications are typically assembled by deploying and connecting a large number of heterogeneous software components, spanning from fine-grained packages to coarse-grained complex services. The complexity of such systems requires a rich set of techniques and tools to support the automation of their deployment process. By relying on a formal model of components, a technique is devised for computing the sequence of actions allowing the deployment of a desired configuration. An efficient algorithm, working in polynomial time, is described and proven to be sound and complete. Finally, a prototype tool implementing the proposed algorithm has been developed. Experimental results support the adoption of this novel approach in real life scenarios.

A Case for CDN-as-a-Service in the Cloud: A Mobile Cloud Networking Argument

Content Distribution Networks are mandatory components of modern web architectures, with plenty of vendors offering their services. Despite its maturity, new paradigms and architecture models are still being developed in this area. Cloud Computing, on the other hand, is a more recent concept which has expanded extremely quickly, with new services being regularly added to cloud management software suites such as OpenStack. The main contribution of this paper is the architecture and the development of an open source CDN that can be provisioned in an on-demand, pay-as-you-go model thereby enabling the CDN as a Service paradigm. We describe our experience with integration of CDNaaS framework in a cloud environment, as a service for enterprise users. We emphasize the flexibility and elasticity of such a model, with each CDN instance being delivered on-demand and associated to personalized caching policies as well as an optimized choice of Points of Presence based on exact requirements of an enterprise customer. Our development is based on the framework developed in the Mobile Cloud Networking EU FP7 project, which offers its enterprise users a common framework to instantiate and control services. CDNaaS is one of the core support components in this project as is tasked to deliver different type of multimedia content to several thousands of users geographically distributed. It integrates seamlessly in the MCN service life-cycle and as such enjoys all benefits of a common design environment, allowing for an improved interoperability with the rest of the services within the MCN ecosystem.

Software Development in the Cloud: Exploring the Affordances of Platform-as-a-Service

Software development teams increasingly adopt platform-as-a-service (PaaS), i.e., cloud services that make software development infrastructure available over the internet. Yet, empirical evidence of whether and how software development work changes with the use of PaaS is difficult to find. We performed a grounded-theory study to explore the affordances of PaaS for software development teams. We find that PaaS enables software development teams to enforce uniformity, to exploit knowledge embedded in technology, to enhance agility, and to enrich jobs. These affordances do not arise in a vacuum. Their emergence is closely interwoven with changes in methodologies, roles, and norms that give rise to self-organizing, loosely coupled teams. Our study provides rich descriptions of PaaS-based software development and an emerging theory of affordances of PaaS for software development teams.