Elastic computing on Cloud resources for the CMS experiment

Nowadays, data handling and data analysis in High Energy Physics requires a vast amount of computational power and storage. In particular, the world-wide LHC Com- puting Grid (LCG), an infrastructure and pool of services developed and deployed by a ample community of physicists and computer scientists, has demonstrated to be a game changer in the efficiency of data analyses during Run-I at the LHC, playing a crucial role in the Higgs boson discovery. Recently, the Cloud computing paradigm is emerging and reaching a considerable adoption level by many different scientific organizations and not only. Cloud allows to access and utilize not-owned large computing resources shared among many scientific communities. Considering the challenging requirements of LHC physics in Run-II and beyond, the LHC computing community is interested in exploring Clouds and see whether they can provide a complementary approach - or even a valid alternative - to the existing technological solutions based on Grid. In the LHC community, several experiments have been adopting Cloud approaches, and in particular the experience of the CMS experiment is of relevance to this thesis. The LHC Run-II has just started, and Cloud-based solutions are already in production for CMS. However, other approaches of Cloud usage are being thought of and are at the prototype level, as the work done in this thesis. This effort is of paramount importance to be able to equip CMS with the capability to elastically and flexibly access and utilize the computing resources needed to face the challenges of Run-III and Run-IV. The main purpose of this thesis is to present forefront Cloud approaches that allow the CMS experiment to extend to on-demand resources dynamically allocated as needed. Moreover, a direct access to Cloud resources is presented as suitable use case to face up with the CMS experiment needs. Chapter 1 presents an overview of High Energy Physics at the LHC and of the CMS experience in Run-I, as well as preparation for Run-II. Chapter 2 describes the current CMS Computing Model, and Chapter 3 provides Cloud approaches pursued and used within the CMS Collaboration. Chapter 4 and Chapter 5 discuss the original and forefront work done in this thesis to develop and test working prototypes of elastic extensions of CMS computing resources on Clouds, and HEP Computing “as a Service”. The impact of such work on a benchmark CMS physics use-cases is also demonstrated.

Serverless Computing il caso Google Cloud Functions

Il serverless é un paradigma del cloud computing al giorno d’oggi sempre più diffuso; si basa sulla scrittura di funzioni stateless in quanto le attività relative alla loro manutenzione e scalabilità fanno capo al provider dei servizi cloud. Lo sviluppatore deve quindi solamente concentrarsi sulla creazione del prodotto. Questo lavoro si apre con un’analisi del cloud computing introducendo i principali modelli di applicazione, passando dal parlare di servizi cloud, con le varie sottocategorie e i relativi utilizzi per poi arrivare a parlare di serverless. Si é scelto di focalizzarsi sulla piattaforma Google con la suite: Google Cloud Platform. In particolare, si parlerà di Google Cloud Functions, una nuova offerta serverless della compagnia, di recente sviluppo e in continuo aggiornamento. Partiremo dalle prime release, analizzeremo l’ambiente di sviluppo, i casi d’uso, vantaggi, svantaggi, parleremo poi di portabilità e verranno mostrati alcuni esempi del loro utilizzo.

Dynamic power- and failure-aware cloud resources allocation for Sets of independent tasks

Cloud computing is increasingly being adopted in different scenarios, like social networking, business applications, scientific experiments, etc. Relying in virtualization technology, the construction of these computing environments targets improvements in the infrastructure, such as power-efficiency and fulfillment of users’ SLA specifications. The methodology usually applied is packing all the virtual machines on the proper physical servers. However, failure occurrences in these networked computing systems can induce substantial negative impact on system performance, deviating the system from ours initial objectives. In this work, we propose adapted algorithms to dynamically map virtual machines to physical hosts, in order to improve cloud infrastructure power-efficiency, with low impact on users’ required performance. Our decision making algorithms leverage proactive fault-tolerance techniques to deal with systems failures, allied with virtual machine technology to share nodes resources in an accurately and controlled manner. The results indicate that our algorithms perform better targeting power-efficiency and SLA fulfillment, in face of cloud infrastructure failures.

Android as a cloud ticket validator

Trabalho Final de Mestrado para a obtenção do grau de Mestre em Engenharia Informática e de Computadores

Explorador integrado de serviços de armazenamento na cloud: cloud cockpit

Projeto para obtenção do grau de Mestre em Engenharia Informática e de Computadores

Data analytics in the cloud with flexible mapreduced workflows

Data analytic applications are characterized by large data sets that are subject to a series of processing phases. Some of these phases are executed sequentially but others can be executed concurrently or in parallel on clusters, grids or clouds. The MapReduce programming model has been applied to process large data sets in cluster and cloud environments. For developing an application using MapReduce there is a need to install/configure/access specific frameworks such as Apache Hadoop or Elastic MapReduce in Amazon Cloud. It would be desirable to provide more flexibility in adjusting such configurations according to the application characteristics. Furthermore the composition of the multiple phases of a data analytic application requires the specification of all the phases and their orchestration. The original MapReduce model and environment lacks flexible support for such configuration and composition. Recognizing that scientific workflows have been successfully applied to modeling complex applications, this paper describes our experiments on implementing MapReduce as subworkflows in the AWARD framework (Autonomic Workflow Activities Reconfigurable and Dynamic). A text mining data analytic application is modeled as a complex workflow with multiple phases, where individual workflow nodes support MapReduce computations. As in typical MapReduce environments, the end user only needs to define the application algorithms for input data processing and for the map and reduce functions. In the paper we present experimental results when using the AWARD framework to execute MapReduce workflows deployed over multiple Amazon EC2 (Elastic Compute Cloud) instances.

Cloud-assisted Wireless Body Area Networks

Wireless Body Area Networks (WBANs) have emerged as a promising technology for medical and non-medical applications. WBANs consist of a number of miniaturized, portable, and autonomous sensor nodes that are used for long-term health monitoring of patients. These sensor nodes continuously collect information of patients, which are used for ubiquitous health monitoring. In addition, WBANs may be used for managing catastrophic events and increasing the effectiveness and performance of rescue forces. The huge amount of data collected by WBAN nodes demands scalable, on-demand, powerful, and secure storage and processing infrastructure. Cloud computing is expected to play a significant role in achieving the aforementioned objectives. The cloud computing environment links different devices ranging from miniaturized sensor nodes to high-performance supercomputers for delivering people-centric and context-centric services to the individuals and industries. The possible integration of WBANs with cloud computing (WBAN-cloud) will introduce viable and hybrid platform that must be able to process the huge amount of data collected from multiple WBANs. This WBAN-cloud will enable users (including physicians and nurses) to globally access the processing and storage infrastructure at competitive costs. Because WBANs forward useful and life-critical information to the cloud – which may operate in distributed and hostile environments, novel security mechanisms are required to prevent malicious interactions to the storage infrastructure. Both the cloud providers and the users must take strong security measures to protect the storage infrastructure.

Ownserver: sistema doméstico de prestação de serviços cloud

Neste trabalho foi considerada a possibilidade de incorporar serviços remotos, normalmente associados a serviços web e cloud computing, numa solução local que centralizasse os vários serviços num único sistema e permitisse aos seus utilizadores consumir e configurar os mesmos, quer a partir da rede local, quer remotamente a partir da Internet. Desta forma seria possível conciliar o acesso a partir de qualquer local com internet, característico nas clouds, com a simplicidade de concentrar num só sistema vários serviços que são por norma oferecidos por entidades distintas e ainda permitir aos seus utilizadores o controlo e configuração sobre os mesmos. De forma a validar que este conceito é viável, prático e funcional, foram implementadas duas componentes. Um cliente que corre nos dispositivos dos utilizadores e que proporciona a interface para consumir os serviços disponíveis e um servidor que irá conter e prestar esses serviços aos clientes. Estes serviços incluem lista de contactos, mensagens instantâneas, salas de conversação, transferência de ficheiros, chamadas e conferências de voz e vídeo, pastas remotas, pastas sincronizadas, backups, pastas partilhadas, VoD (Video-on Demand) e AoD (Audio-on Demand). Para o desenvolvimento do cliente e do servidor foi utilizada a framework Qt que recorre à linguagem de programação C++ e ao conjunto de bibliotecas que possui, para o desenvolvimento de aplicações multiplataforma. Para as comunicações entre clientes e servidor, foi utilizado o protocolo XMPP (Extensible Messaging and Presence Protocol), pela forma da biblioteca qxmpp e do servidor XMPP ejabberd. Pelo facto de conter um conjunto de centenas de extensões atualmente ativas que auferem funcionalidades como salas de conversação, transferências de ficheiros e até estabelecer sessões multimédia, graças à sua flexibilidade permitiu ainda a criação de extensões personalizada necessárias para algumas funcionalidades que se pretendeu implementar. Foi ainda utilizado no servidor a framework ffmpeg para suportar algumas funcionalidades multimédia. Após a implementação do cliente para Windows e Linux, e de implementar o servidor em Linux foi realizado um conjunto de testes funcionais para perceber se as funcionalidades e seus mecanismos funcionam corretamente. No caso onde a análise da performance e do consumo de recursos era importante, foram realizados testes de performance e testes de carga.

On the cloud deployment of a session abstraction for service/data aggregation

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Personalization platform for multimodal ubiquitous computing applications

Dissertação para obtenção do Grau de Mestre em Engenharia Informática

Communications with QoS in IoT&Cloud environment

Dissertação de mestrado integrado em Engenharia de Telecomunicações e Informática

Phylogenetic Code in the Cloud - Can it Meet the Expectations?

Cloud computing has recently become very popular, and several bioinformatics applications exist already in that domain. The aim of this article is to analyse a current cloud system with respect to usability, benchmark its performance and compare its user friendliness with a conventional cluster job submission system. Given the current hype on the theme, user expectations are rather high, but current results show that neither the price/performance ratio nor the usage model is very satisfactory for large-scale embarrassingly parallel applications. However, for small to medium scale applications that require CPU time at certain peak times the cloud is a suitable alternative.

An efficient use of virtualization in grid/cloud environments

Grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational resources. Grid enables access to the resources but it does not guarantee any quality of service. Moreover, Grid does not provide performance isolation; job of one user can influence the performance of other user’s job. The other problem with Grid is that the users of Grid belong to scientific community and the jobs require specific and customized software environment. Providing the perfect environment to the user is very difficult in Grid for its dispersed and heterogeneous nature. Though, Cloud computing provide full customization and control, but there is no simple procedure available to submit user jobs as in Grid. The Grid computing can provide customized resources and performance to the user using virtualization. A virtual machine can join the Grid as an execution node. The virtual machine can also be submitted as a job with user jobs inside. Where the first method gives quality of service and performance isolation, the second method also provides customization and administration in addition. In this thesis, a solution is proposed to enable virtual machine reuse which will provide performance isolation with customization and administration. The same virtual machine can be used for several jobs. In the proposed solution customized virtual machines join the Grid pool on user request. Proposed solution describes two scenarios to achieve this goal. In first scenario, user submits their customized virtual machine as a job. The virtual machine joins the Grid pool when it is powered on. In the second scenario, user customized virtual machines are preconfigured in the execution system. These virtual machines join the Grid pool on user request. Condor and VMware server is used to deploy and test the scenarios. Condor supports virtual machine jobs. The scenario 1 is deployed using Condor VM universe. The second scenario uses VMware-VIX API for scripting powering on and powering off of the remote virtual machines. The experimental results shows that as scenario 2 does not need to transfer the virtual machine image, the virtual machine image becomes live on pool more faster. In scenario 1, the virtual machine runs as a condor job, so it easy to administrate the virtual machine. The only pitfall in scenario 1 is the network traffic.

Designing business models of cloud platforms

Cloud computing and its three facets (Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS)) are terms that denote new developments in the software industry. In particular, PaaS solutions, also referred to as cloud platforms, are changing the way software is being produced, distributed, consumed, and priced. Software vendors have started considering cloud platforms as a strategic option but are battling to redefine their offerings to embrace PaaS. In contrast to SaaS and IaaS, PaaS allows for value co-creation with partners to develop complementary components and applications. It thus requires multisided business models that bring together two or more distinct customer segments. Understanding how to design PaaS business models to establish a flourishing ecosystem is crucial for software vendors. This doctoral thesis aims to address this issue in three interrelated research parts. First, based on case study research, the thesis provides a deeper understanding of current PaaS business models and their evolution. Second, it analyses and simulates consumers' preferences regarding PaaS business models, using a conjoint approach to find out what determines the choice of cloud platforms. Finally, building on the previous research outcomes, the third part introduces a design theory for the emerging class of PaaS business models, which is grounded on an extensive action design research study with a large European software vendor. Understanding PaaS business models from a market as well as a consumer perspective will, together with the design theory, inform and guide decision makers in their business model innovation plans. It also closes gaps in the research related to PaaS business model design and more generally related to platform business models.

Estudi de cloud cumputing i desenvolupament d'un gestor de currículums en eyeOS

El Cloud Computing o computació en el núvol és un conjunt de tecnologies que busca tenir tots els nostres arxius i informació a Internet sense dependre de tenir la capacitat suficient per emmagatzemar informació, tan sols d'oferir-nos el servei que volem en el moment que necessitem. En aquest projecte es detalla l'estudi de les diferents solucions cloud a nivell personal necessàries pel nostre dia a dia amb l'objectiu d'obtenir una eina que ens permeti posteriorment desenvolupar una petita aplicació per a gestionar els nostres currículums vitae.