SmART: an application reconfiguration framework

Dissertação apresentada na Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa para a obtenção do Grau de Mestre em Engenharia Informática.

Estimating Effective Slowdown of Tasks in Energy-Aware Clouds

Consolidation consists in scheduling multiple virtual machines onto fewer servers in order to improve resource utilization and to reduce operational costs due to power consumption. However, virtualization technologies do not offer performance isolation, causing applications’ slowdown. In this work, we propose a performance enforcing mechanism, composed of a slowdown estimator, and a interference- and power-aware scheduling algorithm. The slowdown estimator determines, based on noisy slowdown data samples obtained from state-of-the-art slowdown meters, if tasks will complete within their deadlines, invoking the scheduling algorithm if needed. When invoked, the scheduling algorithm builds performance and power aware virtual clusters to successfully execute the tasks. We conduct simulations injecting synthetic jobs which characteristics follow the last version of the Google Cloud tracelogs. The results indicate that our strategy can be efficiently integrated with state-of-the-art slowdown meters to fulfil contracted SLAs in real-world environments, while reducing operational costs in about 12%.

PIM place & ghost application

Actualment, el Servei d'Informàtica de l'Escola d'Enginyeries (SIEE) s'enfronta a dos problemes: l'augment del nombre d'alumnes, mantenint el mateix número d'ordinadors per fer les pràctiques i, d'altra banda, el també creixent nombre d'aplicacions que s'han desenvolupat i es desenvolupen per resoldre les necessitats generades pels mateixos alumnes. Aquest projecte neix amb la voluntat de solucionar aquests problemes, creant per un costat un aula de màquines virtuals i per altra banda crear un aplicatiu web, que servirà de framework i contenidor de futures aplicacions, on es pugui connectar de manera senzilla amb les màquines virtuals.

Virtualització dels serveis d'una empresa

La virtualització és una tecnologia que permet l’abstracció del programari respecte dels recursos del maquinari, tant d’aplicacions com del sistema operatiu. Aquesta tecnologia està constantment en evolució, i el que fa pocs anys era un entorn de treball exclusiu per test de laboratori i aplicacions molt especifiques, ara permet oferir aplicacions i serveis d'entorns empresarials, permetent aprofitar el màxim els recursos disponibles. Aquest projecte tracta de plantejar la virtualització com a solució per a maximitzar el rendiments dels equips informàtics de una empresa. Introduirem la tecnologia de virtualització, planificarem la conversió dels serveis de l’empresa en màquines virtuals, detallarem els aspectes tècnics de la implementació i mostrarem els resultats d’aquesta virtualització.

Multi-level SLA-driven resource management on Cloud Providers

Nowadays, service providers in the Cloud offer complex services ready to be used as it was a commodity like water or electricity to their customers with any other extra effort for them. However, providing these services implies a high management effort which requires a lot of human interaction. Furthermore, an efficient resource management mechanism considering only provider's resources is, though necessary, not enough, because the provider's profit is limited by the amount of resources it owns. Dynamically outsourcing resources to other providers in response to demand variation avoids this problem and makes the provider to get more profit. A key technology for achieving these goals is virtualization which facilitates provider's management and provides on-demand virtual environments, which are isolated and consolidated in order to achieve a better utilization of the provider's resources. Nevertheless, dealing with some virtualization capabilities implies an effort for the user in order to take benefit from them. In order to avoid this problem, we are contributing the research community with a virtualized environment manager which aims to provide virtual machines that fulfils with the user requirements. Another challenge is sharing resources among different federated Cloud providers while exploiting the features of virtualization in a new approach for facilitating providers' management. This project aims for reducing provider's costs and at the same time fulfilling the quality of service agreed with the customers while maximizing the provider's revenue. It considers resource management at several layers, namely locally to each node in the provider, among different nodes in the provider, and among different federated providers. This latter layer supports the novel capabilities of outsourcing when the local resources are not enough to fulfil the users demand, and offering resources to other providers when the local resources are underused.

An efficient use of virtualization in grid/cloud environments

Grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational resources. Grid enables access to the resources but it does not guarantee any quality of service. Moreover, Grid does not provide performance isolation; job of one user can influence the performance of other user’s job. The other problem with Grid is that the users of Grid belong to scientific community and the jobs require specific and customized software environment. Providing the perfect environment to the user is very difficult in Grid for its dispersed and heterogeneous nature. Though, Cloud computing provide full customization and control, but there is no simple procedure available to submit user jobs as in Grid. The Grid computing can provide customized resources and performance to the user using virtualization. A virtual machine can join the Grid as an execution node. The virtual machine can also be submitted as a job with user jobs inside. Where the first method gives quality of service and performance isolation, the second method also provides customization and administration in addition. In this thesis, a solution is proposed to enable virtual machine reuse which will provide performance isolation with customization and administration. The same virtual machine can be used for several jobs. In the proposed solution customized virtual machines join the Grid pool on user request. Proposed solution describes two scenarios to achieve this goal. In first scenario, user submits their customized virtual machine as a job. The virtual machine joins the Grid pool when it is powered on. In the second scenario, user customized virtual machines are preconfigured in the execution system. These virtual machines join the Grid pool on user request. Condor and VMware server is used to deploy and test the scenarios. Condor supports virtual machine jobs. The scenario 1 is deployed using Condor VM universe. The second scenario uses VMware-VIX API for scripting powering on and powering off of the remote virtual machines. The experimental results shows that as scenario 2 does not need to transfer the virtual machine image, the virtual machine image becomes live on pool more faster. In scenario 1, the virtual machine runs as a condor job, so it easy to administrate the virtual machine. The only pitfall in scenario 1 is the network traffic.

Estudi comparatiu de la gestió de memòria de dues màquines virtuals

En aquest projecte s'han comparat els gestors de memòria de les màquines virtuals de Java i del projecte Mono.

Disseny i implantació de l'estructura informàtica de DaVinci School

Creació d'una infraestructura informàtica d'un centre educatiu, amb xarxes separades per a aules, secretaria i seminaris. Serveis centralitzats en un servidor amb màquines virtuals.

Server Virtualization

Virtualisoinnin ideana on kuvata tietotekniikkaan liittyvät laiteresurssit ryhminä. Kun jonkin tehtävän suoritukseen tarvitaan resursseja, ne kerätään erikseen jokaisesta ryhmästä. Virtualisoinnin yksi osa-alue on palvelimen tai palvelinten virtualisointi, jossa pyritään hyödyntämään palvelinlaitteisto mahdollisimman tehokkaasti. Tehokkuus saavutetaan käyttämällä erillisiä instansseja, joita kutsutaan virtuaalikoneiksi. Tässä diplomityössä esitellään ja verrataan erilaisia palvelinten virtualisointimalleja ja tekniikoita, joita voidaan käyttää IA-32 arkkitehtuurin kanssa. Eroa virtualisoinnin ja eri partitiointitekniikoiden välillä tarkastellaan erikseen. Lisäksi muutoksia, joita palvelinten virtualisointi aiheuttaa infrastruktuuriin, ympäristöön ja laitteistoon käsitellään yleisellä tasolla. Teorian oikeellisuutta todistettiin suorittamalla useita testejä käyttäen kahta eri virtualisointiohjelmistoa. Testien perusteella palvelinten virtualisointi vähentää suorituskykyä ja luo ympäristön, jonka hallitseminen on vaikeampaa verrattuna perinteiseen ympäristöön. Myös tietoturvaa on katsottava uudesta näkökulmasta, sillä fyysistä eristystä ei virtuaalikoneille voida toteuttaa. Jotta virtualisoinnista saataisiin mahdollisimman suuri hyöty tuotantoympäristössä, vaaditaan tarkkaa harkintaa ja suunnitelmallisuutta. Parhaat käyttökohteet ovat erilaiset testiympäristöt, joissa vaatimukset suorituskyvyn ja turvallisuuden suhteen eivät ole niin tarkat.

Inter organizational learning within and by innovation networks

The thesis deals with the phenomenon of learning between organizations in innovation networks that develop new products, services or processes. Inter organizational learning is studied especially at the level of the network. The role of the network can be seen as twofold: either the network is a context for inter organizational learning, if the learner is something else than the network (organization, group, individual), or the network itself is the learner. Innovations are regarded as a primary source of competitiveness and renewal in organizations. Networking has become increasingly common particularly because of the possibility to extend the resource base of the organization through partnerships and to concentrate on core competencies. Especially in innovation activities, networks provide the possibility to answer the complex needs of the customers faster and to share the costs and risks of the development work. Networked innovation activities are often organized in practice as distributed virtual teams, either within one organization or as cross organizational co operation. The role of technology is considered in the research mainly as an enabling tool for collaboration and learning. Learning has been recognized as one important collaborative process in networks or as a motivation for networking. It is even more important in the innovation context as an enabler of renewal, since the essence of the innovation process is creating new knowledge, processes, products and services. The thesis aims at providing enhanced understanding about the inter organizational learning phenomenon in and by innovation networks, especially concentrating on the network level. The perspectives used in the research are the theoretical viewpoints and concepts, challenges, and solutions for learning. The methods used in the study are literature reviews and empirical research carried out with semi structured interviews analyzed with qualitative content analysis. The empirical research concentrates on two different areas, firstly on the theoretical approaches to learning that are relevant to innovation networks, secondly on learning in virtual innovation teams. As a result, the research identifies insights and implications for learning in innovation networks from several viewpoints on organizational learning. Using multiple perspectives allows drawing a many sided picture of the learning phenomenon that is valuable because of the versatility and complexity of situations and challenges of learning in the context of innovation and networks. The research results also show some of the challenges of learning and possible solutions for supporting especially network level learning.

Getting Started with Islandora

Workshop at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Research Data Management in the Cloud

Workshop at Open Repositories 2014, Helsinki, Finland, June 9-13, 2014

Évaluation et amélioration du rendement de la formation en entreprise : vers une démarche basée sur la gestion des processus d’affaires.

La formation est une stratégie clé pour le développement des compétences. Les entreprises continuent à investir dans la formation et le développement, mais elles possèdent rarement des données pour évaluer les résultats de cet investissement. La plupart des entreprises utilisent le modèle Kirkpatrick/Phillips pour évaluer la formation en entreprise. Cependant, il ressort de la littérature que les entreprises ont des difficultés à utiliser ce modèle. Les principales barrières sont la difficulté d’isoler l’apprentissage comme un facteur qui a une incidence sur les résultats, l’absence d’un système d’évaluation utile avec le système de gestion de l’apprentissage (Learning Management System - LMS) et le manque de données standardisées pour pouvoir comparer différentes fonctions d’apprentissage. Dans cette thèse, nous proposons un modèle (Analyse, Modélisation, Monitoring et Optimisation - AM2O) de gestion de projets de formation en entreprise, basée sur la gestion des processus d’affaires (Business Process Management - BPM). Un tel scénario suppose que les activités de formation en entreprise doivent être considérées comme des processus d’affaires. Notre modèle est inspiré de cette méthode (BPM), à travers la définition et le suivi des indicateurs de performance pour gérer les projets de formation dans les organisations. Elle est basée sur l’analyse et la modélisation des besoins de formation pour assurer l’alignement entre les activités de formation et les objectifs d’affaires de l’entreprise. Elle permet le suivi des projets de formation ainsi que le calcul des avantages tangibles et intangibles de la formation (sans coût supplémentaire). En outre, elle permet la production d’une classification des projets de formation en fonction de critères relatifs à l’entreprise. Ainsi, avec assez de données, notre approche peut être utilisée pour optimiser le rendement de la formation par une série de simulations utilisant des algorithmes d’apprentissage machine : régression logistique, réseau de neurones, co-apprentissage. Enfin, nous avons conçu un système informatique, Enterprise TRaining programs Evaluation and Optimization System - ETREOSys, pour la gestion des programmes de formation en entreprise et l’aide à la décision. ETREOSys est une plateforme Web utilisant des services en nuage (cloud services) et les bases de données NoSQL. A travers AM2O et ETREOSys nous résolvons les principaux problèmes liés à la gestion et l’évaluation de la formation en entreprise à savoir : la difficulté d’isoler les effets de la formation dans les résultats de l’entreprise et le manque de systèmes informatiques.

The assessment of benchmarks executed on bare-metal and using para-virtualization

A full assessment of para-­virtualization is important, because without knowledge about the various overheads, users can not understand whether using virtualization is a good idea or not. In this paper we are very interested in assessing the overheads of running various benchmarks on bare-­‐metal, as well as on para-­‐virtualization. The idea is to see what the overheads of para-­‐ virtualization are, as well as looking at the overheads of turning on monitoring and logging. The knowledge from assessing various benchmarks on these different systems will help a range of users understand the use of virtualization systems. In this paper we assess the overheads of using Xen, VMware, KVM and Citrix, see Table 1. These different virtualization systems are used extensively by cloud-­‐users. We are using various Netlib1 benchmarks, which have been developed by the University of Tennessee at Knoxville (UTK), and Oak Ridge National Laboratory (ORNL). In order to assess these virtualization systems, we run the benchmarks on bare-­‐metal, then on the para-­‐virtualization, and finally we turn on monitoring and logging. The later is important as users are interested in Service Level Agreements (SLAs) used by the Cloud providers, and the use of logging is a means of assessing the services bought and used from commercial providers. In this paper we assess the virtualization systems on three different systems. We use the Thamesblue supercomputer, the Hactar cluster and IBM JS20 blade server (see Table 2), which are all servers available at the University of Reading. A functional virtualization system is multi-­‐layered and is driven by the privileged components. Virtualization systems can host multiple guest operating systems, which run on its own domain, and the system schedules virtual CPUs and memory within each Virtual Machines (VM) to make the best use of the available resources. The guest-­‐operating system schedules each application accordingly. You can deploy virtualization as full virtualization or para-­‐virtualization. Full virtualization provides a total abstraction of the underlying physical system and creates a new virtual system, where the guest operating systems can run. No modifications are needed in the guest OS or application, e.g. the guest OS or application is not aware of the virtualized environment and runs normally. Para-­‐virualization requires user modification of the guest operating systems, which runs on the virtual machines, e.g. these guest operating systems are aware that they are running on a virtual machine, and provide near-­‐native performance. You can deploy both para-­‐virtualization and full virtualization across various virtualized systems. Para-­‐virtualization is an OS-­‐assisted virtualization; where some modifications are made in the guest operating system to enable better performance. In this kind of virtualization, the guest operating system is aware of the fact that it is running on the virtualized hardware and not on the bare hardware. In para-­‐virtualization, the device drivers in the guest operating system coordinate the device drivers of host operating system and reduce the performance overheads. The use of para-­‐virtualization [0] is intended to avoid the bottleneck associated with slow hardware interrupts that exist when full virtualization is employed. It has revealed [0] that para-­‐ virtualization does not impose significant performance overhead in high performance computing, and this in turn this has implications for the use of cloud computing for hosting HPC applications. The “apparent” improvement in virtualization has led us to formulate the hypothesis that certain classes of HPC applications should be able to execute in a cloud environment, with minimal performance degradation. In order to support this hypothesis, first it is necessary to define exactly what is meant by a “class” of application, and secondly it will be necessary to observe application performance, both within a virtual machine and when executing on bare hardware. A further potential complication is associated with the need for Cloud service providers to support Service Level Agreements (SLA), so that system utilisation can be audited.

MPJ express: Towards thread safe Java HPC

MPJ Express is a thread-safe Java messaging library that provides a full implementation of the mpiJava 1.2 API specification. This specification defines a MPI-like bindings for the Java language. We have implemented two communication devices as part of our library, the first, called niodev is based on the Java New I/O package and the second, called mxdev is based on the Myrinet eXpress library MPJ Express comes with an experimental runtitne, which allows portable bootstrapping of Java Virtual Machines across a cluster or network of computers. In this paper we describe the implementation of MPJ Express. Also, we present a performance comparison against various other C and Java messaging systems. A beta version of MPJ Express was released in September 2005.