Practical introduction to development of sage applications

Throughout this workshop session we have looked at various configurations of Sage as well as using the Sage UI to run Sage applications (e.g. the image viewer). More advanced usage of Sage has been demonstrated using a Sage compatible version of Paraview highlighting the potential of parallel rendering. The aim of this tutorial session is to give a practical introduction to developing visual content for a tiled display using the Sage libraries. After completing this tutorial you should have the basic tools required to develop your own custom Sage applications. This tutorial is designed for software developers and intermediate programming knowledge is assumed, along with some introductory OpenGL . You will be required to write small portions of C/C++ code to complete this worksheet. However if you do not feel comfortable writing code (or have never written in C or C++), we will be on hand throughout this session so feel free to ask for some help. We have a number of machines in this lab running a VNC client to a virtual machine running Fedora 12. You should all be able to log in with the username “escience”, and password “escience10”. Some of the commands in this worksheet require you to run them as the root user, so note the password as you may need to use it a few times. If you need to access the Internet, then use the username “qpsf01”, password “escience10”

I/O Device Virtualization in the multi-core era, a QoS perspective

In this paper, we propose an extension to the I/O device architecture, as recommended in the PCI-SIG IOV specification, for virtualizing network I/O devices. The aim is to enable fine-grained controls to a virtual machine on the I/O path of a shared device. The architecture allows native access of I/O devices to virtual machines and provides device level QoS hooks for controlling VM specific device usage. For evaluating the architecture we use layered queuing network (LQN) models. We implement the architecture and evaluate it using simulation techniques, on the LQN model, to demonstrate the benefits. With the architecture, the benefit for network I/O is 60% more than what can be expected on the existing architecture. Also, the proposed architecture improves scalability in terms of the number of virtual machines intending to share the I/O device.

LLVM-kääntäjäarkkitehtuuri ja sen soveltuvuus Java-virtuaalikoneen toteutuspohjaksi

Low Level Virtual Machine (LLVM) on moderni koko ohjelman elinkaaren optimointeihin keskittyvä kääntäjäarkkitehtuuri. Java-virtuaalikone on puolestaan suosittu korkean tason virtuaalikone, johon monien ohjelmointikielten toteutus nykyään perustuu. Tutkielmassa esitellään alun perin suorituskykyisen C- ja C++-kääntäjän toteuttamiseksi luotu LLVM-järjestelmä ja arvioidaan, miten hyvin LLVM-infrastruktuuri tukee Java-virtuaalikoneen toteuttamista. Tämän lisäksi tutkielmassa pohditaan, miten dynaamisten kielten usein tarvitsemaa suoritusaikaista ja lähdekieliriippuvaista optimointia voidaan tukea lähdekieliriippumattomassa LLVM-järjestelmässä. Lopuksi tutkielmassa esitellään kehitysehdotelma yleisen roskienkeruuinfrastruktuurin toteuttamiseksi LLVM:ssä, mikä tukisi dynaamista muistia automaattisesti hallitsevien kielten, kuten Javan ja sen virtuaalikoneen toteuttamista.

Modeling Architecture-OS interactions using Layered Queuing Network Models

The prevalent virtualization technologies provide QoS support within the software layers of the virtual machine monitor(VMM) or the operating system of the virtual machine(VM). The QoS features are mostly provided as extensions to the existing software used for accessing the I/O device because of which the applications sharing the I/O device experience loss of performance due to crosstalk effects or usable bandwidth. In this paper we examine the NIC sharing effects across VMs on a Xen virtualized server and present an alternate paradigm that improves the shared bandwidth and reduces the crosstalk effect on the VMs. We implement the proposed hardwaresoftware changes in a layered queuing network (LQN) model and use simulation techniques to evaluate the architecture. We find that simple changes in the device architecture and associated system software lead to application throughput improvement of up to 60%. The architecture also enables finer QoS controls at device level and increases the scalability of device sharing across multiple virtual machines. We find that the performance improvement derived using LQN model is comparable to that reported by similar but real implementations.

Virtualización de red basada en L2PNV para OpenStack

[ES]Este proyecto consiste en obtener un mayor control por parte del usuario a nivel de red en entornos con máquinas virtuales creadas a partir de la plataforma OpenStack. Cada vez que se arranca o inicia una máquina virtual en OpenStack, los parámetros de red se asignan por defecto, haciendo muy difícil su gestión y control tanto para investigación como para mantenimiento. Si estos parámetros siguieran un mismo patrón para cada proyecto o usuario sería mucho más sencillo tener controlado cada interfaz de red, pudiendo así gestionarlos de una manera más eficiente. Para realizar esta tarea será necesario introducir unos cambios en el código fuente de OpenStack, adaptándolo para que cumpla con nuestros requerimientos.

基于虚拟机架构的操作系统监控技术研究

当今随着计算系统资源和规模不断扩展，计算系统的虚拟化作为一种新型的计算模式，成为了研究热点。相对于传统的计算机架构，虚拟化计算系统在很多方面具有优势。在基于虚拟机架构的监控模型中，位于虚拟机监控器(Virtual Machine Monitor, VMM)中的监测模块获得比客户机内核更高的权限，并且对于客户机而言完全透明。所以相对与在传统操作系统环境中的监控模型，基于虚拟化架构可以对客户机操作系统进行更深入的监测。 Xen 是一个开放源代码的 VMM，由剑桥大学开发。由于其开源性质，非常适合在其基础上进行虚拟化的研究和开发。本文调研了Xen 的体系架构，以及 Xen 对 Intel 的 VT硬件虚拟化技术的支持。并研究了 Xen 的几种对客户机的内存管理方式，着重介绍了使用影子页表管理全虚拟化客户机内存的方法。 本文主要贡献是通过对上述知识的学习和分析，设计了基于Xen 虚拟机架构，对全虚拟化客户机操作系统的监控框架。并在这一框架基础上，利用对x86虚拟内存管理的页表属性控制，实现了对客户机 Windows 中指定进程的几种行为的监控实例 CASMonitor。包括通过影响 SYSENTER 指令的执行，监控Windows 中的系统调用；通过捕获虚拟机中进程对指定范围内存的写和执行操作，提供了一种可以监测程序自修改代码的技术，并能获取相关信息以对其进行后续分析处理。相对于现有的自修改代码监测技术，CASMonitor利用虚拟机架构可以实现动态，透明并且自动地监测。 关键词：虚拟化，Xen，自修改代码，监控

an integrity assurance mechanism for run-time programs

Chinese Assoc Cryptol Res, State Key Lab Informat Secur, Inst Software, Grad Univ Chinese Acad Sci, Natl Nat Sci Fdn China

Performance models of storage contention in cloud environments

We propose simple models to predict the performance degradation of disk requests due to storage device contention in consolidated virtualized environments. Model parameters can be deduced from measurements obtained inside Virtual Machines (VMs) from a system where a single VM accesses a remote storage server. The parameterized model can then be used to predict the effect of storage contention when multiple VMs are consolidated on the same server. We first propose a trace-driven approach that evaluates a queueing network with fair share scheduling using simulation. The model parameters consider Virtual Machine Monitor level disk access optimizations and rely on a calibration technique. We further present a measurement-based approach that allows a distinct characterization of read/write performance attributes. In particular, we define simple linear prediction models for I/O request mean response times, throughputs and read/write mixes, as well as a simulation model for predicting response time distributions. We found our models to be effective in predicting such quantities across a range of synthetic and emulated application workloads. 

IO Performance Prediction in Consolidated Virtualized Environments

We propose a trace-driven approach to predict the performance degradation of disk request response times due to storage device contention in consolidated virtualized environments. Our performance model evaluates a queueing network with fair share scheduling using trace-driven simulation. The model parameters can be deduced from measurements obtained inside Virtual Machines (VMs) from a system where a single VM accesses a remote storage server. The parameterized model can then be used to predict the effect of storage contention when multiple VMs are consolidated on the same virtualized server. The model parameter estimation relies on a search technique that tries to estimate the splitting and merging of blocks at the the Virtual Machine Monitor (VMM) level in the case of multiple competing VMs. Simulation experiments based on traces of the Postmark and FFSB disk benchmarks show that our model is able to accurately predict the impact of workload consolidation on VM disk IO response times.

A Lightweight Tool for Anomaly Detection in Cloud Data Centres

Cloud data centres are critical business infrastructures and the fastest growing service providers. Detecting anomalies in Cloud data centre operation is vital. Given the vast complexity of the data centre system software stack, applications and workloads, anomaly detection is a challenging endeavour. Current tools for detecting anomalies often use machine learning techniques, application instance behaviours or system metrics distribu- tion, which are complex to implement in Cloud computing environments as they require training, access to application-level data and complex processing. This paper presents LADT, a lightweight anomaly detection tool for Cloud data centres that uses rigorous correlation of system metrics, implemented by an efficient corre- lation algorithm without need for training or complex infrastructure set up. LADT is based on the hypothesis that, in an anomaly-free system, metrics from data centre host nodes and virtual machines (VMs) are strongly correlated. An anomaly is detected whenever correlation drops below a threshold value. We demonstrate and evaluate LADT using a Cloud environment, where it shows that the hosting node I/O operations per second (IOPS) are strongly correlated with the aggregated virtual machine IOPS, but this correlation vanishes when an application stresses the disk, indicating a node-level anomaly.

DocLite: A Docker-Based Lightweight Cloud Benchmarking Tool

Existing benchmarking methods are time consuming processes as they typically benchmark the entire Virtual Machine (VM) in order to generate accurate performance data, making them less suitable for real-time analytics. The research in this paper is aimed to surmount the above challenge by presenting DocLite - Docker Container-based Lightweight benchmarking tool. DocLite explores lightweight cloud benchmarking methods for rapidly executing benchmarks in near real-time. DocLite is built on the Docker container technology, which allows a user-defined memory size and number of CPU cores of the VM to be benchmarked. The tool incorporates two benchmarking methods - the first referred to as the native method employs containers to benchmark a small portion of the VM and generate performance ranks, and the second uses historic benchmark data along with the native method as a hybrid to generate VM ranks. The proposed methods are evaluated on three use-cases and are observed to be up to 91 times faster than benchmarking the entire VM. In both methods, small containers provide the same quality of rankings as a large container. The native method generates ranks with over 90% and 86% accuracy for sequential and parallel execution of an application compared against benchmarking the whole VM. The hybrid method did not improve the quality of the rankings significantly.

Dynamic power- and failure-aware cloud resources allocation for Sets of independent tasks

Cloud computing is increasingly being adopted in different scenarios, like social networking, business applications, scientific experiments, etc. Relying in virtualization technology, the construction of these computing environments targets improvements in the infrastructure, such as power-efficiency and fulfillment of users’ SLA specifications. The methodology usually applied is packing all the virtual machines on the proper physical servers. However, failure occurrences in these networked computing systems can induce substantial negative impact on system performance, deviating the system from ours initial objectives. In this work, we propose adapted algorithms to dynamically map virtual machines to physical hosts, in order to improve cloud infrastructure power-efficiency, with low impact on users’ required performance. Our decision making algorithms leverage proactive fault-tolerance techniques to deal with systems failures, allied with virtual machine technology to share nodes resources in an accurately and controlled manner. The results indicate that our algorithms perform better targeting power-efficiency and SLA fulfillment, in face of cloud infrastructure failures.

Lecture 2 - Java

Programming Overview The JVM (The Java Virtual Machine) A brief look at Structure Class Method Statement Magic incantations main() output Coding a Dog Programming Principle(1) If and Boolean operations Coding a Bank Account Quick look at ToolBox

The CAFS system today and tomorrow

The CAFS search engine is a real machine in a virtual machine world; it is the hardware component of ICL's CAFS system. The paper is an introduction and prelude to the set of papers in this volume on CAFS applications. It defines The CAFS system and its context together with the function of its hardware and software components. It examines CAFS' role in the broad context of application development and information systems; it highlights some techniques and applications which exploit the CAFS system. Finally, it concludes with some suggestions for possible further developments. 'Search out thy wit for secret policies And we will make thee famous through the world' Henry VI, 1:3

The assessment of benchmarks executed on bare-metal and using para-virtualization

A full assessment of para-­virtualization is important, because without knowledge about the various overheads, users can not understand whether using virtualization is a good idea or not. In this paper we are very interested in assessing the overheads of running various benchmarks on bare-­‐metal, as well as on para-­‐virtualization. The idea is to see what the overheads of para-­‐ virtualization are, as well as looking at the overheads of turning on monitoring and logging. The knowledge from assessing various benchmarks on these different systems will help a range of users understand the use of virtualization systems. In this paper we assess the overheads of using Xen, VMware, KVM and Citrix, see Table 1. These different virtualization systems are used extensively by cloud-­‐users. We are using various Netlib1 benchmarks, which have been developed by the University of Tennessee at Knoxville (UTK), and Oak Ridge National Laboratory (ORNL). In order to assess these virtualization systems, we run the benchmarks on bare-­‐metal, then on the para-­‐virtualization, and finally we turn on monitoring and logging. The later is important as users are interested in Service Level Agreements (SLAs) used by the Cloud providers, and the use of logging is a means of assessing the services bought and used from commercial providers. In this paper we assess the virtualization systems on three different systems. We use the Thamesblue supercomputer, the Hactar cluster and IBM JS20 blade server (see Table 2), which are all servers available at the University of Reading. A functional virtualization system is multi-­‐layered and is driven by the privileged components. Virtualization systems can host multiple guest operating systems, which run on its own domain, and the system schedules virtual CPUs and memory within each Virtual Machines (VM) to make the best use of the available resources. The guest-­‐operating system schedules each application accordingly. You can deploy virtualization as full virtualization or para-­‐virtualization. Full virtualization provides a total abstraction of the underlying physical system and creates a new virtual system, where the guest operating systems can run. No modifications are needed in the guest OS or application, e.g. the guest OS or application is not aware of the virtualized environment and runs normally. Para-­‐virualization requires user modification of the guest operating systems, which runs on the virtual machines, e.g. these guest operating systems are aware that they are running on a virtual machine, and provide near-­‐native performance. You can deploy both para-­‐virtualization and full virtualization across various virtualized systems. Para-­‐virtualization is an OS-­‐assisted virtualization; where some modifications are made in the guest operating system to enable better performance. In this kind of virtualization, the guest operating system is aware of the fact that it is running on the virtualized hardware and not on the bare hardware. In para-­‐virtualization, the device drivers in the guest operating system coordinate the device drivers of host operating system and reduce the performance overheads. The use of para-­‐virtualization [0] is intended to avoid the bottleneck associated with slow hardware interrupts that exist when full virtualization is employed. It has revealed [0] that para-­‐ virtualization does not impose significant performance overhead in high performance computing, and this in turn this has implications for the use of cloud computing for hosting HPC applications. The “apparent” improvement in virtualization has led us to formulate the hypothesis that certain classes of HPC applications should be able to execute in a cloud environment, with minimal performance degradation. In order to support this hypothesis, first it is necessary to define exactly what is meant by a “class” of application, and secondly it will be necessary to observe application performance, both within a virtual machine and when executing on bare hardware. A further potential complication is associated with the need for Cloud service providers to support Service Level Agreements (SLA), so that system utilisation can be audited.