Evaluating the EASY-backfill job scheduling of static workloads on clusters

This research aims at improving our understanding of backfilling job scheduling algorithms. The most frequently used algorithm, EASY-backfilling, was selected for a performance evaluation by scheduling static workloads of parallel jobs on a computer cluster. To achieve the aim, we have developed a batch job scheduler for Linux clusters, implemented several scheduling algorithms including ARCA and EASY-Backfilling, and carried out their performance evaluation by running well known MPI applications on a real cluster. Our performance evaluation carried out for EASY-Backfilling serves two purposes. First, the performance results obtained from our evaluation can be used to validate other researcherspsila results generated by simulation, and second, the methodology used in our evaluation has alleviated many problems existed in the simulations presented in the current literature.

The impact of under-estimated length of jobs on EASY-backfill scheduling

The issue of under-estimated length of jobs (parallel applications) on backfill-based scheduling is ignored in the current literature because users want to avoid their jobs to be killed when the requested time expires. Therefore, users prefer to over-estimate the length of their jobs. This paper shows the impact of underestimated length of jobs on their execution performance in an EASY-backfill scheduling-based system. We have developed a batch job scheduler for Linux clusters that implements an enhanced EASY- backfilling algorithm in such a way that a job with an under-estimated execution time would not be killed unless it would delay other jobs. We have carried out performance evaluation by scheduling static workloads of well known MPI parallel applications on a real cluster. Our results show that most of the jobs do not have to be aborted even though their job lengths are under-estimated whereas the slowdown of jobs and the throughput of the system are only slightly degraded.

Change of the century - free software and the positive possibility

The much touted ‘freedoms’ of FLOSS are coming under increasing scrutiny as they are applied to contexts beyond their original formation. Is ‘freedom as in speech’ enough or are there other freedoms upon which the construction of the commons depends? Martin Hardie has worked extensively on an archeology of how the GNU/Linux operating system was developed, exposing the myths that are at its foundation. Here, he asks how the licensing of FLOSS operates within the constitution of Empire and locates in the new forms of ‘producing in common’ the means to reverse the proliferation of alternative law and instead affirm a true alternative to law.

Clonewise - detecting package-level clones using machine learning

Developers sometimes maintain an internal copy of another software or fork development of an existing project. This practice can lead to software vulnerabilities when the embedded code is not kept up to date with upstream sources. We propose an automated solution to identify clones of packages without any prior knowledge of these relationships. We then correlate clones with vulnerability information to identify outstanding security problems. This approach motivates software maintainers to avoid using cloned packages and link against system wide libraries. We propose over 30 novel features that enable us to use to use pattern classification to accurately identify package-level clones. To our knowledge, we are the first to consider clone detection as a classification problem. Our results show our system, Clonewise, compares well to manually tracked databases. Based on our work, over 30 unknown package clones and vulnerabilities have been identified and patched.

J-Circos: an interactive Circos plotter

Circos plots are graphical outputs that display three dimensional chromosomal interactions and fusion transcripts. However, the Circos plot tool is not an interactive visualization tool, but rather a figure generator. For example, it does not enable data to be added dynamically nor does it provide information for specific data points interactively. Recently, an R-based Circos tool (RCircos) has been developed to integrate Circos to R, but similarly, Rcircos can only be used to generate plots. Thus, we have developed a Circos plot tool (J-Circos) that is an interactive visualization tool that can plot Circos figures, as well as being able to dynamically add data to the figure, and providing information for specific data points using mouse hover display and zoom in/out functions. J-Circos uses the Java computer language to enable, it to be used on most operating systems (Windows, MacOS, Linux). Users can input data into J-Circos using flat data formats, as well as from the Graphical user interface (GUI). J-Circos will enable biologists to better study more complex chromosomal interactions and fusion transcripts that are otherwise difficult to visualize from next-generation sequencing data. Availability and implementation: J-circos and its manual are freely available at http://www.australianprostatecentre.org/research/software/jcircos CONTACT: j.an@qut.edu.au Supplementary information: Supplementary data are available at Bioinformatics online.

A survey of approaches and frameworks to carry out genomic data analysis on the cloud

High Performance Computing (HPC) clouds have started to change the way how research in science, in particular medicine and genomics (bioinformatics) is being carried out. Researchers who have taken advantage of this technology can process larger amounts of data and speed up scientific discovery. However, most HPC clouds are provided at an Infrastructure as a Service (IaaS) level, users are presented with a set of virtual servers which need to be put together to form HPC environments via time consuming resource management and software configuration tasks, which make them practically unusable by discipline, non-computing specialists. In response, there is a new trend to expose cloud applications as services to simplify access and execution on clouds. This paper firstly examines commonly used cloud-based genomic analysis services (Tuxedo Suite, Galaxy and Cloud Bio Linux). As a follow up, we propose two new solutions (HPCaaS and Uncinus), which aim to automate aspects of the service development and deployment process. By comparing and contrasting these five solutions, we identify key mechanisms of service creation, execution and access that are required to support genomic research on the SaaS cloud, in particular by discipline specialists. © 2014 IEEE.

An improved TIGER2 implementation for NAMD suitable for the Blue Gene architecture

Here we present an improved implementation of the TIGER2 Replica Exchange Molecular Dynamics (REMD) method, using the replica exchange Application Programming Interface (API) found in contemporary versions of the NAMD Molecular Dynamics Package. The implementation takes the form of a TCL script which is used in conjunction with the standard configuration file. This implementation is validated against a previous TIGER2 implementation, as well as data reported for the original TIGER2 simulations. Our implementation is compatible with a range of architectures; crucially it enables the use of this wrapper with the BlueGene/Q architecture, in addition to the x86 architecture. Program summary: Program title: TIGER2-NAMD. Catalogue identifier: AEWC_v1_0. Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEWC_v1_0.html Program obtainable from: CPC Program Library, Queen's University, Belfast, N. Ireland. Licensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.html No. of lines in distributed program, including test data, etc.: 34151. No. of bytes in distributed program, including test data, etc.: 424217. Distribution format: tar.gz. Programming language: Tcl 8.5. Computer: x86 Clusters, BlueGene/Q, Workstations. Operating system: Linux, IBM Compute Node Kernel. Has the code been vectorised or parallelised?: Yes. MPI Parallelism. Classification: 3. External routines: NAMD 2.9 (http://www.ks.uiuc.edu/Research/namd/). Nature of problem: Replica Exchange Molecular Dynamics. Solution method: Each replica runs through multiple cycles of heating and cooling with exchanges between them being attempted. Running time: Typically 30 mins, up to an hour.

Providing fairer resource allocation for multi-tenant cloud-based systems

A fundamental premise in cloud computing is trying to provide a more sophisticated computing resource sharing capability. In order to provide better allocation, the Dominant Resource Fairness (DRF) approach has been developed to address the "fair resource allocation problem" at the application layer for multi-tenant cloud applications. Nevertheless conventional DRF only considers the interplay of CPU and memory, which may result in over allocation of resources to one tenant's application to the detriment of others. In this paper, we propose an improved DRF algorithm with 3-dimensional demand vector to support disk resources as the third dominant shared resource, enhancing fairer resource sharing. Our technique is integrated with LINUX 'group' controls resource utilisation and realises data isolation to avoid undesirable interactions between co-located tasks. Our method ensures all tenants receive system resources fairly, which improves overall utilisation and throughput as well as reducing traffic in an over-crowded system. We evaluate the performance of different types of workload using different algorithms and compare ours to the default algorithm. Results show an increase of 15% resource utilisation and a reduction of 59% completion time on average, indicating that our DRF algorithm provides a better, smoother, fairer high-performance resource allocation scheme for both continuous workloads and batch jobs.