Running climate models on grids using G-Rex

Compute grids are used widely in many areas of environmental science, but there has been limited uptake of grid computing by the climate modelling community, partly because the characteristics of many climate models make them difficult to use with popular grid middleware systems. In particular, climate models usually produce large volumes of output data, and running them also involves complicated workflows implemented as shell scripts. A new grid middleware system that is well suited to climate modelling applications is presented in this paper. Grid Remote Execution (G-Rex) allows climate models to be deployed as Web services on remote computer systems and then launched and controlled as if they were running on the user's own computer. Output from the model is transferred back to the user while the run is in progress to prevent it from accumulating on the remote system and to allow the user to monitor the model. G-Rex has a REST architectural style, featuring a Java client program that can easily be incorporated into existing scientific workflow scripts. Some technical details of G-Rex are presented, with examples of its use by climate modellers.

Running Climate Models On The NERC Cluster Grid Using G-Rex

Compute grids are used widely in many areas of environmental science, but there has been limited uptake of grid computing by the climate modelling community, partly because the characteristics of many climate models make them difficult to use with popular grid middleware systems. In particular, climate models usually produce large volumes of output data, and running them usually involves complicated workflows implemented as shell scripts. For example, NEMO (Smith et al. 2008) is a state-of-the-art ocean model that is used currently for operational ocean forecasting in France, and will soon be used in the UK for both ocean forecasting and climate modelling. On a typical modern cluster, a particular one year global ocean simulation at 1-degree resolution takes about three hours when running on 40 processors, and produces roughly 20 GB of output as 50000 separate files. 50-year simulations are common, during which the model is resubmitted as a new job after each year. Running NEMO relies on a set of complicated shell scripts and command utilities for data pre-processing and post-processing prior to job resubmission. Grid Remote Execution (G-Rex) is a pure Java grid middleware system that allows scientific applications to be deployed as Web services on remote computer systems, and then launched and controlled as if they are running on the user's own computer. Although G-Rex is general purpose middleware it has two key features that make it particularly suitable for remote execution of climate models: (1) Output from the model is transferred back to the user while the run is in progress to prevent it from accumulating on the remote system and to allow the user to monitor the model; (2) The client component is a command-line program that can easily be incorporated into existing model work-flow scripts. G-Rex has a REST (Fielding, 2000) architectural style, which allows client programs to be very simple and lightweight and allows users to interact with model runs using only a basic HTTP client (such as a Web browser or the curl utility) if they wish. This design also allows for new client interfaces to be developed in other programming languages with relatively little effort. The G-Rex server is a standard Web application that runs inside a servlet container such as Apache Tomcat and is therefore easy to install and maintain by system administrators. G-Rex is employed as the middleware for the NERC1 Cluster Grid, a small grid of HPC2 clusters belonging to collaborating NERC research institutes. Currently the NEMO (Smith et al. 2008) and POLCOMS (Holt et al, 2008) ocean models are installed, and there are plans to install the Hadley Centre’s HadCM3 model for use in the decadal climate prediction project GCEP (Haines et al., 2008). The science projects involving NEMO on the Grid have a particular focus on data assimilation (Smith et al. 2008), a technique that involves constraining model simulations with observations. The POLCOMS model will play an important part in the GCOMS project (Holt et al, 2008), which aims to simulate the world’s coastal oceans. A typical use of G-Rex by a scientist to run a climate model on the NERC Cluster Grid proceeds as follows :(1) The scientist prepares input files on his or her local machine. (2) Using information provided by the Grid’s Ganglia3 monitoring system, the scientist selects an appropriate compute resource. (3) The scientist runs the relevant workflow script on his or her local machine. This is unmodified except that calls to run the model (e.g. with “mpirun”) are simply replaced with calls to "GRexRun" (4) The G-Rex middleware automatically handles the uploading of input files to the remote resource, and the downloading of output files back to the user, including their deletion from the remote system, during the run. (5) The scientist monitors the output files, using familiar analysis and visualization tools on his or her own local machine. G-Rex is well suited to climate modelling because it addresses many of the middleware usability issues that have led to limited uptake of grid computing by climate scientists. It is a lightweight, low-impact and easy-to-install solution that is currently designed for use in relatively small grids such as the NERC Cluster Grid. A current topic of research is the use of G-Rex as an easy-to-use front-end to larger-scale Grid resources such as the UK National Grid service.

Accessing biodiversity resources in computational environments from workflow applications

In the Biodiversity World (BDW) project we have created a flexible and extensible Web Services-based Grid environment for biodiversity researchers to solve problems in biodiversity and analyse biodiversity patterns. In this environment, heterogeneous and globally distributed biodiversity-related resources such as data sets and analytical tools are made available to be accessed and assembled by users into workflows to perform complex scientific experiments. One such experiment is bioclimatic modelling of the geographical distribution of individual species using climate variables in order to predict past and future climate-related changes in species distribution. Data sources and analytical tools required for such analysis of species distribution are widely dispersed, available on heterogeneous platforms, present data in different formats and lack interoperability. The BDW system brings all these disparate units together so that the user can combine tools with little thought as to their availability, data formats and interoperability. The current Web Servicesbased Grid environment enables execution of the BDW workflow tasks in remote nodes but with a limited scope. The next step in the evolution of the BDW architecture is to enable workflow tasks to utilise computational resources available within and outside the BDW domain. We describe the present BDW architecture and its transition to a new framework which provides a distributed computational environment for mapping and executing workflows in addition to bringing together heterogeneous resources and analytical tools.

Exploring context-awareness in the construction logistics services delivery

Managing a construction project supply chain effectively and efficiently is extremely difficult due to involvement of numerous sectors that are supported by ineffective communication system. An efficient construction supply chain system ensures the delivery of materials and other services to construction site while minimising costs and rewarding all sectors based on value added to the supply chain. The advancement of information, communication and wireless technologies is driving construction companies to deploy supply chain management strategies to seek better outputs. As part of the emerging wireless technologies, contextaware computing capability represents the next generation of ICT to the construction services. Conceptually, context-awareness could be integrated with Web Services in order to ensure the delivery of pertinent information to construction site and enhance construction supply chain collaboration. An initial study has indicated that this integrated system has the potential of serving and improving the construction services delivery through access to context-specific data, information and services on as-needed basis.

e-learning services provision and management

With the rapid development of information technology, learners demand effective personalised learning support, which imposes a new learning paradigm in learning content management. Standards as well as best practice in industry and research community have taken place to address the paradigm shift. With respect to this trend, it is recognised that finding learning content which meet personal learning requirements remains challenging. This paper describes a model of e-learning services provision which integrates the best practice in e-learning and Web services technology so that learning content management is capable of supporting applications of learning services.

Plug-and-play remote portlet publishing

Web Services for Remote Portlets (WSRP) is gaining attention among portal developers and vendors to enable easy development, increased richness in functionality, pluggability, and flexibility of deployment. Whilst currently not supporting all WSRP functionalities, open-source portal frameworks could in future use WSRP Consumers to access remote portlets found from a WSRP Producer registry service. This implies that we need a central registry for the remote portlets and a more expressive WSRP Consumer interface to implement the remote portlet functions. This paper reports on an investigation into a new system architecture, which includes a Web Services repository, registry, and client interface. The Web Services repository holds portlets as remote resource producers. A new data structure for expressing remote portlets is found and published by populating a Universal Description, Discovery and Integration (UDDI) registry. A remote portlet publish and search engine for UDDI has also been developed. Finally, a remote portlet client interface was developed as a Web application. The client interface supports remote portlet features, as well as window status and mode functions. Copyright (c) 2007 John Wiley & Sons, Ltd.

Styx grid services: lightweight middleware for efficient scientific workflows

The service-oriented approach to performing distributed scientific research is potentially very powerful but is not yet widely used in many scientific fields. This is partly due to the technical difficulties involved in creating services and workflows and the inefficiency of many workflow systems with regard to handling large datasets. We present the Styx Grid Service, a simple system that wraps command-line programs and allows them to be run over the Internet exactly as if they were local programs. Styx Grid Services are very easy to create and use and can be composed into powerful workflows with simple shell scripts or more sophisticated graphical tools. An important feature of the system is that data can be streamed directly from service to service, significantly increasing the efficiency of workflows that use large data volumes. The status and progress of Styx Grid Services can be monitored asynchronously using a mechanism that places very few demands on firewalls. We show how Styx Grid Services can interoperate with with Web Services and WS-Resources using suitable adapters.

A conceptual life event framework for government-to-citizen electronic service provision

In recent years, life event approach has been widely used by governments all over the world for designing and providing web services to citizens through their e-government portals. Despite the wide usage of this approach, there is still a challenge of how to use this approach to design e-government portals in order to automatically provide personalised services to citizens. We propose a conceptual framework for e-government service provision based on life event approach and the use of citizen profile to capture the citizen needs, since the process of finding Web services from a government-to-citizen (G2C) system involves understanding the citizens’ needs and demands, selecting the relevant services, and delivering services that matches the requirements. The proposed framework that incorporates the citizen profile is based on three components that complement each other, namely, anticipatory life events, non-anticipatory life events and recurring services.

Using a RESTful messaging and registry system to support a range a distributed applications

Tycho was conceived in 2003 in response to a need by the GridRM [1] resource-monitoring project for a ldquolight-weightrdquo, scalable and easy to use wide-area distributed registry and messaging system. Since Tycho's first release in 2006 a number of modifications have been made to the system to make it easier to use and more flexible. Since its inception, Tycho has been utilised across a number of application domains including widearea resource monitoring, distributed queries across archival databases, providing services for the nodes of a Cray supercomputer, and as a system for transferring multi-terabyte scientific datasets across the Internet. This paper provides an overview of the initial Tycho system, describes a number of applications that utilise Tycho, discusses a number of new utilities, and how the Tycho infrastructure has evolved in response to experience of building applications with it.

Parameterisation of orographic cloud dynamics in a GCM

A new parameterisation is described that predicts the temperature perturbations due to sub-grid scale orographic gravity waves in the atmosphere of the 19 level HadAM3 version of the United Kingdom Met Office Unified Model. The explicit calculation of the wave phase allows the sign of the temperature perturbation to be predicted. The scheme is used to create orographic clouds, including cirrus, that were previously absent in model simulations. A novel approach to the validation of this parameterisation makes use of both satellite observations of a case study, and a simulation in which the Unified Model is nudged towards ERA-40 assimilated winds, temperatures and humidities. It is demonstrated that this approach offers a feasible way of introducing large scale orographic cirrus clouds into GCMs.

Validation of ocean model syntheses against hydrography using a new web application

Results are presented from a new web application called OceanDIVA - Ocean Data Intercomparison and Visualization Application. This tool reads hydrographic profiles and ocean model output and presents the data on either depth levels or isotherms for viewing in Google Earth, or as probability density functions (PDFs) of regional model-data misfits. As part of the CLIVAR Global Synthesis and Observations Panel, an intercomparison of water mass properties of various ocean syntheses has been undertaken using OceanDIVA. Analysis of model-data misfits reveals significant differences between the water mass properties of the syntheses, such as the ability to capture mode water properties.

Effect of ENSO phase on large-scale snow water equivalent distribution in a GCM

Understanding links between the El Nino-Southern Oscillation (ENSO) and snow would be useful for seasonal forecasting, but also for understanding natural variability and interpreting climate change predictions. Here, a 545-year run of the general circulation model HadCM3, with prescribed external forcings and fixed greenhouse gas concentrations, is used to explore the impact of ENSO on snow water equivalent (SWE) anomalies. In North America, positive ENSO events reduce the mean SWE and skew the distribution towards lower values, and vice versa during negative ENSO events. This is associated with a dipole SWE anomaly structure, with anomalies of opposite sign centered in western Canada and the central United States. In Eurasia, warm episodes lead to a more positively skewed distribution and the mean SWE is raised. Again, the opposite effect is seen during cold episodes. In Eurasia the largest anomalies are concentrated in the Himalayas. These correlations with February SWE distribution are seen to exist from the previous June-July-August (JJA) ENSO index onwards, and are weakly detected in 50-year subsections of the control run, but only a shifted North American response can be detected in the anaylsis of 40 years of ERA40 reanalysis data. The ENSO signal in SWE from the long run could still contribute to regional predictions although it would be a weak indicator only

Using idealized snow forcing to test teleconnections with the Indian summer monsoon in the Hadley Centre GCM

Anomalous heavy snow during winter or spring has long been regarded as a possible precursor of deficient Indian monsoon rainfall during the subsequent summer. However previous work in this field is inconclusive, in terms of the mechanism that communicates snow anomalies to the monsoon summer, and even the region from which snow has the most impact. In this study we explore these issues in coupled and atmosphere-only versions of the Hadley Centre model. A 1050-year control integration of the HadCM3 coupled model, which well represents the seasonal cycle of snow cover over the Eurasian continent, is analysed and shows evidence for weakened monsoons being preceded by strong snow forcing (in the absence of ENSO) over either the Himalaya/Tibetan Plateau or north/west Eurasia regions. However, empirical orthogonal function (EOF) analysis of springtime interannual variability in snow depth shows the leading mode to have opposite signs between these two regions, suggesting that competing mechanisms may be possible. To determine the dominant region, ensemble integrations are carried out using HadAM3, the atmospheric component of HadCM3, and a variety of anomalous snow forcing initial conditions obtained from the control integration of the coupled model. Forcings are applied during spring in separate experiments over the Himalaya/Tibetan Plateau and north/west Eurasia regions, in conjunction with climatological SSTs in order to avoid the direct effects of ENSO. With the aid of idealized forcing conditions in sensitivity tests, we demonstrate that forcing from the Himalaya region is dominant in this model via a Blanford-type mechanism involving reduced surface sensible heat and longwave fluxes, reduced heating of the troposphere over the Tibetan Plateau and consequently a reduced meridional tropospheric temperature gradient which weakens the monsoon during early summer. Snow albedo is shown to be key to the mechanism, explaining around 50% of the perturbation in sensible heating over the Tibetan Plateau, and accounting for the majority of cooling through the troposphere.