Cloud computing in e-science: research challenges and opportunities

SOA (Service Oriented Architecture), workflow, the Semantic Web, and Grid computing are key enabling information technologies in the development of increasingly sophisticated e-Science infrastructures and application platforms. While the emergence of Cloud computing as a new computing paradigm has provided new directions and opportunities for e-Science infrastructure development, it also presents some challenges. Scientific research is increasingly finding that it is difficult to handle “big data” using traditional data processing techniques. Such challenges demonstrate the need for a comprehensive analysis on using the above mentioned informatics techniques to develop appropriate e-Science infrastructure and platforms in the context of Cloud computing. This survey paper describes recent research advances in applying informatics techniques to facilitate scientific research particularly from the Cloud computing perspective. Our particular contributions include identifying associated research challenges and opportunities, presenting lessons learned, and describing our future vision for applying Cloud computing to e-Science. We believe our research findings can help indicate the future trend of e-Science, and can inform funding and research directions in how to more appropriately employ computing technologies in scientific research. We point out the open research issues hoping to spark new development and innovation in the e-Science field.

What is the right balance for performance and isolation with virtualization in HPC?

The use of virtualization in high-performance computing (HPC) has been suggested as a means to provide tailored services and added functionality that many users expect from full-featured Linux cluster environments. The use of virtual machines in HPC can offer several benefits, but maintaining performance is a crucial factor. In some instances the performance criteria are placed above the isolation properties. This selective relaxation of isolation for performance is an important characteristic when considering resilience for HPC environments that employ virtualization. In this paper we consider some of the factors associated with balancing performance and isolation in configurations that employ virtual machines. In this context, we propose a classification of errors based on the concept of “error zones”, as well as a detailed analysis of the trade-offs between resilience and performance based on the level of isolation provided by virtualization solutions. Finally, a set of experiments are performed using different virtualization solutions to elucidate the discussion.

The MELODIES project: integrating diverse data using Linked Data and cloud computing

We present an overview of the MELODIES project, which is developing new data-intensive environmental services based on data from Earth Observation satellites, government databases, national and European agencies and more. We focus here on the capabilities and benefits of the project’s “technical platform”, which applies cloud computing and Linked Data technologies to enable the development of these services, providing flexibility and scalability.

Aerosol direct radiative effect of smoke over clouds over the southeast Atlantic Ocean from 2006 to 2009

The aerosol direct radiative effect (DRE) of African smoke was analyzed in cloud scenes over the southeast Atlantic Ocean, using Scanning Imaging Absorption Spectrometer for Atmospheric Chartography (SCIAMACHY) satellite observations and Hadley Centre Global Environmental Model version 2 (HadGEM2) climate model simulations. The observed mean DRE was about 30–35 W m−2 in August and September 2006–2009. In some years, short episodes of high-aerosol DRE can be observed, due to high-aerosol loadings, while in other years the loadings are lower but more prolonged. Climate models that use evenly distributed monthly averaged emission fields will not reproduce these high-aerosol loadings. Furthermore, the simulated monthly mean aerosol DRE in HadGEM2 is only about 6 W m−2 in August. The difference with SCIAMACHY mean observations can be partly explained by an underestimation of the aerosol absorption Ångström exponent in the ultraviolet. However, the subsequent increase of aerosol DRE simulation by about 20% is not enough to explain the observed discrepancy between simulations and observations.

Observing wind, aerosol particles, clouds and precipitation: Finland's new ground-based remote-sensing network

The Finnish Meteorological Institute, in collaboration with the University of Helsinki, has established a new ground-based remote-sensing network in Finland. The network consists of five topographically, ecologically and climatically different sites distributed from southern to northern Finland. The main goal of the network is to monitor air pollution and boundary layer properties in near real time, with a Doppler lidar and ceilometer at each site. In addition to these operational tasks, two sites are members of the Aerosols, Clouds and Trace gases Research InfraStructure Network (ACTRIS); a Ka band cloud radar at Sodankylä will provide cloud retrievals within CloudNet, and a multi-wavelength Raman lidar, PollyXT (POrtabLe Lidar sYstem eXTended), in Kuopio provides optical and microphysical aerosol properties through EARLINET (the European Aerosol Research Lidar Network). Three C-band weather radars are located in the Helsinki metropolitan area and are deployed for operational and research applications. We performed two inter-comparison campaigns to investigate the Doppler lidar performance, compare the backscatter signal and wind profiles, and to optimize the lidar sensitivity through adjusting the telescope focus length and data-integration time to ensure sufficient signal-to-noise ratio (SNR) in low-aerosol-content environments. In terms of statistical characterization, the wind-profile comparison showed good agreement between different lidars. Initially, there was a discrepancy in the SNR and attenuated backscatter coefficient profiles which arose from an incorrectly reported telescope focus setting from one instrument, together with the need to calibrate. After diagnosing the true telescope focus length, calculating a new attenuated backscatter coefficient profile with the new telescope function and taking into account calibration, the resulting attenuated backscatter profiles all showed good agreement with each other. It was thought that harsh Finnish winters could pose problems, but, due to the built-in heating systems, low ambient temperatures had no, or only a minor, impact on the lidar operation – including scanning-head motion. However, accumulation of snow and ice on the lens has been observed, which can lead to the formation of a water/ice layer thus attenuating the signal inconsistently. Thus, care must be taken to ensure continuous snow removal.

Major differences of cloud computing adoption in universities: Europe vs. Middle East

The extensive use of cloud computing in educational institutes around the world brings unique challenges for universities. Some of these challenges are due to clear differences between Europe and Middle East universities. These differences stem from the natural variation between people. Cloud computing has created a new concept to deal with software services and hardware infrastructure. Some benefits are immediately gained, for instance, to allow students to share their information easily and to discover new experiences of the education system. However, this introduces more challenges, such as security and configuration of resources in shared environments. Educational institutes cannot escape from these challenges. Yet some differences occur between universities which use cloud computing as an educational tool or a form of social connection. This paper discusses some benefits and limitations of using cloud computing and major differences in using cloud computing at universities in Europe and the Middle East, based on the social perspective, security and economics concepts, and personal responsibility.

Clouds, circulation and climate sensitivity

Fundamental puzzles of climate science remain unsolved because of our limited understanding of how clouds, circulation and climate interact. One example is our inability to provide robust assessments of future global and regional climate changes. However, ongoing advances in our capacity to observe, simulate and conceptualize the climate system now make it possible to fill gaps in our knowledge. We argue that progress can be accelerated by focusing research on a handful of important scientific questions that have become tractable as a result of recent advances. We propose four such questions below; they involve understanding the role of cloud feedbacks and convective organization in climate, and the factors that control the position, the strength and the variability of the tropical rain belts and the extratropical storm tracks.

Two fast radiative transfer methods to improve the temporal sampling of clouds in numerical weather prediction and climate models

The high computational cost of calculating the radiative heating rates in numerical weather prediction (NWP) and climate models requires that calculations are made infrequently, leading to poor sampling of the fast-changing cloud field and a poor representation of the feedback that would occur. This paper presents two related schemes for improving the temporal sampling of the cloud field. Firstly, the ‘split time-stepping’ scheme takes advantage of the independent nature of the monochromatic calculations of the ‘correlated-k’ method to split the calculation into gaseous absorption terms that are highly dependent on changes in cloud (the optically thin terms) and those that are not (optically thick). The small number of optically thin terms can then be calculated more often to capture changes in the grey absorption and scattering associated with cloud droplets and ice crystals. Secondly, the ‘incremental time-stepping’ scheme uses a simple radiative transfer calculation using only one or two monochromatic calculations representing the optically thin part of the atmospheric spectrum. These are found to be sufficient to represent the heating rate increments caused by changes in the cloud field, which can then be added to the last full calculation of the radiation code. We test these schemes in an operational forecast model configuration and find a significant improvement is achieved, for a small computational cost, over the current scheme employed at the Met Office. The ‘incremental time-stepping’ scheme is recommended for operational use, along with a new scheme to correct the surface fluxes for the change in solar zenith angle between radiation calculations.

Clouds, aerosols, and precipitation in the marine boundary layer: an ARM Mobile Facility Deployment

The Clouds, Aerosol, and Precipitation in the Marine Boundary Layer (CAP-MBL) deployment at Graciosa Island in the Azores generated a 21-month (April 2009–December 2010) comprehensive dataset documenting clouds, aerosols, and precipitation using the Atmospheric Radiation Measurement Program (ARM) Mobile Facility (AMF). The scientific aim of the deployment is to gain improved understanding of the interactions of clouds, aerosols, and precipitation in the marine boundary layer. Graciosa Island straddles the boundary between the subtropics and midlatitudes in the northeast Atlantic Ocean and consequently experiences a great diversity of meteorological and cloudiness conditions. Low clouds are the dominant cloud type, with stratocumulus and cumulus occurring regularly. Approximately half of all clouds contained precipitation detectable as radar echoes below the cloud base. Radar and satellite observations show that clouds with tops from 1 to 11 km contribute more or less equally to surface-measured precipitation at Graciosa. A wide range of aerosol conditions was sampled during the deployment consistent with the diversity of sources as indicated by back-trajectory analysis. Preliminary findings suggest important two-way interactions between aerosols and clouds at Graciosa, with aerosols affecting light precipitation and cloud radiative properties while being controlled in part by precipitation scavenging. The data from Graciosa are being compared with short-range forecasts made with a variety of models. A pilot analysis with two climate and two weather forecast models shows that they reproduce the observed time-varying vertical structure of lower-tropospheric cloud fairly well but the cloud-nucleating aerosol concentrations less well. The Graciosa site has been chosen to be a permanent fixed ARM site that became operational in October 2013.

Joint retrievals of cloud and drizzle in marine boundary layer clouds using ground-based radar, lidar and zenith radiances

Active remote sensing of marine boundary-layer clouds is challenging as drizzle drops often dominate the observed radar reflectivity. We present a new method to simultaneously retrieve cloud and drizzle vertical profiles in drizzling boundary-layer clouds using surface-based observations of radar reflectivity, lidar attenuated backscatter, and zenith radiances under conditions when precipitation does not reach the surface. Specifically, the vertical structure of droplet size and water content of both cloud and drizzle is characterised throughout the cloud. An ensemble optimal estimation approach provides full error statistics given the uncertainty in the observations. To evaluate the new method, we first perform retrievals using synthetic measurements from large-eddy simulation snapshots of cumulus under stratocumulus, where cloud water path is retrieved with an error of 31 g m−2 . The method also performs well in non-drizzling clouds where no assumption of the cloud profile is required. We then apply the method to observations of marine stratocumulus obtained during the Atmospheric Radiation Measurement MAGIC deployment in the Northeast Pacific. Here, retrieved cloud water path agrees well with independent three-channel microwave radiometer retrievals, with a root mean square difference of 10–20 g m−2.

Connections between clouds, radiation, and midlatitude dynamics: a review

We review the effects of dynamical variability on clouds and radiation in observations and models and discuss their implications for cloud feedbacks. Jet shifts produce robust meridional dipoles in upper-level clouds and longwave cloud-radiative effect (CRE), but low-level clouds, which do not simply shift with the jet, dominate the shortwave CRE. Because the effect of jet variability on CRE is relatively small, future poleward jet shifts with global warming are only a second-order contribution to the total CRE changes around the midlatitudes, suggesting a dominant role for thermodynamic effects. This implies that constraining the dynamical response is unlikely to reduce the uncertainty in extratropical cloud feedback. However, we argue that uncertainty in the cloud-radiative response does affect the atmospheric circulation response to global warming, by modulating patterns of diabatic forcing. How cloud feedbacks can affect the dynamical response to global warming is an important topic of future research.

Clouds and the atmospheric circulation response to warming

We study the effect of clouds on the atmospheric circulation response to CO2 quadrupling in an aquaplanet model with a slab-ocean lower boundary. The cloud effect is isolated by locking the clouds to either the control or 4xCO2 state in the shortwave (SW) or longwave (LW) radiation schemes. In our model, cloud-radiative changes explain more than half of the total poleward expansion of the Hadley cells, midlatitude jets, and storm tracks under CO2 quadrupling, even though they cause only one-fourth of the total global-mean surface warming. The effect of clouds on circulation results mainly from the SW cloud-radiative changes, which strongly enhance the Equator-to-pole temperature gradient at all levels in the troposphere, favoring stronger and poleward-shifted midlatitude eddies. By contrast, quadrupling CO2 while holding the clouds fixed causes strong polar amplification and weakened midlatitude baroclinicity at lower levels, yielding only a small poleward expansion of the circulation. Our results show that (a) the atmospheric circulation responds sensitively to cloud-driven changes in meridional and vertical temperature distribution, and (b) the spatial structure of cloud feedbacks likely plays a dominant role in the circulation response to greenhouse gas forcing. While the magnitude and spatial structure of the cloud feedback are expected to be highly model-dependent, an analysis of 4xCO2 simulations of CMIP5 models shows that the SW cloud feedback likely forces a poleward expansion of the tropospheric circulation in most climate models.

Simulating the effects of mid- to upper-tropospheric clouds on microwave emissions in EC-Earth using COSP

In this work, the Cloud Feedback Model Intercomparison (CFMIP) Observation Simulation Package (COSP) is expanded to include scattering and emission effects of clouds and precipitation at passive microwave frequencies. This represents an advancement over the official version of COSP (version 1.4.0) in which only clear-sky brightness temperatures are simulated. To highlight the potential utility of this new microwave simulator, COSP results generated using the climate model EC-Earth's version 3 atmosphere as input are compared with Microwave Humidity Sounder (MHS) channel (190.311 GHz) observations. Specifically, simulated seasonal brightness temperatures (TB) are contrasted with MHS observations for the period December 2005 to November 2006 to identify possible biases in EC-Earth's cloud and atmosphere fields. The EC-Earth's atmosphere closely reproduces the microwave signature of many of the major large-scale and regional scale features of the atmosphere and surface. Moreover, greater than 60 % of the simulated TB are within 3 K of the NOAA-18 observations. However, COSP is unable to simulate sufficiently low TB in areas of frequent deep convection. Within the Tropics, the model's atmosphere can yield an underestimation of TB by nearly 30 K for cloudy areas in the ITCZ. Possible reasons for this discrepancy include both incorrect amount of cloud ice water in the model simulations and incorrect ice particle scattering assumptions used in the COSP microwave forward model. These multiple sources of error highlight the non-unique nature of the simulated satellite measurements, a problem exacerbated by the fact that EC-Earth lacks detailed micro-physical parameters necessary for accurate forward model calculations. Such issues limit the robustness of our evaluation and suggest a general note of caution when making COSP-satellite observation evaluations.