Economic models and algorithms for distributed systems

Distributed computing paradigms for sharing resources such as Clouds, Grids, Peer-to-Peer systems, or voluntary computing are becoming increasingly popular. While there are some success stories such as PlanetLab, OneLab, BOINC, BitTorrent, and SETI@home, a widespread use of these technologies for business applications has not yet been achieved. In a business environment, mechanisms are needed to provide incentives to potential users for participating in such networks. These mechanisms may range from simple non-monetary access rights, monetary payments to specific policies for sharing. Although a few models for a framework have been discussed (in the general area of a "Grid Economy"), none of these models has yet been realised in practice. This book attempts to fill this gap by discussing the reasons for such limited take-up and exploring incentive mechanisms for resource sharing in distributed systems. The purpose of this book is to identify research challenges in successfully using and deploying resource sharing strategies in open-source and commercial distributed systems.

Case study: object and ground point separation on a river bank using airborne based LIDAR data

In the past decade, airborne based LIght Detection And Ranging (LIDAR) has been recognised by both the commercial and public sectors as a reliable and accurate source for land surveying in environmental, engineering and civil applications. Commonly, the first task to investigate LIDAR point clouds is to separate ground and object points. Skewness Balancing has been proven to be an efficient non-parametric unsupervised classification algorithm to address this challenge. Initially developed for moderate terrain, this algorithm needs to be adapted to handle sloped terrain. This paper addresses the difficulty of object and ground point separation in LIDAR data in hilly terrain. A case study on a diverse LIDAR data set in terms of data provider, resolution and LIDAR echo has been carried out. Several sites in urban and rural areas with man-made structure and vegetation in moderate and hilly terrain have been investigated and three categories have been identified. A deeper investigation on an urban scene with a river bank has been selected to extend the existing algorithm. The results show that an iterative use of Skewness Balancing is suitable for sloped terrain.

Multiple-scattering in radar systems: a review

Although extensively studied within the lidar community, the multiple scattering phenomenon has always been considered a rare curiosity by radar meteorologists. Up to few years ago its appearance has only been associated with two- or three-body-scattering features (e.g. hail flares and mirror images) involving highly reflective surfaces. Recent atmospheric research aimed at better understanding of the water cycle and the role played by clouds and precipitation in affecting the Earth's climate has driven the deployment of high frequency radars in space. Examples are the TRMM 13.5 GHz, the CloudSat 94 GHz, the upcoming EarthCARE 94 GHz, and the GPM dual 13-35 GHz radars. These systems are able to detect the vertical distribution of hydrometeors and thus provide crucial feedbacks for radiation and climate studies. The shift towards higher frequencies increases the sensitivity to hydrometeors, improves the spatial resolution and reduces the size and weight of the radar systems. On the other hand, higher frequency radars are affected by stronger extinction, especially in the presence of large precipitating particles (e.g. raindrops or hail particles), which may eventually drive the signal below the minimum detection threshold. In such circumstances the interpretation of the radar equation via the single scattering approximation may be problematic. Errors will be large when the radiation emitted from the radar after interacting more than once with the medium still contributes substantially to the received power. This is the case if the transport mean-free-path becomes comparable with the instrument footprint (determined by the antenna beam-width and the platform altitude). This situation resembles to what has already been experienced in lidar observations, but with a predominance of wide- versus small-angle scattering events. At millimeter wavelengths, hydrometeors diffuse radiation rather isotropically compared to the visible or near infrared region where scattering is predominantly in the forward direction. A complete understanding of radiation transport modeling and data analysis methods under wide-angle multiple scattering conditions is mandatory for a correct interpretation of echoes observed by space-borne millimeter radars. This paper reviews the status of research in this field. Different numerical techniques currently implemented to account for higher order scattering are reviewed and their weaknesses and strengths highlighted. Examples of simulated radar backscattering profiles are provided with particular emphasis given to situations in which the multiple scattering contributions become comparable or overwhelm the single scattering signal. We show evidences of multiple scattering effects from air-borne and from CloudSat observations, i.e. unique signatures which cannot be explained by single scattering theory. Ideas how to identify and tackle the multiple scattering effects are discussed. Finally perspectives and suggestions for future work are outlined. This work represents a reference-guide for studies focused at modeling the radiation transport and at interpreting data from high frequency space-borne radar systems that probe highly opaque scattering media such as thick ice clouds or precipitating clouds.

Effect of improving representation of horizontal and vertical cloud structure on the Earth's global radiation budget. Part I: review and parametrization

A poor representation of cloud structure in a general circulation model (GCM) is widely recognised as a potential source of error in the radiation budget. Here, we develop a new way of representing both horizontal and vertical cloud structure in a radiation scheme. This combines the ‘Tripleclouds’ parametrization, which introduces inhomogeneity by using two cloudy regions in each layer as opposed to one, each with different water content values, with ‘exponential-random’ overlap, in which clouds in adjacent layers are not overlapped maximally, but according to a vertical decorrelation scale. This paper, Part I of two, aims to parametrize the two effects such that they can be used in a GCM. To achieve this, we first review a number of studies for a globally applicable value of fractional standard deviation of water content for use in Tripleclouds. We obtain a value of 0.75 ± 0.18 from a variety of different types of observations, with no apparent dependence on cloud type or gridbox size. Then, through a second short review, we create a parametrization of decorrelation scale for use in exponential-random overlap, which varies the scale linearly with latitude from 2.9 km at the Equator to 0.4 km at the poles. When applied to radar data, both components are found to have radiative impacts capable of offsetting biases caused by cloud misrepresentation. Part II of this paper implements Tripleclouds and exponential-random overlap into a radiation code and examines both their individual and combined impacts on the global radiation budget using re-analysis data.

Mid-nineteenth century smoke concentrations near London

It has been proposed that Earth's climate could be affected by changes in cloudiness caused by variations in the intensity of galactic cosmic rays in the atmosphere. This proposal stems from an observed correlation between cosmic ray intensity and Earth's average cloud cover over the course of one solar cycle. Some scientists question the reliability of the observations, whereas others, who accept them as reliable, suggest that the correlation may be caused by other physical phenomena with decadal periods or by a response to volcanic activity or El Niño. Nevertheless, the observation has raised the intriguing possibility that a cosmic ray–cloud interaction may help explain how a relatively small change in solar output can produce much larger changes in Earth's climate. Physical mechanisms have been proposed to explain how cosmic rays could affect clouds, but they need to be investigated further if the observation is to become more than just another correlation among geophysical variables.

The global atmospheric electrical circuit and climate

Evidence is emerging for physical links among clouds, global temperatures, the global atmospheric electrical circuit and cosmic ray ionisation. The global circuit extends throughout the atmosphere from the planetary surface to the lower layers of the ionosphere. Cosmic rays are the principal source of atmospheric ions away from the continental boundary layer: the ions formed permit a vertical conduction current to flow in the fair weather part of the global circuit. Through the (inverse) solar modulation of cosmic rays, the resulting columnar ionisation changes may allow the global circuit to convey a solar influence to meteorological phenomena of the lower atmosphere. Electrical effects on non-thunderstorm clouds have been proposed to occur via the ion-assisted formation of ultra-fine aerosol, which can grow to sizes able to act as cloud condensation nuclei, or through the increased ice nucleation capability of charged aerosols. Even small atmospheric electrical modulations on the aerosol size distribution can affect cloud properties and modify the radiative balance of the atmosphere, through changes communicated globally by the atmospheric electrical circuit. Despite a long history of work in related areas of geophysics, the direct and inverse relationships between the global circuit and global climate remain largely quantitatively unexplored. From reviewing atmospheric electrical measurements made over two centuries and possible paleoclimate proxies, global atmospheric electrical circuit variability should be expected on many timescales

Evaluating forecasts of the evolution of the cloudy boundary layer using diurnal composites of radar and lidar observations

Observations of boundary-layer cloud have been made using radar and lidar at Chilbolton, Hampshire, UK. These have been compared with output from 7 different global and regional models. Fifty-five cloudy days have been composited to reveal the mean diurnal variation of cloud top and base heights, cloud thickness and liquid water path of the clouds. To enable like-for-like comparison between model and observations, the observations have been averaged on to the grid of each model. The composites show a distinct diurnal cycle in observed cloud; the cloud height exhibits a sinusoidal variation throughout the day with a maximum at around 1600 and a minimum at around 0700 UTC. This diurnal cycle is captured by six of the seven models analysed, although the models generally under-predict both cloud top and cloud base heights throughout the day. The two worst performing models in terms of cloud boundaries also have biases of around a factor of two in liquid water path; these were the only two models that did not include an explicit formulation for cloud-top entrainment.

Verification of cloud-fraction forecasts

Cloud radar and lidar can be used to evaluate the skill of numerical weather prediction models in forecasting the timing and placement of clouds, but care must be taken in choosing the appropriate metric of skill to use due to the non- Gaussian nature of cloud-fraction distributions. We compare the properties of a number of different verification measures and conclude that of existing measures the Log of Odds Ratio is the most suitable for cloud fraction. We also propose a new measure, the Symmetric Extreme Dependency Score, which has very attractive properties, being equitable (for large samples), difficult to hedge and independent of the frequency of occurrence of the quantity being verified. We then use data from five European ground-based sites and seven forecast models, processed using the ‘Cloudnet’ analysis system, to investigate the dependence of forecast skill on cloud fraction threshold (for binary skill scores), height, horizontal scale and (for the Met Office and German Weather Service models) forecast lead time. The models are found to be least skillful at predicting the timing and placement of boundary-layer clouds and most skilful at predicting mid-level clouds, although in the latter case they tend to underestimate mean cloud fraction when cloud is present. It is found that skill decreases approximately inverse-exponentially with forecast lead time, enabling a forecast ‘half-life’ to be estimated. When considering the skill of instantaneous model snapshots, we find typical values ranging between 2.5 and 4.5 days. Copyright c 2009 Royal Meteorological Society

Cloud optical depth retrievals from the Aerosol Robotic Network (AERONET) cloud mode observations

Cloud optical depth is one of the most poorly observed climate variables. The new “cloud mode” capability in the Aerosol Robotic Network (AERONET) will inexpensively yet dramatically increase cloud optical depth observations in both number and accuracy. Cloud mode optical depth retrievals from AERONET were evaluated at the Atmospheric Radiation Measurement program’s Oklahoma site in sky conditions ranging from broken clouds to overcast. For overcast cases, the 1.5 min average AERONET cloud mode optical depths agreed to within 15% of those from a standard ground‐based flux method. For broken cloud cases, AERONET retrievals also captured rapid variations detected by the microwave radiometer. For 3 year climatology derived from all nonprecipitating clouds, AERONET monthly mean cloud optical depths are generally larger than cloud radar retrievals because of the current cloud mode observation strategy that is biased toward measurements of optically thick clouds. This study has demonstrated a new way to enhance the existing AERONET infrastructure to observe cloud optical properties on a global scale.

Spectrally-invariant behavior of zenith radiance around cloud edges simulated by radiative transfer

In a previous paper, we discovered a surprising spectrally-invariant relationship in shortwave spectrometer observations taken by the Atmospheric Radiation Measurement (ARM) program. The relationship suggests that the shortwave spectrum near cloud edges can be determined by a linear combination of zenith radiance spectra of the cloudy and clear regions. Here, using radiative transfer simulations, we study the sensitivity of this relationship to the properties of aerosols and clouds, to the underlying surface type, and to the finite field-of-view (FOV) of the spectrometer. Overall, the relationship is mostly sensitive to cloud properties and has little sensitivity to other factors. At visible wavelengths, the relationship primarily depends on cloud optical depth regardless of cloud phase function, thermodynamic phase and drop size. At water-absorbing wavelengths, the slope of the relationship depends primarily on cloud optical depth; the intercept, by contrast, depends primarily on cloud absorbing and scattering properties, suggesting a new retrieval method for cloud drop effective radius. These results suggest that the spectrally-invariant relationship can be used to infer cloud properties near cloud edges even with insufficient or no knowledge about spectral surface albedo and aerosol properties.

Retrievals of thick cloud optical depth from the Geoscience Laser Altimeter System (GLAS) by calibration of solar background signal

Laser beams emitted from the Geoscience Laser Altimeter System (GLAS), as well as other spaceborne laser instruments, can only penetrate clouds to a limit of a few optical depths. As a result, only optical depths of thinner clouds (< about 3 for GLAS) are retrieved from the reflected lidar signal. This paper presents a comprehensive study of possible retrievals of optical depth of thick clouds using solar background light and treating GLAS as a solar radiometer. To do so one must first calibrate the reflected solar radiation received by the photon-counting detectors of the GLAS 532-nm channel, the primary channel for atmospheric products. Solar background radiation is regarded as a noise to be subtracted in the retrieval process of the lidar products. However, once calibrated, it becomes a signal that can be used in studying the properties of optically thick clouds. In this paper, three calibration methods are presented: (i) calibration with coincident airborne and GLAS observations, (ii) calibration with coincident Geostationary Opera- tional Environmental Satellite (GOES) and GLAS observations of deep convective clouds, and (iii) cali- bration from first principles using optical depth of thin water clouds over ocean retrieved by GLAS active remote sensing. Results from the three methods agree well with each other. Cloud optical depth (COD) is retrieved from the calibrated solar background signal using a one-channel retrieval. Comparison with COD retrieved from GOES during GLAS overpasses shows that the average difference between the two retriev- als is 24%. As an example, the COD values retrieved from GLAS solar background are illustrated for a marine stratocumulus cloud field that is too thick to be penetrated by the GLAS laser. Based on this study, optical depths for thick clouds will be provided as a supplementary product to the existing operational GLAS cloud products in future GLAS data releases.

Cloud optical depth retrievals from solar background "signals" of micropulse lidars

Pulsed lidars are commonly used to retrieve vertical distributions of cloud and aerosol layers. It is widely believed that lidar cloud retrievals (other than cloud base altitude) are limited to optically thin clouds. Here, we demonstrate that lidars can retrieve optical depths of thick clouds using solar background light as a signal, rather than (as now) merely a noise to be subtracted. Validations against other instruments show that retrieved cloud optical depths agree within 10%–15% for overcast stratus and broken clouds. In fact, for broken cloud situations, one can retrieve not only the aerosol properties in clear-sky periods using lidar signals, but also the optical depth of thick clouds in cloudy periods using solar background signals. This indicates that, in general, it may be possible to retrieve both aerosol and cloud properties using a single lidar. Thus, lidar observations have great untapped potential to study interactions between clouds and aerosols.

Remote sensing of cloud properties using ground-based measurements of zenith radiance

We have conducted the first extensive field test of two new methods to retrieve optical properties for overhead clouds that range from patchy to overcast. The methods use measurements of zenith radiance at 673 and 870 nm wavelengths and require the presence of green vegetation in the surrounding area. The test was conducted at the Atmospheric Radiation Measurement Program Oklahoma site during September–November 2004. These methods work because at 673 nm (red) and 870 nm (near infrared (NIR)), clouds have nearly identical optical properties, while vegetated surfaces reflect quite differently. The first method, dubbed REDvsNIR, retrieves not only cloud optical depth τ but also radiative cloud fraction. Because of the 1-s time resolution of our radiance measurements, we are able for the first time to capture changes in cloud optical properties at the natural timescale of cloud evolution. We compared values of τ retrieved by REDvsNIR to those retrieved from downward shortwave fluxes and from microwave brightness temperatures. The flux method generally underestimates τ relative to the REDvsNIR method. Even for overcast but inhomogeneous clouds, differences between REDvsNIR and the flux method can be as large as 50%. In addition, REDvsNIR agreed to better than 15% with the microwave method for both overcast and broken clouds. The second method, dubbed COUPLED, retrieves τ by combining zenith radiances with fluxes. While extra information from fluxes was expected to improve retrievals, this is not always the case. In general, however, the COUPLED and REDvsNIR methods retrieve τ to within 15% of each other.

Using Doppler radar with a simple explicit microphysics model to diagnose problems with ice sublimation depth-scales in forecast models

Several previous studies have attempted to assess the sublimation depth-scales of ice particles from clouds into clear air. Upon examining the sublimation depth-scales in the Met Office Unified Model (MetUM), it was found that the MetUM has evaporation depth-scales 2–3 times larger than radar observations. Similar results can be seen in the European Centre for Medium-Range Weather Forecasts (ECMWF), Regional Atmospheric Climate Model (RACMO) and Météo-France models. In this study, we use radar simulation (converting model variables into radar observations) and one-dimensional explicit microphysics numerical modelling to test and diagnose the cause of the deep sublimation depth-scales in the forecast model. The MetUM data and parametrization scheme are used to predict terminal velocity, which can be compared with the observed Doppler velocity. This can then be used to test the hypothesis as to why the sublimation depth-scale is too large within the MetUM. Turbulence could lead to dry air entrainment and higher evaporation rates; particle density may be wrong, particle capacitance may be too high and lead to incorrect evaporation rates or the humidity within the sublimating layer may be incorrectly represented. We show that the most likely cause of deep sublimation zones is an incorrect representation of model humidity in the layer. This is tested further by using a one-dimensional explicit microphysics model, which tests the sensitivity of ice sublimation to key atmospheric variables and is capable of including sonde and radar measurements to simulate real cases. Results suggest that the MetUM grid resolution at ice cloud altitudes is not sufficient enough to maintain the sharp drop in humidity that is observed in the sublimation zone.

Multicore-enabling the MPJ express messaging library

With the transition to multicore processors almost complete, the parallel processing community is seeking efficient ways to port legacy message passing applications on shared memory and multicore processors. MPJ Express is our reference implementation of Message Passing Interface (MPI)-like bindings for the Java language. Starting with the current release, the MPJ Express software can be configured in two modes: the multicore and the cluster mode. In the multicore mode, parallel Java applications execute on shared memory or multicore processors. In the cluster mode, Java applications parallelized using MPJ Express can be executed on distributed memory platforms like compute clusters and clouds. The multicore device has been implemented using Java threads in order to satisfy two main design goals of portability and performance. We also discuss the challenges of integrating the multicore device in the MPJ Express software. This turned out to be a challenging task because the parallel application executes in a single JVM in the multicore mode. On the contrary in the cluster mode, the parallel user application executes in multiple JVMs. Due to these inherent architectural differences between the two modes, the MPJ Express runtime is modified to ensure correct semantics of the parallel program. Towards the end, we compare performance of MPJ Express (multicore mode) with other C and Java message passing libraries---including mpiJava, MPJ/Ibis, MPICH2, MPJ Express (cluster mode)---on shared memory and multicore processors. We found out that MPJ Express performs signicantly better in the multicore mode than in the cluster mode. Not only this but the MPJ Express software also performs better in comparison to other Java messaging libraries including mpiJava and MPJ/Ibis when used in the multicore mode on shared memory or multicore processors. We also demonstrate effectiveness of the MPJ Express multicore device in Gadget-2, which is a massively parallel astrophysics N-body siimulation code.