Fault tolerant decentralised K-Means clustering for asynchronous large-scale networks

The K-Means algorithm for cluster analysis is one of the most influential and popular data mining methods. Its straightforward parallel formulation is well suited for distributed memory systems with reliable interconnection networks, such as massively parallel processors and clusters of workstations. However, in large-scale geographically distributed systems the straightforward parallel algorithm can be rendered useless by a single communication failure or high latency in communication paths. The lack of scalable and fault tolerant global communication and synchronisation methods in large-scale systems has hindered the adoption of the K-Means algorithm for applications in large networked systems such as wireless sensor networks, peer-to-peer systems and mobile ad hoc networks. This work proposes a fully distributed K-Means algorithm (EpidemicK-Means) which does not require global communication and is intrinsically fault tolerant. The proposed distributed K-Means algorithm provides a clustering solution which can approximate the solution of an ideal centralised algorithm over the aggregated data as closely as desired. A comparative performance analysis is carried out against the state of the art sampling methods and shows that the proposed method overcomes the limitations of the sampling-based approaches for skewed clusters distributions. The experimental analysis confirms that the proposed algorithm is very accurate and fault tolerant under unreliable network conditions (message loss and node failures) and is suitable for asynchronous networks of very large and extreme scale.

The large-scale circulation change at the end of the Baiu season in Japan as seen in ERA40 data

The time evolution of the circulation change at the end of the Baiu season is investigated using ERA40 data. An end-day is defined for each of the 23 years based on the 850 hPa θe value at 40˚Nin the 130-140˚E sector exceeding 330 K. Daily time series of variables are composited with respect to this day. These composite time-series exhibit a clearer and more rapid change in the precipitation and the large-scale circulation over the whole East Asia region than those performed using calendar days. The precipitation change includes the abrupt end of the Baiu rain, the northward shift of tropical convection perhaps starting a few days before this, and the start of the heavier rain at higher latitudes. The northward migration of lower tropospheric warm, moist tropical air, a general feature of the seasonal march in the region, is fast over the continent and slow over the ocean. By mid to late July the cooler air over the Sea of Japan is surrounded on 3 sides by the tropical air. It is suggestive that the large-scale stage has been set for a jump to the post-Baiu state, i.e., for the end of the Baiu season. Two likely triggers for the actual change emerge from the analysis. The first is the northward movement of tropical convection into the Philippine region. The second is an equivalent barotropic Rossby wave-train, that over a 10-day period develops downstream across Eurasia. It appears likely that in most years one or both mechanisms can be important in triggering the actual end of the Baiu season.

The effects of explicit versus parameterized convection on the MJO in a large-domain high-resolution tropical case study. Part I: Characterization of large-scale organization and propagation

High-resolution simulations over a large tropical domain (∼20◦S–20◦N and 42◦E–180◦E) using both explicit and parameterized convection are analyzed and compared to observations during a 10-day case study of an active Madden-Julian Oscillation (MJO) event. The parameterized convection model simulations at both 40 km and 12 km grid spacing have a very weak MJO signal and little eastward propagation. A 4 km explicit convection simulation using Smagorinsky subgrid mixing in the vertical and horizontal dimensions exhibits the best MJO strength and propagation speed. 12 km explicit convection simulations also perform much better than the 12 km parameterized convection run, suggesting that the convection scheme, rather than horizontal resolution, is key for these MJO simulations. Interestingly, a 4 km explicit convection simulation using the conventional boundary layer scheme for vertical subgrid mixing (but still using Smagorinsky horizontal mixing) completely loses the large-scale MJO organization, showing that relatively high resolution with explicit convection does not guarantee a good MJO simulation. Models with a good MJO representation have a more realistic relationship between lower-free-tropospheric moisture and precipitation, supporting the idea that moisture-convection feedback is a key process for MJO propagation. There is also increased generation of available potential energy and conversion of that energy into kinetic energy in models with a more realistic MJO, which is related to larger zonal variance in convective heating and vertical velocity, larger zonal temperature variance around 200 hPa, and larger correlations between temperature and ascent (and between temperature and diabatic heating) between 500–400 hPa.

The detection of atmospheric rivers in atmospheric reanalyses and their links to British winter floods and the large-scale climatic circulation

Atmospheric Rivers (ARs), narrow plumes of enhanced moisture transport in the lower troposphere, are a key synoptic feature behind winter flooding in midlatitude regions. This article develops an algorithm which uses the spatial and temporal extent of the vertically integrated horizontal water vapor transport for the detection of persistent ARs (lasting 18 h or longer) in five atmospheric reanalysis products. Applying the algorithm to the different reanalyses in the vicinity of Great Britain during the winter half-years of 1980–2010 (31 years) demonstrates generally good agreement of AR occurrence between the products. The relationship between persistent AR occurrences and winter floods is demonstrated using winter peaks-over-threshold (POT) floods (with on average one flood peak per winter). In the nine study basins, the number of winter POT-1 floods associated with persistent ARs ranged from approximately 40 to 80%. A Poisson regression model was used to describe the relationship between the number of ARs in the winter half-years and the large-scale climate variability. A significant negative dependence was found between AR totals and the Scandinavian Pattern (SCP), with a greater frequency of ARs associated with lower SCP values.

An evaluation of the spatiotemporal structure of large-scale European drought within the HiGEM climate model

We compare the characteristics of synthetic European droughts generated by the HiGEM1 coupled climate model run with present day atmospheric composition with observed drought events extracted from the CRU TS3 data set. The results demonstrate consistency in both the rate of drought occurrence and the spatiotemporal structure of the events. Estimates of the probability density functions for event area, duration and severity are shown to be similar with confidence > 90%. Encouragingly, HiGEM is shown to replicate the extreme tails of the observed distributions and thus the most damaging European drought events. The soil moisture state is shown to play an important role in drought development. Once a large-scale drought has been initiated it is found to be 50% more likely to continue if the local soil moisture is below the 40th percentile. In response to increased concentrations of atmospheric CO2, the modelled droughts are found to increase in duration, area and severity. The drought response can be largely attributed to temperature driven changes in relative humidity. 1 HiGEM is based on the latest climate configuration of the Met Office Hadley Centre Unified Model (HadGEM1) with the horizontal resolution increased to 1.25 x 0.83 degrees in longitude and latitude in the atmosphere and 1/3 x 1/3 degrees in the ocean.

Examining the large-scale spatial coherence of European drought using regional indicators of precipitation and streamflow deficit

Droughts tend to evolve slowly and affect large areas simultaneously, which suggests that improved understanding of spatial coherence of drought would enable better mitigation of drought impacts through enhanced monitoring and forecasting strategies. This study employs an up-to-date dataset of over 500 river flow time series from 11 European countries, along with a gridded precipitation dataset, to examine the spatial coherence of drought in Europe using regional indicators of precipitation and streamflow deficit. The drought indicators were generated for 24 homogeneous regions and, for selected regions, historical drought characteristics were corroborated with previous work. The spatial coherence of drought characteristics was then examined at a European scale. Historical droughts generally have distinctive signatures in their spatio-temporal development, so there was limited scope for using the evolution of historical events to inform forecasting. Rather, relationships were explored in time series of drought indicators between regions. Correlations were generally low, but multivariate analyses revealed broad continental-scale patterns, which appear to be related to large-scale atmospheric circulation indices (in particular, the North Atlantic Oscillation and the East Atlantic West Russia pattern). A novel methodology for forecasting was developed (and demonstrated with reference to the United Kingdom), which predicts drought from drought i.e. uses spatial coherence of drought to facilitate early warning of drought in a target region, from drought which is developing elsewhere in Europe.Whilst the skill of the methodology is relatively modest at present, this approach presents a potential new avenue for forecasting, which offers significant advantages in that it allows prediction for all seasons, and also shows some potential for forecasting the termination of drought conditions.

Large-scale surface mass balance of ice sheets from a comprehensive atmospheric model

The surface mass balance for Greenland and Antarctica has been calculated using model data from an AMIP-type experiment for the period 1979–2001 using the ECHAM5 spectral transform model at different triangular truncations. There is a significant reduction in the calculated ablation for the highest model resolution, T319 with an equivalent grid distance of ca 40 km. As a consequence the T319 model has a positive surface mass balance for both ice sheets during the period. For Greenland, the models at lower resolution, T106 and T63, on the other hand, have a much stronger ablation leading to a negative surface mass balance. Calculations have also been undertaken for a climate change experiment using the IPCC scenario A1B, with a T213 resolution (corresponding to a grid distance of some 60 km) and comparing two 30-year periods from the end of the twentieth century and the end of the twenty-first century, respectively. For Greenland there is change of 495 km3/year, going from a positive to a negative surface mass balance corresponding to a sea level rise of 1.4 mm/year. For Antarctica there is an increase in the positive surface mass balance of 285 km3/year corresponding to a sea level fall by 0.8 mm/year. The surface mass balance changes of the two ice sheets lead to a sea level rise of 7 cm at the end of this century compared to end of the twentieth century. Other possible mass losses such as due to changes in the calving of icebergs are not considered. It appears that such changes must increase significantly, and several times more than the surface mass balance changes, if the ice sheets are to make a major contribution to sea level rise this century. The model calculations indicate large inter-annual variations in all relevant parameters making it impossible to identify robust trends from the examined periods at the end of the twentieth century. The calculated inter-annual variations are similar in magnitude to observations. The 30-year trend in SMB at the end of the twenty-first century is significant. The increase in precipitation on the ice sheets follows closely the Clausius-Clapeyron relation and is the main reason for the increase in the surface mass balance of Antarctica. On Greenland precipitation in the form of snow is gradually starting to decrease and cannot compensate for the increase in ablation. Another factor is the proportionally higher temperature increase on Greenland leading to a larger ablation. It follows that a modest increase in temperature will not be sufficient to compensate for the increase in accumulation, but this will change when temperature increases go beyond any critical limit. Calculations show that such a limit for Greenland might well be passed during this century. For Antarctica this will take much longer and probably well into following centuries.

Large-scale dynamics of the mesosphere and lower thermosphere: an analysis using the extended Canadian Middle Atmosphere Model

The extended Canadian Middle Atmosphere Model is used to investigate the large-scale dynamics of the mesosphere and lower thermosphere (MLT). It is shown that the 4-day wave is substantially amplified in southern polar winter in the presence of instabilities arising from strong vertical shears in the MLT zonal mean zonal winds brought about by parameterized nonorographic gravity wave drag. A weaker 4-day wave in northern polar winter is attributed to the weaker wind shears that result from weaker parameterized wave drag. The 2-day wave also exhibits a strong dependence on zonal wind shears, in agreement with previous modeling studies. In the equatorial upper mesosphere, the migrating diurnal tide provides most of the resolved westward wave forcing, which varies semiannually in conjunction with the tide itself; resolved forcing by eastward traveling disturbances is dominated by smaller scales. Nonmigrating tides and other planetary-scale waves play only a minor role in the zonal mean zonal momentum budget in the tropics at these heights. Resolved waves are shown to play a significant role in the zonal mean meridional momentum budget in the MLT, impacting significantly on gradient wind balance. Balance fails at low latitudes as a result of a strong Reynolds stress associated with the migrating diurnal tide, an effect which is most pronounced at equinox when the tide is strongest. Resolved and parameterized waves account for most of the imbalance at higher latitudes in summer. This results in the gradient wind underestimating the actual eastward wind reversal by up to 40%.

Consistent large-scale temperature responses in warm and cold climates

Climate-model simulations of the large-scale temperature responses to increased radiative forcing include enhanced land-sea contrast, stronger response at higher latitudes than in the tropics, and differential responsesin warm and cool season climates to uniform forcing. Here we show that these patterns are also characteristic of model simulations of past climates. The differences in the responses over land as opposed to over the ocean, between high and low latitudes, and between summer and winter are remarkably consistent (proportional and nearly linear) across simulations of both cold and warm climates. Similar patterns also appear in historical observations and paleoclimatic reconstructions, implying that such responses are characteristic features of the climate system, and not simple model artifacts, thereby increasing our confidence in the ability of climate models to correctly simulate different climatic states.

A combined statistical and dynamical approach for downscaling large-scale footprints of European windstorms

The occurrence of mid-latitude windstorms is related to strong socio-economic effects. For detailed and reliable regional impact studies, large datasets of high-resolution wind fields are required. In this study, a statistical downscaling approach in combination with dynamical downscaling is introduced to derive storm related gust speeds on a high-resolution grid over Europe. Multiple linear regression models are trained using reanalysis data and wind gusts from regional climate model simulations for a sample of 100 top ranking windstorm events. The method is computationally inexpensive and reproduces individual windstorm footprints adequately. Compared to observations, the results for Germany are at least as good as pure dynamical downscaling. This new tool can be easily applied to large ensembles of general circulation model simulations and thus contribute to a better understanding of the regional impact of windstorms based on decadal and climate change projections.

Large-scale atmospheric dynamics of the wet winter 2009–2010 and its impact on hydrology in Portugal

The anomalously wet winter of 2010 had a very important impact on the Portuguese hydrological system. Owing to the detrimental effects of reduced precipitation in Portugal on the environmental and socio-economic systems, the 2010 winter was predominantly beneficial by reversing the accumulated precipitation deficits during the previous hydrological years. The recorded anomalously high precipitation amounts have contributed to an overall increase in river runoffs and dam recharges in the 4 major river basins. In synoptic terms, the winter 2010 was characterised by an anomalously strong westerly flow component over the North Atlantic that triggered high precipitation amounts. A dynamically coherent enhancement in the frequencies of mid-latitude cyclones close to Portugal, also accompanied by significant increases in the occurrence of cyclonic, south and south-westerly circulation weather types, are noteworthy. Furthermore, the prevalence of the strong negative phase of the North Atlantic Oscillation (NAO) also emphasises the main dynamical features of the 2010 winter. A comparison of the hydrological and atmospheric conditions between the 2010 winter and the previous 2 anomalously wet winters (1996 and 2001) was also carried out to isolate not only their similarities, but also their contrasting conditions, highlighting the limitations of estimating winter precipitation amounts in Portugal using solely the NAO phase as a predictor.

Cyclones causing wind storms in the Mediterranean: characteristics, trends and links to large-scale patterns

A climatology of cyclones with a focus on their relation to wind storm tracks in the Mediterranean region (MR) is presented. Trends in the frequency of cyclones and wind storms, as well as variations associated with the North Atlantic Oscillation (NAO), the East Atlantic/West Russian (EAWR) and the Scandinavian variability pattern (SCAND) are discussed. The study is based on the ERA40 reanalysis dataset. Wind storm tracks are identified by tracking clusters of adjacent grid boxes characterised by extremely high local wind speeds. The wind track is assigned to a cyclone track independently identified with an objective scheme. Areas with high wind activity – quantified by extreme wind tracks – are typically located south of the Golf of Genoa, south of Cyprus, southeast of Sicily and west of the Iberian Peninsula. About 69% of the wind storms are caused by cyclones located in the Mediterranean region, while the remaining 31% can be attributed to North Atlantic or Northern European cyclones. The North Atlantic Oscillation, the East Atlantic/West Russian pattern and the Scandinavian pattern all influence the amount and spatial distribution of wind inducing cyclones and wind events in the MR. The strongest signals exist for the NAO and the EAWR pattern, which are both associated with an increase in the number of organised strong wind events in the eastern MR during their positive phase. On the other hand, the storm numbers decrease over the western MR for the positive phase of the NAO and over the central MR during the positive phase of the EAWR pattern. The positive phase of the Scandinavian pattern is associated with a decrease in the number of winter wind storms over most of the MR. A third of the trends in the number of wind storms and wind producing cyclones during the winter season of the ERA40 period may be attributed to the variability of the North Atlantic Oscillation.

The Empirical Forcing Function as a tool for the diagnosis of large-scale atmospheric anomalies

The time-mean quasi-geostrophic potential vorticity equation of the atmospheric flow on isobaric surfaces can explicitly include an atmospheric (internal) forcing term of the stationary-eddy flow. In fact, neglecting some non-linear terms in this equation, this forcing can be mathematically expressed as a single function, called Empirical Forcing Function (EFF), which is equal to the material derivative of the time-mean potential vorticity. Furthermore, the EFF can be decomposed as a sum of seven components, each one representing a forcing mechanism of different nature. These mechanisms include diabatic components associated with the radiative forcing, latent heat release and frictional dissipation, and components related to transient eddy transports of heat and momentum. All these factors quantify the role of the transient eddies in forcing the atmospheric circulation. In order to assess the relevance of the EFF in diagnosing large-scale anomalies in the atmospheric circulation, the relationship between the EFF and the occurrence of strong North Atlantic ridges over the Eastern North Atlantic is analyzed, which are often precursors of severe droughts over Western Iberia. For such events, the EFF pattern depicts a clear dipolar structure over the North Atlantic; cyclonic (anticyclonic) forcing of potential vorticity is found upstream (downstream) of the anomalously strong ridges. Results also show that the most significant components are related to the diabatic processes. Lastly, these results highlight the relevance of the EFF in diagnosing large-scale anomalies, also providing some insight into their interaction with different physical mechanisms.