Scaling up data mining techniques to large datasets using parallel and distributed processing

Advances in hardware and software technology enable us to collect, store and distribute large quantities of data on a very large scale. Automatically discovering and extracting hidden knowledge in the form of patterns from these large data volumes is known as data mining. Data mining technology is not only a part of business intelligence, but is also used in many other application areas such as research, marketing and financial analytics. For example medical scientists can use patterns extracted from historic patient data in order to determine if a new patient is likely to respond positively to a particular treatment or not; marketing analysts can use extracted patterns from customer data for future advertisement campaigns; finance experts have an interest in patterns that forecast the development of certain stock market shares for investment recommendations. However, extracting knowledge in the form of patterns from massive data volumes imposes a number of computational challenges in terms of processing time, memory, bandwidth and power consumption. These challenges have led to the development of parallel and distributed data analysis approaches and the utilisation of Grid and Cloud computing. This chapter gives an overview of parallel and distributed computing approaches and how they can be used to scale up data mining to large datasets.

Power allocation strategies for distributed space-time codes in two-way relay networks

We study a two-way relay network (TWRN), where distributed space-time codes are constructed across multiple relay terminals in an amplify-and-forward mode. Each relay transmits a scaled linear combination of its received symbols and their conjugates,with the scaling factor chosen based on automatic gain control. We consider equal power allocation (EPA) across the relays, as well as the optimal power allocation (OPA) strategy given access to instantaneous channel state information (CSI). For EPA, we derive an upper bound on the pairwise-error-probability (PEP), from which we prove that full diversity is achieved in TWRNs. This result is in contrast to one-way relay networks, in which case a maximum diversity order of only unity can be obtained. When instantaneous CSI is available at the relays, we show that the OPA which minimizes the conditional PEP of the worse link can be cast as a generalized linear fractional program, which can be solved efficiently using the Dinkelback-type procedure.We also prove that, if the sum-power of the relay terminals is constrained, then the OPA will activate at most two relays.

The effect of nonlinearity in CO2 heating rates on the attribution of stratospheric ozone and temperature changes

An analysis of the attribution of past and future changes in stratospheric ozone and temperature to anthropogenic forcings is presented. The analysis is an extension of the study of Shepherd and Jonsson (2008) who analyzed chemistry-climate simulations from the Canadian Middle Atmosphere Model (CMAM) and attributed both past and future changes to changes in the external forcings, i.e. the abundances of ozone-depleting substances (ODS) and well-mixed greenhouse gases. The current study is based on a new CMAM dataset and includes two important changes. First, we account for the nonlinear radiative response to changes in CO2. It is shown that over centennial time scales the radiative response in the upper stratosphere to CO2 changes is significantly nonlinear and that failure to account for this effect leads to a significant error in the attribution. To our knowledge this nonlinearity has not been considered before in attribution analysis, including multiple linear regression studies. For the regression analysis presented here the nonlinearity was taken into account by using CO2 heating rate, rather than CO2 abundance, as the explanatory variable. This approach yields considerable corrections to the results of the previous study and can be recommended to other researchers. Second, an error in the way the CO2 forcing changes are implemented in the CMAM was corrected, which significantly affects the results for the recent past. As the radiation scheme, based on Fomichev et al. (1998), is used in several other models we provide some description of the problem and how it was fixed.

On the tropospheric response to anomalous stratospheric wave drag and radiative heating

Observational and numerical evidence suggest that variability in the extratropical stratospheric circulation has a demonstrable impact on tropospheric variability on intraseasonal time scales. In this study, it is demonstrated that the amplitude of the observed tropospheric response to vacillations in the stratospheric flow is quantitatively similar to the zonal-mean balanced response to the anomalous wave forcing at stratospheric levels. It is further demonstrated that the persistence of the tropospheric response is consistent with the impact of anomalous diabatic heating in the polar stratosphere as stratospheric temperatures relax to climatology. The results contradict previous studies that suggest that variations in stratospheric wave drag are too weak to account for the attendant changes in the tropospheric flow. However, the results also reveal that stratospheric processes alone cannot account for the observed meridional redistribution of momentum within the troposphere.

Using paint to investigate fires: an ATR-IR study of the degradation of paint samples upon heating

Fire investigation is a challenging area for the forensic investigator. The aim of this work was to use spectral changes to paint samples to estimate the temperatures to which a paint has been heated. Five paint samples (one clay paint, two car paints, one metallic paint, and one matt emulsion) have been fully characterized by a combination of attenuated total reflectance Fourier transform infrared (ATR-IR), Raman, X-ray fluorescence spectroscopy and powder X-ray diffraction. The thermal decomposition of these paints has been investigated by means of ATR-IR and thermal gravimetric analysis. Clear temperature markers are observed in the ATR-IR spectra namely: loss of m(C = O) band, >300°C; appearance of water bands on cooling, >500°C; alterations to m(Si–O) bands due to dehydration of silicate clays, >700°C; diminution of m(CO3) and d(CO3) modes of CaCO3, >950°C. We suggest the possible use of portable ATR-IR for nondestructive, in situ analysis of paints.

A generalised Bayesian instrumental variable approach under student t-distributed errors with application

Bayesian analysis is given of an instrumental variable model that allows for heteroscedasticity in both the structural equation and the instrument equation. Specifically, the approach for dealing with heteroscedastic errors in Geweke (1993) is extended to the Bayesian instrumental variable estimator outlined in Rossi et al. (2005). Heteroscedasticity is treated by modelling the variance for each error using a hierarchical prior that is Gamma distributed. The computation is carried out by using a Markov chain Monte Carlo sampling algorithm with an augmented draw for the heteroscedastic case. An example using real data illustrates the approach and shows that ignoring heteroscedasticity in the instrument equation when it exists may lead to biased estimates.

A semi-distributed ntegrated itrogen model for multiple source assessment in tchments (INCA): Part I — model structure and process equations

A new model has been developed for assessing multiple sources of nitrogen in catchments. The model (INCA) is process based and uses reaction kinetic equations to simulate the principal mechanisms operating. The model allows for plant uptake, surface and sub-surface pathways and can simulate up to six land uses simultaneously. The model can be applied to catchment as a semi-distributed simulation and has an inbuilt multi-reach structure for river systems. Sources of nitrogen can be from atmospheric deposition, from the terrestrial environment (e.g. agriculture, leakage from forest systems etc.), from urban areas or from direct discharges via sewage or intensive farm units. The model is a daily simulation model and can provide information in the form of time series at key sites, or as profiles down river systems or as statistical distributions. The process model is described and in a companion paper the model is applied to the River Tywi catchment in South Wales and the Great Ouse in Bedfordshire.

Dynamic distributed lanes: motion planning for multiple autonomous vehicles

Unorganized traffic is a generalized form of travel wherein vehicles do not adhere to any predefined lanes and can travel in-between lanes. Such travel is visible in a number of countries e.g. India, wherein it enables a higher traffic bandwidth, more overtaking and more efficient travel. These advantages are visible when the vehicles vary considerably in size and speed, in the absence of which the predefined lanes are near-optimal. Motion planning for multiple autonomous vehicles in unorganized traffic deals with deciding on the manner in which every vehicle travels, ensuring no collision either with each other or with static obstacles. In this paper the notion of predefined lanes is generalized to model unorganized travel for the purpose of planning vehicles travel. A uniform cost search is used for finding the optimal motion strategy of a vehicle, amidst the known travel plans of the other vehicles. The aim is to maximize the separation between the vehicles and static obstacles. The search is responsible for defining an optimal lane distribution among vehicles in the planning scenario. Clothoid curves are used for maintaining a lane or changing lanes. Experiments are performed by simulation over a set of challenging scenarios with a complex grid of obstacles. Additionally behaviours of overtaking, waiting for a vehicle to cross and following another vehicle are exhibited.

The effect of chosen extraterrestrial solar spectrum on clear-sky atmospheric absorption and heating rates in the near infrared

The extraterrestrial solar spectrum (ESS) is an important component in near infrared (near-IR) radiative transfer calculations. However, the impact of a particular choice of the ESS in these regions has been given very little attention. A line-by-line (LBL) transfer model has been used to calculate the absorbed solar irradiance and solar heating rates in the near-IR from 2000-10000 cm−1(1-5 μm) using different ESS. For overhead sun conditions in a mid-latitude summer atmosphere, the absorbed irradiances could differ by up to about 11 Wm−2 (8.2%) while the tropospheric and stratospheric heating rates could differ by up to about 0.13 K day−1 (8.1%) and 0.19 K day−1 (7.6%). The spectral shape of the ESS also has a small but non-negligible impact on these factors in the near-IR.

Using standby generators in support of an electricity network: logistics and experience

Electricity load shifting is becoming a big topic in the world of ‘green’ retail. Marks & Spencer (M&S) aim to become the world’s most sustainable retailer (1) and part of that commitment means contributing to the future electricity network. While intelligent operation of fridges and Heating, Ventilation and Air Conditioning (HVAC) systems are a wide area of research, standby generators should be considered too, as they are the most widely adopted form of distributed generation. In this paper, the experience of using standby generators in Northern Ireland to support the grid is shared and the logistics of future projects are discussed. Interactions with maintenance schedules, electricity costs, grid code, staffing and store opening times are discussed as well as the financial implications associated with running generators for grid support.

Calibration of structure in a distributed forecasting model for a semiarid flash flood: dynamic surface storage and channel roughness

Flash floods pose a significant danger for life and property. Unfortunately, in arid and semiarid environment the runoff generation shows a complex non-linear behavior with a strong spatial and temporal non-uniformity. As a result, the predictions made by physically-based simulations in semiarid areas are subject to great uncertainty, and a failure in the predictive behavior of existing models is common. Thus better descriptions of physical processes at the watershed scale need to be incorporated into the hydrological model structures. For example, terrain relief has been systematically considered static in flood modelling at the watershed scale. Here, we show that the integrated effect of small distributed relief variations originated through concurrent hydrological processes within a storm event was significant on the watershed scale hydrograph. We model these observations by introducing dynamic formulations of two relief-related parameters at diverse scales: maximum depression storage, and roughness coefficient in channels. In the final (a posteriori) model structure these parameters are allowed to be both time-constant or time-varying. The case under study is a convective storm in a semiarid Mediterranean watershed with ephemeral channels and high agricultural pressures (the Rambla del Albujón watershed; 556 km 2 ), which showed a complex multi-peak response. First, to obtain quasi-sensible simulations in the (a priori) model with time-constant relief-related parameters, a spatially distributed parameterization was strictly required. Second, a generalized likelihood uncertainty estimation (GLUE) inference applied to the improved model structure, and conditioned to observed nested hydrographs, showed that accounting for dynamic relief-related parameters led to improved simulations. The discussion is finally broadened by considering the use of the calibrated model both to analyze the sensitivity of the watershed to storm motion and to attempt the flood forecasting of a stratiform event with highly different behavior.

Ionospheric ion and electron heating at the poleward boundary of a poleward expanding substorm-disturbed region

We present observations of a poleward propagating substorm-disturbed region which was observed by the European Incoherent SCATter (EISCAT) radar and the Svalbard International Monitor for Auroral Geomagnetic Effects (IMAGE) magnetometers in the postmidnight sector. The expansion of the disturbance was launched by a substorm intensification which started similar to 25 min after the initial onset, and similar to 10 min before the disturbance arrived over Svalbard. In association with the magnetic disturbance, a poleward expanding enduring enhancement in the F region electron temperature was observed, indicative of soft electron precipitation, with a narrow band of enhanced ion temperature straddling its poleward edge, indicative of fast ion flows and ion-neutral collisional heating. This electron temperature boundary was coincident with the poleward propagating electrojet current system detected by the high-latitude IMAGE magnetometer stations and is taken to be a proxy for the observation of a substorm auroral bulge. The electron temperature boundary is inferred to have a width comparable or less than one radar range gate (similar to 60 km transverse to the magnetic field), while the region of high ion temperature was found to be approximately three gates wide, extending approximately two gates (similar to 120 km) poleward of the electron temperature boundary, and approximately one gate (similar to 60 km) equatorward. The two-beam radar line-of-sight velocity data are found to be consistent with the existence of a layer of high-speed flow in the boundary, peaking at values similar to1.5-3 km s(-1), roughly consistent with the ion temperature data. The flow is directed either east or west along the boundary depending on the direction of the flow in the poleward region. We infer that the flow is deflected along and around the boundary of the substorm-disturbed region due to the high conductivity of the latter. Variations in the flow poleward of the boundary produced no discernible magnetic effects on the ground, confirming the low conductivity of the preboundary ionosphere.

Ion flows and heating at a contracting polar-cap boundary: GISMOS observations indicating viscous-like interaction on the flanks of the magnetotail

This paper complements that in this issue by Clauer et al. concerning the international GISMOS campaign of 3–5 June 1987. From a detailed study of the EISCAT data, the polar-cap boundary, as defined by an almost shear east-west convection reversal, is found to contract across the EISCAT field of view between 04 and 07 MLT. An annulus of enhanced ion temperature and non-thermal plasma is observed immediately equatorward of the contracting boundary due to the lag in the response of the neutral-wind pattern to the change in ion flows. The ion flow inside the polar cap and at the boundary is shown to be relatively smooth, compared with that in the auroral oval, at 15-second resolution. The flow at the boundary is directed poleward, with velocities which exceed that of the boundary itself. The effect of velocity shears on the beamswinging technique used to derive the ion flows has been analysed in detail and it is found that spurious flows across a moving boundary can be generated. However, these are much smaller than the observed flows into the polar cap and cannot explain the 7 kV potential difference across the observed segment of the cap boundary between 04:30–06:30 UT. The ion temperature enhancements at the two observing azimuths is used to define the boundary orientation. The results are consistent with recent observations of slow anti-sunward flow of closed field lines on the flanks of the geomagnetic tail, which appears to be generated by some form of “viscous” coupling to the magnetosheath plasma.

Ion flows and heating at a contracting polar-cap boundary

Data recorded by the POLAR experiment run on the EISCAT radar during the international GISMOS campaign of 3–5 June 1987 are studied in detail. The polar-cap boundary, as denned by an almost shear East-West convection reversal, was observed to jump southward across the EISCAT field of view in two steps at 02:00 and 03:00 Magnetic Local Time and subsequently to contract back between 04:00 and 07:00 M.L.T. An annulus of enhanced ion temperature and non-thermal plasma was observed immediately equatorward of the contracting boundary due to the lag in the response of the neutral-wind pattern to the change in ion flows. The ion flow at the boundary is shown to be relatively smooth at 15 s resolution and directed northward, with velocities which exceed that of the boundary itself. The effect of velocity shears on the beamswinging technique used to derive the ion flows is analyzed in detail and it is shown that, for certain orientations of the cap boundary, spurious flows into the cap can be generated. However, these are much smaller than the observed flows into the polar cap and cannot explain the potential difference across the observed segment of the cap boundary (extending over 2 h of M.L.T.) which is roughly 7 kV. Similarly, an observed slowing of the zonal flow near the boundary cannot be explained as an error introduced by the use of the beamswinging technique. The results could be interpreted as being due to reconnection occurring on the dawn flank of the magnetopause (mapping to the polar cap at 04:30 06:30 M.L.T.). However, they are more consistent with recent observations of slow anti-sunward flow of closed field lines on the flanks of the geomagnetic tail, which appears to be generated by some form of “viscous” coupling to the magnetosheath plasma.