Storing and manipulating environmental big data with JASMIN

JASMIN is a super-data-cluster designed to provide a high-performance high-volume data analysis environment for the UK environmental science community. Thus far JASMIN has been used primarily by the atmospheric science and earth observation communities, both to support their direct scientific workflow, and the curation of data products in the STFC Centre for Environmental Data Archival (CEDA). Initial JASMIN configuration and first experiences are reported here. Useful improvements in scientific workflow are presented. It is clear from the explosive growth in stored data and use that there was a pent up demand for a suitable big-data analysis environment. This demand is not yet satisfied, in part because JASMIN does not yet have enough compute, the storage is fully allocated, and not all software needs are met. Plans to address these constraints are introduced.

An exploration of the equivalent weights particle filter

Particle filters are fully non-linear data assimilation techniques that aim to represent the probability distribution of the model state given the observations (the posterior) by a number of particles. In high-dimensional geophysical applications the number of particles required by the sequential importance resampling (SIR) particle filter in order to capture the high probability region of the posterior, is too large to make them usable. However particle filters can be formulated using proposal densities, which gives greater freedom in how particles are sampled and allows for a much smaller number of particles. Here a particle filter is presented which uses the proposal density to ensure that all particles end up in the high probability region of the posterior probability density function. This gives rise to the possibility of non-linear data assimilation in large dimensional systems. The particle filter formulation is compared to the optimal proposal density particle filter and the implicit particle filter, both of which also utilise a proposal density. We show that when observations are available every time step, both schemes will be degenerate when the number of independent observations is large, unlike the new scheme. The sensitivity of the new scheme to its parameter values is explored theoretically and demonstrated using the Lorenz (1963) model.

Towards a parallel computationally efficient approach to scaling up data stream classification

Advances in hardware technologies allow to capture and process data in real-time and the resulting high throughput data streams require novel data mining approaches. The research area of Data Stream Mining (DSM) is developing data mining algorithms that allow us to analyse these continuous streams of data in real-time. The creation and real-time adaption of classification models from data streams is one of the most challenging DSM tasks. Current classifiers for streaming data address this problem by using incremental learning algorithms. However, even so these algorithms are fast, they are challenged by high velocity data streams, where data instances are incoming at a fast rate. This is problematic if the applications desire that there is no or only a very little delay between changes in the patterns of the stream and absorption of these patterns by the classifier. Problems of scalability to Big Data of traditional data mining algorithms for static (non streaming) datasets have been addressed through the development of parallel classifiers. However, there is very little work on the parallelisation of data stream classification techniques. In this paper we investigate K-Nearest Neighbours (KNN) as the basis for a real-time adaptive and parallel methodology for scalable data stream classification tasks.

The effect of the equivalent-weights particle filter on dynamical balance in a primitive equation model

The disadvantage of the majority of data assimilation schemes is the assumption that the conditional probability density function of the state of the system given the observations [posterior probability density function (PDF)] is distributed either locally or globally as a Gaussian. The advantage, however, is that through various different mechanisms they ensure initial conditions that are predominantly in linear balance and therefore spurious gravity wave generation is suppressed. The equivalent-weights particle filter is a data assimilation scheme that allows for a representation of a potentially multimodal posterior PDF. It does this via proposal densities that lead to extra terms being added to the model equations and means the advantage of the traditional data assimilation schemes, in generating predominantly balanced initial conditions, is no longer guaranteed. This paper looks in detail at the impact the equivalent-weights particle filter has on dynamical balance and gravity wave generation in a primitive equation model. The primary conclusions are that (i) provided the model error covariance matrix imposes geostrophic balance, then each additional term required by the equivalent-weights particle filter is also geostrophically balanced; (ii) the relaxation term required to ensure the particles are in the locality of the observations has little effect on gravity waves and actually induces a reduction in gravity wave energy if sufficiently large; and (iii) the equivalent-weights term, which leads to the particles having equivalent significance in the posterior PDF, produces a change in gravity wave energy comparable to the stochastic model error. Thus, the scheme does not produce significant spurious gravity wave energy and so has potential for application in real high-dimensional geophysical applications.

Twin experiments with the equivalent weights particle filter and HadCM3

This paper investigates the use of a particle filter for data assimilation with a full scale coupled ocean–atmosphere general circulation model. Synthetic twin experiments are performed to assess the performance of the equivalent weights filter in such a high-dimensional system. Artificial 2-dimensional sea surface temperature fields are used as observational data every day. Results are presented for different values of the free parameters in the method. Measures of the performance of the filter are root mean square errors, trajectories of individual variables in the model and rank histograms. Filter degeneracy is not observed and the performance of the filter is shown to depend on the ability to keep maximum spread in the ensemble.

High-throughput monitoring of wild bee diversity and abundance via mitogenomics

1. Bee populations and other pollinators face multiple, synergistically acting threats, which have led to population declines, loss of local species richness and pollination services, and extinctions. However, our understanding of the degree, distribution and causes of declines is patchy, in part due to inadequate monitoring systems, with the challenge of taxonomic identification posing a major logistical barrier. Pollinator conservation would benefit from a high-throughput identification pipeline. 2. We show that the metagenomic mining and resequencing of mitochondrial genomes (mitogenomics) can be applied successfully to bulk samples of wild bees. We assembled the mitogenomes of 48 UK bee species and then shotgun-sequenced total DNA extracted from 204 whole bees that had been collected in 10 pan-trap samples from farms in England and been identified morphologically to 33 species. Each sample data set was mapped against the 48 reference mitogenomes. 3. The morphological and mitogenomic data sets were highly congruent. Out of 63 total species detections in the morphological data set, the mitogenomic data set made 59 correct detections (93�7% detection rate) and detected six more species (putative false positives). Direct inspection and an analysis with species-specific primers suggested that these putative false positives were most likely due to incorrect morphological IDs. Read frequency significantly predicted species biomass frequency (R2 = 24�9%). Species lists, biomass frequencies, extrapolated species richness and community structure were recovered with less error than in a metabarcoding pipeline. 4. Mitogenomics automates the onerous task of taxonomic identification, even for cryptic species, allowing the tracking of changes in species richness and istributions. A mitogenomic pipeline should thus be able to contain costs, maintain consistently high-quality data over long time series, incorporate retrospective taxonomic revisions and provide an auditable evidence trail. Mitogenomic data sets also provide estimates of species counts within samples and thus have potential for tracking population trajectories.

Tensor LRR and sparse coding-based subspace clustering

Subspace clustering groups a set of samples from a union of several linear subspaces into clusters, so that the samples in the same cluster are drawn from the same linear subspace. In the majority of the existing work on subspace clustering, clusters are built based on feature information, while sample correlations in their original spatial structure are simply ignored. Besides, original high-dimensional feature vector contains noisy/redundant information, and the time complexity grows exponentially with the number of dimensions. To address these issues, we propose a tensor low-rank representation (TLRR) and sparse coding-based (TLRRSC) subspace clustering method by simultaneously considering feature information and spatial structures. TLRR seeks the lowest rank representation over original spatial structures along all spatial directions. Sparse coding learns a dictionary along feature spaces, so that each sample can be represented by a few atoms of the learned dictionary. The affinity matrix used for spectral clustering is built from the joint similarities in both spatial and feature spaces. TLRRSC can well capture the global structure and inherent feature information of data, and provide a robust subspace segmentation from corrupted data. Experimental results on both synthetic and real-world data sets show that TLRRSC outperforms several established state-of-the-art methods.

Influence of south Atlantic sea surface temperatures on rainfall variability and extremes over southern Africa

It is generally agreed that changing climate variability, and the associated change in climate extremes, may have a greater impact on environmentally vulnerable regions than a changing mean. This research investigates rainfall variability, rainfall extremes, and their associations with atmospheric and oceanic circulations over southern Africa, a region that is considered particularly vulnerable to extreme events because of numerous environmental, social, and economic pressures. Because rainfall variability is a function of scale, high-resolution data are needed to identify extreme events. Thus, this research uses remotely sensed rainfall data and climate model experiments at high spatial and temporal resolution, with the overall aim being to investigate the ways in which sea surface temperature (SST) anomalies influence rainfall extremes over southern Africa. Extreme rainfall identification is achieved by the high-resolution microwave/infrared rainfall algorithm dataset. This comprises satellite-derived daily rainfall from 1993 to 2002 and covers southern Africa at a spatial resolution of 0.1° latitude–longitude. Extremes are extracted and used with reanalysis data to study possible circulation anomalies associated with extreme rainfall. Anomalously cold SSTs in the central South Atlantic and warm SSTs off the coast of southwestern Africa seem to be statistically related to rainfall extremes. Further, through a number of idealized climate model experiments, it would appear that both decreasing SSTs in the central South Atlantic and increasing SSTs off the coast of southwestern Africa lead to a demonstrable increase in daily rainfall and rainfall extremes over southern Africa, via local effects such as increased convection and remote effects such as an adjustment of the Walker-type circulation.

Particle filtering in geophysical systems

The application of particle filters in geophysical systems is reviewed. Some background on Bayesian filtering is provided, and the existing methods are discussed. The emphasis is on the methodology, and not so much on the applications themselves. It is shown that direct application of the basic particle filter (i.e., importance sampling using the prior as the importance density) does not work in high-dimensional systems, but several variants are shown to have potential. Approximations to the full problem that try to keep some aspects of the particle filter beyond the Gaussian approximation are also presented and discussed.

Lagrangian validation of numerical drifter trajectories using drifting buoys: Application to the Agulhas system

The skill of numerical Lagrangian drifter trajectories in three numerical models is assessed by comparing these numerically obtained paths to the trajectories of drifting buoys in the real ocean. The skill assessment is performed using the two-sample Kolmogorov–Smirnov statistical test. To demonstrate the assessment procedure, it is applied to three different models of the Agulhas region. The test can either be performed using crossing positions of one-dimensional sections in order to test model performance in specific locations, or using the total two-dimensional data set of trajectories. The test yields four quantities: a binary decision of model skill, a confidence level which can be used as a measure of goodness-of-fit of the model, a test statistic which can be used to determine the sensitivity of the confidence level, and cumulative distribution functions that aid in the qualitative analysis. The ordering of models by their confidence levels is the same as the ordering based on the qualitative analysis, which suggests that the method is suited for model validation. Only one of the three models, a 1/10° two-way nested regional ocean model, might have skill in the Agulhas region. The other two models, a 1/2° global model and a 1/8° assimilative model, might have skill only on some sections in the region

The microwave spectrum of 1-pyrazoline

The microwave spectrum of 1-pyrazoline has been observed from 18 to 40 GHz in the six lowest states of the ring-puckering vibration. It is an a-type spectrum of a near oblate asymmetric top. Each vibrational state has been fitted to a separate effective Hamiltonian, and the vibrational dependence of both the rotational constants and the quartic centrifugal distortion constants has been observed and analyzed. The v = 0 and 1 states have also been analyzed using a coupled Hamiltonian; this gives consistent results, with an improved fit to the high J data. The preferred choice of Durig et al. [J. Chem. Phys. 52, 6096 (1970)] for the ring-puckering potential is confirmed as essentially correct, but the A and B inertial axes are shown to be interchanged from those assumed by Durig et al. in their analysis of the mid-infrared spectrum.

MALDI MS

Matrix-assisted laser desorption/ionization (MALDI) is a key technique in mass spectrometry (MS)-based proteomics. MALDI MS is extremely sensitive, easy-to-apply, and relatively tolerant to contaminants. Its high-speed data acquisition and large-scale, off-line sample preparation has made it once again the focus for high-throughput proteomic analyses. These and other unique properties of MALDI offer new possibilities in applications such as rapid molecular profiling and imaging by MS. Proteomics and its employment in Systems Biology and other areas that require sensitive and high-throughput bioanalytical techniques greatly depend on these methodologies. This chapter provides a basic introduction to the MALDI methodology and its general application in proteomic research. It describes the basic MALDI sample preparation steps and two easy-to-follow examples for protein identification including extensive notes on these topics with practical tips that are often not available in the Subheadings 2 and 3 of research articles.

Artichoke leaf extract (Cynara scolymus) reduces plasma cholesterol in otherwise healthy hypercholesterolemic adults: A randomized, double blind placebo controlled trial

Cardiovascular diseases are the chief causes of death in the UK, and are associated with high circulating levels of total cholesterol in the plasma. Artichoke leaf extracts (ALEs) have been reported to reduce plasma lipids levels, including total cholesterol, although high quality data is lacking. The objective of this trial was to assess the effect of ALE on plasma lipid levels and general well-being in otherwise healthy adults with mild to moderate hypercholesterolemia. 131 adults were screened for total plasma cholesterol in the range 6.0-8.0 mmol/l, with 75 suitable volunteers randomised onto the trial. Volunteers consumed 1280 mg of a standardised ALE, or matched placebo, daily for 12 weeks. Plasma total cholesterol decreased in the treatment group by an average of 4.2% (from 7.16 (SD 0.62) mmol/l to 6.86 (SD 0.68) mmol/l) and increased in the control group by an average of 1.9% (6.90 (SD 0.49) mmol/l to 7.03 (0.61) mmol/l), the difference between groups being statistically significant (p = 0.025). No significant differences between groups were observed for LDL cholesterol, HDL cholesterol or triglyceride levels. General well-being improved significantly in both the treatment (11%) and control groups (9%) with no significant differences between groups. In conclusion, ALE consumption resulted in a modest but favourable statistically significant difference in total cholesterol after 12 weeks. In comparison with a previous trial, it is suggested that the apparent positive health status of the study population may have contributed to the modesty of the observed response. (C) 2008 Elsevier GmbH. All rights reserved.

Clustering quality and topology preservation in fast learning SOMs

The Self-Organizing Map (SOM) is a popular unsupervised neural network able to provide effective clustering and data visualization for data represented in multidimensional input spaces. In this paper, we describe Fast Learning SOM (FLSOM) which adopts a learning algorithm that improves the performance of the standard SOM with respect to the convergence time in the training phase. We show that FLSOM also improves the quality of the map by providing better clustering quality and topology preservation of multidimensional input data. Several tests have been carried out on different multidimensional datasets, which demonstrate better performances of the algorithm in comparison with the original SOM.

Ensemble prediction for nowcasting with a convection-permitting model - II: forecast error statistics

A 24-member ensemble of 1-h high-resolution forecasts over the Southern United Kingdom is used to study short-range forecast error statistics. The initial conditions are found from perturbations from an ensemble transform Kalman filter. Forecasts from this system are assumed to lie within the bounds of forecast error of an operational forecast system. Although noisy, this system is capable of producing physically reasonable statistics which are analysed and compared to statistics implied from a variational assimilation system. The variances for temperature errors for instance show structures that reflect convective activity. Some variables, notably potential temperature and specific humidity perturbations, have autocorrelation functions that deviate from 3-D isotropy at the convective-scale (horizontal scales less than 10 km). Other variables, notably the velocity potential for horizontal divergence perturbations, maintain 3-D isotropy at all scales. Geostrophic and hydrostatic balances are studied by examining correlations between terms in the divergence and vertical momentum equations respectively. Both balances are found to decay as the horizontal scale decreases. It is estimated that geostrophic balance becomes less important at scales smaller than 75 km, and hydrostatic balance becomes less important at scales smaller than 35 km, although more work is required to validate these findings. The implications of these results for high-resolution data assimilation are discussed.