Scaling up data mining techniques to large datasets using parallel and distributed processing

Advances in hardware and software technology enable us to collect, store and distribute large quantities of data on a very large scale. Automatically discovering and extracting hidden knowledge in the form of patterns from these large data volumes is known as data mining. Data mining technology is not only a part of business intelligence, but is also used in many other application areas such as research, marketing and financial analytics. For example medical scientists can use patterns extracted from historic patient data in order to determine if a new patient is likely to respond positively to a particular treatment or not; marketing analysts can use extracted patterns from customer data for future advertisement campaigns; finance experts have an interest in patterns that forecast the development of certain stock market shares for investment recommendations. However, extracting knowledge in the form of patterns from massive data volumes imposes a number of computational challenges in terms of processing time, memory, bandwidth and power consumption. These challenges have led to the development of parallel and distributed data analysis approaches and the utilisation of Grid and Cloud computing. This chapter gives an overview of parallel and distributed computing approaches and how they can be used to scale up data mining to large datasets.

Cyclones causing wind storms in the Mediterranean: characteristics, trends and links to large-scale patterns

A climatology of cyclones with a focus on their relation to wind storm tracks in the Mediterranean region (MR) is presented. Trends in the frequency of cyclones and wind storms, as well as variations associated with the North Atlantic Oscillation (NAO), the East Atlantic/West Russian (EAWR) and the Scandinavian variability pattern (SCAND) are discussed. The study is based on the ERA40 reanalysis dataset. Wind storm tracks are identified by tracking clusters of adjacent grid boxes characterised by extremely high local wind speeds. The wind track is assigned to a cyclone track independently identified with an objective scheme. Areas with high wind activity – quantified by extreme wind tracks – are typically located south of the Golf of Genoa, south of Cyprus, southeast of Sicily and west of the Iberian Peninsula. About 69% of the wind storms are caused by cyclones located in the Mediterranean region, while the remaining 31% can be attributed to North Atlantic or Northern European cyclones. The North Atlantic Oscillation, the East Atlantic/West Russian pattern and the Scandinavian pattern all influence the amount and spatial distribution of wind inducing cyclones and wind events in the MR. The strongest signals exist for the NAO and the EAWR pattern, which are both associated with an increase in the number of organised strong wind events in the eastern MR during their positive phase. On the other hand, the storm numbers decrease over the western MR for the positive phase of the NAO and over the central MR during the positive phase of the EAWR pattern. The positive phase of the Scandinavian pattern is associated with a decrease in the number of winter wind storms over most of the MR. A third of the trends in the number of wind storms and wind producing cyclones during the winter season of the ERA40 period may be attributed to the variability of the North Atlantic Oscillation.

Rossby number expansions, slaving principles, and balance dynamics

We consider the problem of constructing balance dynamics for rapidly rotating fluid systems. It is argued that the conventional Rossby number expansion—namely expanding all variables in a series in Rossby number—is secular for all but the simplest flows. In particular, the higher-order terms in the expansion grow exponentially on average, and for moderate values of the Rossby number the expansion is, at best, useful only for times of the order of the doubling times of the instabilities of the underlying quasi-geostrophic dynamics. Similar arguments apply in a wide class of problems involving a small parameter and sufficiently complex zeroth-order dynamics. A modified procedure is proposed which involves expanding only the fast modes of the system; this is equivalent to an asymptotic approximation of the slaving relation that relates the fast modes to the slow modes. The procedure is systematic and thus capable, at least in principle, of being carried to any order—unlike procedures based on truncations. We apply the procedure to construct higher-order balance approximations of the shallow-water equations. At the lowest order quasi-geostrophy emerges. At the next order the system incorporates gradient-wind balance, although the balance relations themselves involve only linear inversions and hence are easily applied. There is a large class of reduced systems associated with various choices for the slow variables, but the simplest ones appear to be those based on potential vorticity.

Candidate genes for obesity-susceptibility show enriched association within a large genome-wide association study for BMI

Before the advent of genome-wide association studies (GWASs), hundreds of candidate genes for obesity-susceptibility had been identified through a variety of approaches. We examined whether those obesity candidate genes are enriched for associations with body mass index (BMI) compared with non-candidate genes by using data from a large-scale GWAS. A thorough literature search identified 547 candidate genes for obesity-susceptibility based on evidence from animal studies, Mendelian syndromes, linkage studies, genetic association studies and expression studies. Genomic regions were defined to include the genes ±10 kb of flanking sequence around candidate and non-candidate genes. We used summary statistics publicly available from the discovery stage of the genome-wide meta-analysis for BMI performed by the genetic investigation of anthropometric traits consortium in 123 564 individuals. Hypergeometric, rank tail-strength and gene-set enrichment analysis tests were used to test for the enrichment of association in candidate compared with non-candidate genes. The hypergeometric test of enrichment was not significant at the 5% P-value quantile (P = 0.35), but was nominally significant at the 25% quantile (P = 0.015). The rank tail-strength and gene-set enrichment tests were nominally significant for the full set of genes and borderline significant for the subset without SNPs at P < 10(-7). Taken together, the observed evidence for enrichment suggests that the candidate gene approach retains some value. However, the degree of enrichment is small despite the extensive number of candidate genes and the large sample size. Studies that focus on candidate genes have only slightly increased chances of detecting associations, and are likely to miss many true effects in non-candidate genes, at least for obesity-related traits.

Centennial variations in sunspot number, open solar flux and streamer belt width: 1. correction of the sunspot number record since 1874

We analyse the widely-used international/ Zürich sunspot number record, R, with a view to quantifying a suspected calibration discontinuity around 1945 (which has been termed the “Waldmeier discontinuity” [Svalgaard, 2011]). We compare R against the composite sunspot group data from the Royal Greenwich Observatory (RGO) network and the Solar Optical Observing Network (SOON), using both the number of sunspot groups, N{sub}G{\sub}, and the total area of the sunspots, A{sub}G{\sub}. In addition, we compare R with the recently developed interdiurnal variability geomagnetic indices IDV and IDV(1d). In all four cases, linearity of the relationship with R is not assumed and care is taken to ensure that the relationship of each with R is the same before and after the putative calibration change. It is shown the probability that a correction is not needed is of order 10{sup}−8{\sup} and that R is indeed too low before 1945. The optimum correction to R for values before 1945 is found to be 11.6%, 11.7%, 10.3% and 7.9% using A{sub}G{\sub}, N{sub)G{\sub}, IDV, and IDV(1d), respectively. The optimum value obtained by combining the sunspot group data is 11.6% with an uncertainty range 8.1-14.8% at the 2σ level. The geomagnetic indices provide an independent yet less stringent test but do give values that fall within the 2σ uncertainty band with optimum values are slightly lower than from the sunspot group data. The probability of the correction needed being as large as 20%, as advocated by Svalgaard [2011], is shown to be 1.6 × 10{sup}−5{\sup}.

A statistical study of large field-aligned flows of thermal ions at high-latitudes

A number of case studies of large, transient, field-aligned ion flows in the topside ionosphere at high-latitudes have been reported, showing that these events occur during periods of frictional heating and/or intense particle precipitation. This study examines the frequency of occurrence of such events for the altitude range 200–500 km, based on 3 years of incoherent scatter data. Correlations of the upgoing ion flux at 400 km with ion and electron temperatures at lower altitudes are presented, together with a discussion of possible mechanisms for the production of such large flows. The influence of low-altitude electron precipitation on the production of these events is also considered.

Improving a global model from the boundary layer: total turbulent energy and the neutral limit Prandtl number

Model intercomparisons have identified important deficits in the representation of the stable boundary layer by turbulence parametrizations used in current weather and climate models. However, detrimental impacts of more realistic schemes on the large-scale flow have hindered progress in this area. Here we implement a total turbulent energy scheme into the climate model ECHAM6. The total turbulent energy scheme considers the effects of Earth’s rotation and static stability on the turbulence length scale. In contrast to the previously used turbulence scheme, the TTE scheme also implicitly represents entrainment flux in a dry convective boundary layer. Reducing the previously exaggerated surface drag in stable boundary layers indeed causes an increase in southern hemispheric zonal winds and large-scale pressure gradients beyond observed values. These biases can be largely removed by increasing the parametrized orographic drag. Reducing the neutral limit turbulent Prandtl number warms and moistens low-latitude boundary layers and acts to reduce longstanding radiation biases in the stratocumulus regions, the Southern Ocean and the equatorial cold tongue that are common to many climate models.

Tests of sunspot number sequences: 3. Effects of regression procedures on the calibration of historic sunspot data

We use sunspot group observations from the Royal Greenwich Observatory (RGO) to investigate the effects of intercalibrating data from observers with different visual acuities. The tests are made by counting the number of groups RB above a variable cut-off threshold of observed total whole-spot area (uncorrected for foreshortening) to simulate what a lower acuity observer would have seen. The synthesised annual means of RB are then re-scaled to the full observed RGO group number RA using a variety of regression techniques. It is found that a very high correlation between RA and RB (rAB > 0.98) does not prevent large errors in the intercalibration (for example sunspot maximum values can be over 30 % too large even for such levels of rAB). In generating the backbone sunspot number (RBB), Svalgaard and Schatten (2015, this issue) force regression fits to pass through the scatter plot origin which generates unreliable fits (the residuals do not form a normal distribution) and causes sunspot cycle amplitudes to be exaggerated in the intercalibrated data. It is demonstrated that the use of Quantile-Quantile (“Q  Q”) plots to test for a normal distribution is a useful indicator of erroneous and misleading regression fits. Ordinary least squares linear fits, not forced to pass through the origin, are sometimes reliable (although the optimum method used is shown to be different when matching peak and average sunspot group numbers). However, other fits are only reliable if non-linear regression is used. From these results it is entirely possible that the inflation of solar cycle amplitudes in the backbone group sunspot number as one goes back in time, relative to related solar-terrestrial parameters, is entirely caused by the use of inappropriate and non-robust regression techniques to calibrate the sunspot data.

Tests of sunspot number sequences: 1. Using ionosonde data

More than 70 years ago it was recognised that ionospheric F2-layer critical frequencies [foF2] had a strong relationship to sunspot number. Using historic datasets from the Slough and Washington ionosondes, we evaluate the best statistical fits of foF2 to sunspot numbers (at each Universal Time [UT] separately) in order to search for drifts and abrupt changes in the fit residuals over Solar Cycles 17-21. This test is carried out for the original composite of the Wolf/Zürich/International sunspot number [R], the new “backbone” group sunspot number [RBB] and the proposed “corrected sunspot number” [RC]. Polynomial fits are made both with and without allowance for the white-light facular area, which has been reported as being associated with cycle-to-cycle changes in the sunspot number - foF2 relationship. Over the interval studied here, R, RBB, and RC largely differ in their allowance for the “Waldmeier discontinuity” around 1945 (the correction factor for which for R, RBB and RC is, respectively, zero, effectively over 20 %, and explicitly 11.6 %). It is shown that for Solar Cycles 18-21, all three sunspot data sequences perform well, but that the fit residuals are lowest and most uniform for RBB. We here use foF2 for those UTs for which R, RBB, and RC all give correlations exceeding 0.99 for intervals both before and after the Waldmeier discontinuity. The error introduced by the Waldmeier discontinuity causes R to underestimate the fitted values based on the foF2 data for 1932-1945 but RBB overestimates them by almost the same factor, implying that the correction for the Waldmeier discontinuity inherent in RBB is too large by a factor of two. Fit residuals are smallest and most uniform for RC and the ionospheric data support the optimum discontinuity multiplicative correction factor derived from the independent Royal Greenwich Observatory (RGO) sunspot group data for the same interval.

Screening of ARHSP-TCC Patients Expands the Spectrum of SPG11 Mutations and Includes a Large Scale Gene Deletion

Autosomal recessive spastic paraplegia with thinning of corpus callosum (ARHSP-TCC) is a complex form of HSP initially described in Japan but subsequently reported to have a worldwide distribution with a particular high frequency in multiple families from the Mediterranean basin. We recently showed that ARHSP-TCC is commonly associated with mutations in SPG11/KIAA1840 on chromosome 15q. We have now screened a collection of new patients mainly originating from Italy and Brazil, in order to further ascertain the spectrum of mutations in SPG11, enlarge the ethnic origin of SPG11 patients, determine the relative frequency at the level of single Countries (i.e., Italy), and establish whether there is one or more common mutation. In 25 index cases we identified 32 mutations; 22 are novel, including 9 nonsense, 3 small deletions, 4 insertions, 1 in/del, 1 small duplication, 1 missense, 2 splice-site, and for the first time a large genomic rearrangement. This brings the total number of SPG11 mutated patients in the SPATAX collection to 111 cases in 44 families and in 17 isolated cases, from 16 Countries, all assessed using homogeneous clinical criteria. While expanding the spectrum of mutations in SPG11, this larger series also corroborated the notion that even within apparently homogeneous population a molecular diagnosis cannot be achieved without full gene sequencing. (C) 2008 Wiley-Liss, Inc.

On the efficiency of transportation systems in large cities

We report an analysis of the accessibility between different locations in big cities, which is illustrated with respect to London and Paris. The effects of the respective underground systems in facilitating more uniform access to diverse places are also quantified and investigated. It is shown that London and Paris have markedly different patterns of accessibility, as a consequence of the number of bridges and large parks of London, and that in both cases the respective underground systems imply in general, thought in distinct manners, an increase of accessibility. Copyright (C) EPLA, 2010

A multi-GPU algorithm for large-scale neuronal networks

Large-scale simulations of parts of the brain using detailed neuronal models to improve our understanding of brain functions are becoming a reality with the usage of supercomputers and large clusters. However, the high acquisition and maintenance cost of these computers, including the physical space, air conditioning, and electrical power, limits the number of simulations of this kind that scientists can perform. Modern commodity graphical cards, based on the CUDA platform, contain graphical processing units (GPUs) composed of hundreds of processors that can simultaneously execute thousands of threads and thus constitute a low-cost solution for many high-performance computing applications. In this work, we present a CUDA algorithm that enables the execution, on multiple GPUs, of simulations of large-scale networks composed of biologically realistic Hodgkin-Huxley neurons. The algorithm represents each neuron as a CUDA thread, which solves the set of coupled differential equations that model each neuron. Communication among neurons located in different GPUs is coordinated by the CPU. We obtained speedups of 40 for the simulation of 200k neurons that received random external input and speedups of 9 for a network with 200k neurons and 20M neuronal connections, in a single computer with two graphic boards with two GPUs each, when compared with a modern quad-core CPU. Copyright (C) 2010 John Wiley & Sons, Ltd.

The 3-colored Ramsey number of even cycles

Denote by R(L, L, L) the minimum integer N such that any 3-coloring of the edges of the complete graph on N vertices contains a monochromatic copy of a graph L. Bondy and Erdos conjectured that when L is the cycle C(n) on n vertices, R(C(n), C(n), C(n)) = 4n - 3 for every odd n > 3. Luczak proved that if n is odd, then R(C(n), C(n), C(n)) = 4n + o(n), as n -> infinity, and Kohayakawa, Simonovits and Skokan confirmed the Bondy-Erdos conjecture for all sufficiently large values of n. Figaj and Luczak determined an asymptotic result for the `complementary` case where the cycles are even: they showed that for even n, we have R(C(n), C(n), C(n)) = 2n + o(n), as n -> infinity. In this paper, we prove that there exists n I such that for every even n >= n(1), R(C(n), C(n), C(n)) = 2n. (C) 2009 Elsevier Inc. All rights reserved.

A computational comparison of different algorithms for very large p-median problems

In this paper, we propose a new method for solving large scale p-median problem instances based on real data. We compare different approaches in terms of runtime, memory footprint and quality of solutions obtained. In order to test the different methods on real data, we introduce a new benchmark for the p-median problem based on real Swedish data. Because of the size of the problem addressed, up to 1938 candidate nodes, a number of algorithms, both exact and heuristic, are considered. We also propose an improved hybrid version of a genetic algorithm called impGA. Experiments show that impGA behaves as well as other methods for the standard set of medium-size problems taken from Beasley’s benchmark, but produces comparatively good results in terms of quality, runtime and memory footprint on our specific benchmark based on real Swedish data.

Estimated Number and Location of Future Moose-Vehicle Collisions (MVC) in Maine

Moose (Alces alces) are a keystone herbivore in Maine. Because of the large number of rural roads in Maine, there is a high rate of moose-vehicle collisions (MVCs), which is increasing. On-road encounters with animals resulted in 231 fatalities in the United States in 1999. Because of the fatality of MVCs, it is important to know where they are most likely to occur. I used GIS analysis to estimate where future MVCs would occur, factoring in the variables of land cover suitability for moose, distance from water bodies, locations of past MVCs, and speed limits on the roads. I ran four different analyses, each one weighting the variables equally. I also ran a regression to determine if increasing road speed was associated with the increase in the number of MVCs per length of road. There was not a strong positive relationship between the number of MVCs per length of road and the speed limit, but it was interesting to note that there were more MVCs per length of road on 35mph and 40mph roads than on 45, 50, 55 or 65mph roads. Future research on MVCs would benefit from the inclusion of include moose population density and road traffic data.