Sequential anomaly detection in the presence of noise and limited feedback

This paper describes a methodology for detecting anomalies from sequentially observed and potentially noisy data. The proposed approach consists of two main elements: 1) filtering, or assigning a belief or likelihood to each successive measurement based upon our ability to predict it from previous noisy observations and 2) hedging, or flagging potential anomalies by comparing the current belief against a time-varying and data-adaptive threshold. The threshold is adjusted based on the available feedback from an end user. Our algorithms, which combine universal prediction with recent work on online convex programming, do not require computing posterior distributions given all current observations and involve simple primal-dual parameter updates. At the heart of the proposed approach lie exponential-family models which can be used in a wide variety of contexts and applications, and which yield methods that achieve sublinear per-round regret against both static and slowly varying product distributions with marginals drawn from the same exponential family. Moreover, the regret against static distributions coincides with the minimax value of the corresponding online strongly convex game. We also prove bounds on the number of mistakes made during the hedging step relative to the best offline choice of the threshold with access to all estimated beliefs and feedback signals. We validate the theory on synthetic data drawn from a time-varying distribution over binary vectors of high dimensionality, as well as on the Enron email dataset. © 1963-2012 IEEE.

Evaluating the SEVIRI Fire Thermal Anomaly Detection Algorithm across the Central African Republic Using the MODIS Active Fire Product

Satellite-based remote sensing of active fires is the only practical way to consistently and continuously monitor diurnal fluctuations in biomass burning from regional, to continental, to global scales. Failure to understand, quantify, and communicate the performance of an active fire detection algorithm, however, can lead to improper interpretations of the spatiotemporal distribution of biomass burning, and flawed estimates of fuel consumption and trace gas and aerosol emissions. This work evaluates the performance of the Spinning Enhanced Visible and Infrared Imager (SEVIRI) Fire Thermal Anomaly (FTA) detection algorithm using seven months of active fire pixels detected by the Moderate Resolution Imaging Spectroradiometer (MODIS) across the Central African Republic (CAR). Results indicate that the omission rate of the SEVIRI FTA detection algorithm relative to MODIS varies spatially across the CAR, ranging from 25% in the south to 74% in the east. In the absence of confounding artifacts such as sunglint, uncertainties in the background thermal characterization, and cloud cover, the regional variation in SEVIRI's omission rate can be attributed to a coupling between SEVIRI's low spatial resolution detection bias (i.e., the inability to detect fires below a certain size and intensity) and a strong geographic gradient in active fire characteristics across the CAR. SEVIRI's commission rate relative to MODIS increases from 9% when evaluated near MODIS nadir to 53% near the MODIS scene edges, indicating that SEVIRI errors of commission at the MODIS scene edges may not be false alarms but rather true fires that MODIS failed to detect as a result of larger pixel sizes at extreme MODIS scan angles. Results from this work are expected to facilitate (i) future improvements to the SEVIRI FTA detection algorithm; (ii) the assimilation of the SEVIRI and MODIS active fire products; and (iii) the potential inclusion of SEVIRI into a network of geostationary sensors designed to achieve global diurnal active fire monitoring.

Magnetic tight binding and the iron-chromium enthalpy anomaly

We describe a self-consistent magnetic tight-binding theory based in an expansion of the Hohenberg-Kohn density functional to second order, about a non-spin-polarized reference density. We show how a first order expansion about a density having a trial input magnetic moment leads to a fixed moment model. We employ a simple set of tight-binding parameters that accurately describes electronic structure and energetics, and show these to be transferable between first row transition metals and their alloys. We make a number of calculations of the electronic structure of dilute Cr impurities in Fe, which we compare with results using the local spin density approximation. The fixed moment model provides a powerful means for interpreting complex magnetic configurations in alloys; using this approach, we are able to advance a simple and readily understood explanation for the observed anomaly in the enthalpy of mixing.

Resolution of an ancient surface science anomaly: Work function change induced by N adsorption on W{100}

For many decades it has been assumed that an adsorbate centered above a metal surface and with a net negative charge should increase the work function of the surface. However, despite their electronegativity, N adatoms on W{100} cause a significant work function decrease. Here we present a resolution of this anomaly. Using density functional theory, we demonstrate that while the N atom carries a negative charge, of overriding importance is a reduction in the surface overspill electron density into the vacuum, when that charge is engaged in bonding to the adatom. This novel interpretation is fundamentally important in the general understanding of work function changes induced by atomic adsorbates.

The anomaly in the candidate microlensing event PA-99-N2

The light curve of PA-99-N2, one of the recently announced microlensing candidates toward M31, shows small deviations from the standard Paczynski form. We explore a number of possible explanations, including correlations with the seeing, the parallax effect, and a binary lens. We find that the observations are consistent with an unresolved red giant branch or asymptotic giant branch star in M31 being microlensed by a binary lens. We find that the best-fit binary lens mass ratio is similar to1.2x10(-2), which is one of the most extreme values found for a binary lens so far. If both the source and lens lie in the M31 disk, then the standard M31 model predicts the probable mass range of the system to be 0.02-3.6 M-circle dot (95% confidence limit). In this scenario, the mass of the secondary component is therefore likely to be below the hydrogen-burning limit. On the other hand, if a compact halo object in M31 is lensing a disk or spheroid source, then the total lens mass is likely to lie between 0.09 and 32 M-circle dot, which is consistent with the primary being a stellar remnant and the secondary being a low-mass star or brown dwarf. The optical depth (or, alternatively, the differential rate) along the line of sight toward the event indicates that a halo lens is more likely than a stellar lens, provided that dark compact objects comprise no less than 15% (or 5%) of halos.

Electronic stopping power in gold:The role of d electrons and the H/He anomaly

The electronic stopping power of H and He moving through gold is obtained to high accuracy using time-evolving density-functional theory, thereby bringing usual first principles accuracies into this kind of strongly coupled, continuum nonadiabatic processes in condensed matter. The two key unexplained features of what observed experimentally have been reproduced and understood: (i)The nonlinear behavior of stopping power versus velocity is a gradual crossover as excitations tail into the d-electron spectrum; and (ii)the low-velocity H/He anomaly (the relative stopping powers are contrary to established theory) is explained by the substantial involvement of the d electrons in the screening of the projectile even at the lowest velocities where the energy loss is generated by s-like electron-hole pair formation only. © 2012 American Physical Society.

A Lightweight Tool for Anomaly Detection in Cloud Data Centres

Cloud data centres are critical business infrastructures and the fastest growing service providers. Detecting anomalies in Cloud data centre operation is vital. Given the vast complexity of the data centre system software stack, applications and workloads, anomaly detection is a challenging endeavour. Current tools for detecting anomalies often use machine learning techniques, application instance behaviours or system metrics distribu- tion, which are complex to implement in Cloud computing environments as they require training, access to application-level data and complex processing. This paper presents LADT, a lightweight anomaly detection tool for Cloud data centres that uses rigorous correlation of system metrics, implemented by an efficient corre- lation algorithm without need for training or complex infrastructure set up. LADT is based on the hypothesis that, in an anomaly-free system, metrics from data centre host nodes and virtual machines (VMs) are strongly correlated. An anomaly is detected whenever correlation drops below a threshold value. We demonstrate and evaluate LADT using a Cloud environment, where it shows that the hosting node I/O operations per second (IOPS) are strongly correlated with the aggregated virtual machine IOPS, but this correlation vanishes when an application stresses the disk, indicating a node-level anomaly.

Anomaly Detection for Data with Spatial Attributes

The problem of detecting spatially-coherent groups of data that exhibit anomalous behavior has started to attract attention due to applications across areas such as epidemic analysis and weather forecasting. Earlier efforts from the data mining community have largely focused on finding outliers, individual data objects that display deviant behavior. Such point-based methods are not easy to extend to find groups of data that exhibit anomalous behavior. Scan Statistics are methods from the statistics community that have considered the problem of identifying regions where data objects exhibit a behavior that is atypical of the general dataset. The spatial scan statistic and methods that build upon it mostly adopt the framework of defining a character for regions (e.g., circular or elliptical) of objects and repeatedly sampling regions of such character followed by applying a statistical test for anomaly detection. In the past decade, there have been efforts from the statistics community to enhance efficiency of scan statstics as well as to enable discovery of arbitrarily shaped anomalous regions. On the other hand, the data mining community has started to look at determining anomalous regions that have behavior divergent from their neighborhood.In this chapter,we survey the space of techniques for detecting anomalous regions on spatial data from across the data mining and statistics communities while outlining connections to well-studied problems in clustering and image segmentation. We analyze the techniques systematically by categorizing them appropriately to provide a structured birds eye view of the work on anomalous region detection;we hope that this would encourage better cross-pollination of ideas across communities to help advance the frontier in anomaly detection.

LS-ADT: Lightweight and Scalable Anomaly Detection for Cloud Datacentres

Cloud data centres are implemented as large-scale clusters with demanding requirements for service performance, availability and cost of operation. As a result of scale and complexity, data centres typically exhibit large numbers of system anomalies resulting from operator error, resource over/under provisioning, hardware or software failures and security issus anomalies are inherently difficult to identify and resolve promptly via human inspection. Therefore, it is vital in a cloud system to have automatic system monitoring that detects potential anomalies and identifies their source. In this paper we present a lightweight anomaly detection tool for Cloud data centres which combines extended log analysis and rigorous correlation of system metrics, implemented by an efficient correlation algorithm which does not require training or complex infrastructure set up. The LADT algorithm is based on the premise that there is a strong correlation between node level and VM level metrics in a cloud system. This correlation will drop significantly in the event of any performance anomaly at the node-level and a continuous drop in the correlation can indicate the presence of a true anomaly in the node. The log analysis of LADT assists in determining whether the correlation drop could be caused by naturally occurring cloud management activity such as VM migration, creation, suspension, termination or resizing. In this way, any potential anomaly alerts are reasoned about to prevent false positives that could be caused by the cloud operator’s activity. We demonstrate LADT with log analysis in a Cloud environment to show how the log analysis is combined with the correlation of systems metrics to achieve accurate anomaly detection.