Constructing detectors in schema complementary space for anomaly detection

This paper proposes an extended negative selection algorithm for anomaly detection. Unlike previously proposed negative selection algorithms which directly construct detectors in the complementary space of self-data space, our approach first evolves a number of common schemata through coevolutionary genetic algorithm in self-data space, and then constructs detectors in the complementary space of the schemata. These common schemata characterize self-data space and thus guide the generation of detection rules. By converting data space into schema space, we can efficiently generate an appropriate number of detectors with diversity for anomaly detection. The approach is tested for its effectiveness through experiment with the published data set iris.

An extended negative selection algorithm for anomaly detection

This paper proposes an extended negative selection algorithm for anomaly detection. Unlike previously proposed negative selection algorithms which do not make use of non-self data, the extended negative selection algorithm first acquires prior knowledge about the characteristics of the Problem space from the historial sample data by using machine learning techniques. Such data consists of both self data and non-self data. The acquired prior knowledge is represented in the form of production rules and thus viewed as common schemata which characterise the two subspaces: self-subspace and non-self-subspace, and provide important information to the generation of detection rules. One advantage of our approach is that it does not rely on the structured representation of the data and can be applied to general anomaly detection. To test the effectiveness, we test our approach through experiments with the public data set iris and KDDrsquo99 published data set.

Applying both positive and negative selection to supervised learning for anomaly detection

This paper presents a novel approach of applying both positive selection and negative selection to supervised learning for anomaly detection. It first learns the patterns of the normal class via co-evolutionary genetic algorithm, which is inspired from the positive selection, and then generates synthetic samples of the anomaly class, which is based on the negative selection in the immune system. Two algorithms about synthetic generation of the anomaly class are proposed. One deals with data sets containing a few anomalous samples; while the other deals with data sets containing no anomalous samples at all. The experimental results on some benchmark data sets from UCI data set repertory show that the detection rate is improved evidently, accompanied by a slight increase in false alarm rate via introducing novel synthetic samples of the anomaly class. The advantages of our method are the increased ability of classifiers in identifying both previously known and innovative anomalies, and the maximal degradation of overfitting phenomenon.

Immunology-based subspace detectors for anomaly detection

A key problem in high dimensional anomaly detection is that the time spent in constructing detectors by the means of generateand-test is tolerable. In fact, due to the high sparsity. of the data, it is ineffective to construct detectors in the whole data space. Previous investigations have shown that most essentIal patterns can be discovered in different subspaces. This inspires us to construct detectors in signIficant subspaces only for anomaly detection. We first use ENCLUS-based method to discover all significant subspaces and .then use a greedy-growth algorithm to construct detectors in each subspace. The elements used to constItute a detector are gods Instead of data points, which makes the time-consumption irrelevant to the size of the nonnal data. We test the effectiveness and efficiency of our method on both synthetic and benchmark datasets. The results reveal that our method is particularly useful in anomaly detection in high dimensional data spaces.

The halloween effect and Japanese equity prices : myth or exploitable anomaly

Bouman and Jacobsen (American Economic Review 92(5), 1618–1635, 2002) examine monthly stock returns for major world stock markets and conclude that returns are significantly lower during the May–October periods versus the November–April periods in 36 of 37 markets examined. They argue that, in general, the Halloween strategy outperforms the buy and hold strategy thereby casting doubt on the validity of the efficient market paradigm. More recently, Maberly and Pierce (Econ Journal Watch 1(1), 29–46, 2004) re-examine the evidence for U.S. equity prices and conclude that Bouman and Jacobsen’s results are not robust to alternative model specifications. Extending prior research, this paper examines the robustness of the Halloween strategy to alternative model specifications for Japanese equity prices. The Halloween effect is concentrated in the period prior to the introduction of Nikkei 225 index futures in September 1986. After the internationalization of Japanese financial markets in the mid-1980s, the Halloween effect disappears.

Computational immunology for anomaly detection

The thesis makes a significant contribution to the issue of anomaly detection by introducing a computational immunology approach. Immunity-based anomaly detection in high dimensional space is systematically investigated and the proposed hybrid method (combining data mining techniques and computational immunology) improves both accuracy and efficiency.

Effective anomaly detection in sensor networks data streams

This paper addresses a major challenge in data mining applications where the full information about the underlying processes, such as sensor networks or large online database, cannot be practically obtained due to physical limitations such as low bandwidth or memory, storage, or computing power. Motivated by the recent theory on direct information sampling called compressed sensing (CS), we propose a framework for detecting anomalies from these largescale data mining applications where the full information is not practically possible to obtain. Exploiting the fact that the intrinsic dimension of the data in these applications are typically small relative to the raw dimension and the fact that compressed sensing is capable of capturing most information with few measurements, our work show that spectral methods that used for volume anomaly detection can be directly applied to the CS data with guarantee on performance. Our theoretical contributions are supported by extensive experimental results on large datasets which show satisfactory performance.

A hierarchical PCA-based anomaly detection model

A hierarchical intrusion detection model is proposed to detect both anomaly and misuse attacks. In order to further speed up the training and testing, PCA-based feature extraction algorithm is used to reduce the dimensionality of the data. A PCA-based algorithm is used to filter normal data out in the upper level. The experiment results show that PCA can reduce noise in the original data set and the PCA-based algorithm can reach the desirable performance.

Time correlated anomaly detection based on inferences

Anomaly detection techniques are used to find the presence of anomalous activities in a network by comparing traffic data activities against a "normal" baseline. Although it has several advantages which include detection of "zero-day" attacks, the question surrounding absolute definition of systems deviations from its "normal" behaviour is important to reduce the number of false positives in the system. This study proposes a novel multi-agent network-based framework known as Statistical model for Correlation and Detection (SCoDe), an anomaly detection framework that looks for timecorrelated anomalies by leveraging statistical properties of a large network, monitoring the rate of events occurrence based on their intensity. SCoDe is an instantaneous learning-based anomaly detector, practically shifting away from the conventional technique of having a training phase prior to detection. It does acquire its training using the improved extension of Exponential Weighted Moving Average (EWMA) which is proposed in this study. SCoDe does not require any previous knowledge of the network traffic, or network administrators chosen reference window as normal but effectively builds upon the statistical properties from different attributes of the network traffic, to correlate undesirable deviations in order to identify abnormal patterns. The approach is generic as it can be easily modified to fit particular types of problems, with a predefined attribute, and it is highly robust because of the proposed statistical approach. The proposed framework was targeted to detect attacks that increase the number of activities on the network server, examples which include Distributed Denial of Service (DDoS) and, flood and flash-crowd events. This paper provides a mathematical foundation for SCoDe, describing the specific implementation and testing of the approach based on a network log file generated from the cyber range simulation experiment of the industrial partner of this project.

Wingbeat frequency and the body drag anomaly: wind-tunnel observations on a thrush nightingale (Luscinia Luscinia) and a teal (Anas crecca)

A teal (Anas crecca) and a thrush nightingale (Luscinia luscinia) were trained to fly in the Lund wind tunnel for periods of up to 3 and 16 h respectively. Both birds flew in steady flapping flight, with such regularity that their wingbeat frequencies could be determined by viewing them through a shutter stroboscope. When flying at a constant air speed, the teal's wingbeat frequency varied with the 0.364 power of the body mass and the thrush nightingale's varied with the 0.430 power. Both exponents differed from zero, but neither differed from the predicted value (0.5) at the 1 % level of significance. The teal continued to flap steadily as the tunnel tilt angle was varied from -1° (climb) to +6° (descent), while the wingbeat frequency declined progressively by about 11%. In both birds, the plot of wingbeat frequency against air speed in level flight was U-shaped, with small but statistically significant curvature. We identified the minima of these curves with the minimum power speed (Vmp) and found that the values predicted for Vmp, using previously published default values for the required variables, were only about two-thirds of the observed minimum-frequency speeds. The discrepancy could be resolved if the body drag coefficients (CDb) of both birds were near 0.08, rather than near 0.40 as previously assumed. The previously published high values for body drag coefficients were derived from wind-tunnel measurements on frozen bird bodies, from which the wings had been removed, and had long been regarded as anomalous, as values below 0.01 are given in the engineering literature for streamlined bodies. We suggest that birds of any size that have well-streamlined bodies can achieve minimum body drag coefficients of around 0.05 if the feet can be fully retracted under the flank feathers. In such birds, field observations of flight speeds may need to be reinterpreted in the light of higher estimates of Vmp. Estimates of the effective lift:drag ratio and range can also be revised upwards. Birds that have large feet or trailing legs may have higher body drag coefficients. The original estimates of around CDb=0.4 could be correct for species, such as pelicans and large herons, that also have prominent heads. We see no evidence for any progressive reduction of body drag coefficient in the Reynolds number range covered by our experiments, that is 21600-215 000 on the basis of body cross-sectional diameter.

High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

High-dimensional problem domains pose significant challenges for anomaly detection. The presence of irrelevant features can conceal the presence of anomalies. This problem, known as the '. curse of dimensionality', is an obstacle for many anomaly detection techniques. Building a robust anomaly detection model for use in high-dimensional spaces requires the combination of an unsupervised feature extractor and an anomaly detector. While one-class support vector machines are effective at producing decision surfaces from well-behaved feature vectors, they can be inefficient at modelling the variation in large, high-dimensional datasets. Architectures such as deep belief networks (DBNs) are a promising technique for learning robust features. We present a hybrid model where an unsupervised DBN is trained to extract generic underlying features, and a one-class SVM is trained from the features learned by the DBN. Since a linear kernel can be substituted for nonlinear ones in our hybrid model without loss of accuracy, our model is scalable and computationally efficient. The experimental results show that our proposed model yields comparable anomaly detection performance with a deep autoencoder, while reducing its training and testing time by a factor of 3 and 1000, respectively.