Time correlated anomaly detection based on inferences

Anomaly detection techniques are used to find the presence of anomalous activities in a network by comparing traffic data activities against a "normal" baseline. Although it has several advantages which include detection of "zero-day" attacks, the question surrounding absolute definition of systems deviations from its "normal" behaviour is important to reduce the number of false positives in the system. This study proposes a novel multi-agent network-based framework known as Statistical model for Correlation and Detection (SCoDe), an anomaly detection framework that looks for timecorrelated anomalies by leveraging statistical properties of a large network, monitoring the rate of events occurrence based on their intensity. SCoDe is an instantaneous learning-based anomaly detector, practically shifting away from the conventional technique of having a training phase prior to detection. It does acquire its training using the improved extension of Exponential Weighted Moving Average (EWMA) which is proposed in this study. SCoDe does not require any previous knowledge of the network traffic, or network administrators chosen reference window as normal but effectively builds upon the statistical properties from different attributes of the network traffic, to correlate undesirable deviations in order to identify abnormal patterns. The approach is generic as it can be easily modified to fit particular types of problems, with a predefined attribute, and it is highly robust because of the proposed statistical approach. The proposed framework was targeted to detect attacks that increase the number of activities on the network server, examples which include Distributed Denial of Service (DDoS) and, flood and flash-crowd events. This paper provides a mathematical foundation for SCoDe, describing the specific implementation and testing of the approach based on a network log file generated from the cyber range simulation experiment of the industrial partner of this project.