Anomaly Detection for Data with Spatial Attributes

The problem of detecting spatially-coherent groups of data that exhibit anomalous behavior has started to attract attention due to applications across areas such as epidemic analysis and weather forecasting. Earlier efforts from the data mining community have largely focused on finding outliers, individual data objects that display deviant behavior. Such point-based methods are not easy to extend to find groups of data that exhibit anomalous behavior. Scan Statistics are methods from the statistics community that have considered the problem of identifying regions where data objects exhibit a behavior that is atypical of the general dataset. The spatial scan statistic and methods that build upon it mostly adopt the framework of defining a character for regions (e.g., circular or elliptical) of objects and repeatedly sampling regions of such character followed by applying a statistical test for anomaly detection. In the past decade, there have been efforts from the statistics community to enhance efficiency of scan statstics as well as to enable discovery of arbitrarily shaped anomalous regions. On the other hand, the data mining community has started to look at determining anomalous regions that have behavior divergent from their neighborhood.In this chapter,we survey the space of techniques for detecting anomalous regions on spatial data from across the data mining and statistics communities while outlining connections to well-studied problems in clustering and image segmentation. We analyze the techniques systematically by categorizing them appropriately to provide a structured birds eye view of the work on anomalous region detection;we hope that this would encourage better cross-pollination of ideas across communities to help advance the frontier in anomaly detection.

LS-ADT: Lightweight and Scalable Anomaly Detection for Cloud Datacentres

Cloud data centres are implemented as large-scale clusters with demanding requirements for service performance, availability and cost of operation. As a result of scale and complexity, data centres typically exhibit large numbers of system anomalies resulting from operator error, resource over/under provisioning, hardware or software failures and security issus anomalies are inherently difficult to identify and resolve promptly via human inspection. Therefore, it is vital in a cloud system to have automatic system monitoring that detects potential anomalies and identifies their source. In this paper we present a lightweight anomaly detection tool for Cloud data centres which combines extended log analysis and rigorous correlation of system metrics, implemented by an efficient correlation algorithm which does not require training or complex infrastructure set up. The LADT algorithm is based on the premise that there is a strong correlation between node level and VM level metrics in a cloud system. This correlation will drop significantly in the event of any performance anomaly at the node-level and a continuous drop in the correlation can indicate the presence of a true anomaly in the node. The log analysis of LADT assists in determining whether the correlation drop could be caused by naturally occurring cloud management activity such as VM migration, creation, suspension, termination or resizing. In this way, any potential anomaly alerts are reasoned about to prevent false positives that could be caused by the cloud operator’s activity. We demonstrate LADT with log analysis in a Cloud environment to show how the log analysis is combined with the correlation of systems metrics to achieve accurate anomaly detection.

Developments of the cellular frustration approach to anomaly detection

Neste trabalho é desenvolvido um método de detecção de anomalias, baseado no mecanismo da frustração celular. Este método é capaz de detectar com grande precisão desvios de um comportamento característico de um sistema complexo. Estes desvios podem ser devidos a intrusões ou a anomalias no seu funcionamento. O método propõe ainda uma compreensão alternativa de diversos fenómenos observados em Imunologia.

Maternal coping, appraisals and adjustment following diagnosis of fetal anomaly.

OBJECTIVE: So far, associations between appraisals, maternal adjustment and coping following diagnosis of fetal anomaly have not been investigated in women who continue with their pregnancy. METHOD: This study measured maternal coping and adjustment after and appraisal of a diagnosis of fetal anomaly in 40 mothers who had continued with their pregnancy using a cross-sectional questionnaire design. RESULTS: Based on retrospective reporting, 35% of participants met full diagnostic criteria for post-traumatic stress disorder after having received the diagnosis. Women were significantly more depressed (p < 0.001) and anxious (p < 0.001) and reported significantly less positive affect (p < 0.05) after having received the diagnosis in comparison to the time after childbirth. There were no significant differences between emotion-focused and problem-focused coping. Stressful life events, women's age, number of people providing support and problem-focused coping explained 57.6% of variance in anxiety and depression after childbirth. Satisfaction with social support, emotion-focused coping and problem-focused coping significantly explained 40.6% of variance in positive affect after childbirth. CONCLUSION: Following a prenatal diagnosis and for the remainder of their pregnancy, particular attention should be paid to older mothers, those experiencing additional stressful life events and those who are socially isolated, as these women may experience greater distress after childbirth.

Rare chromosome abnormalities, prevalence and prenatal diagnosis rates from population-based congenital anomaly registers in Europe.

The aim of this study is to quantify the prevalence and types of rare chromosome abnormalities (RCAs) in Europe for 2000-2006 inclusive, and to describe prenatal diagnosis rates and pregnancy outcome. Data held by the European Surveillance of Congenital Anomalies database were analysed on all the cases from 16 population-based registries in 11 European countries diagnosed prenatally or before 1 year of age, and delivered between 2000 and 2006. Cases were all unbalanced chromosome abnormalities and included live births, fetal deaths from 20 weeks gestation and terminations of pregnancy for fetal anomaly. There were 10,323 cases with a chromosome abnormality, giving a total birth prevalence rate of 43.8/10,000 births. Of these, 7335 cases had trisomy 21,18 or 13, giving individual prevalence rates of 23.0, 5.9 and 2.3/10,000 births, respectively (53, 13 and 5% of all reported chromosome errors, respectively). In all, 473 cases (5%) had a sex chromosome trisomy, and 778 (8%) had 45,X, giving prevalence rates of 2.0 and 3.3/10,000 births, respectively. There were 1,737 RCA cases (17%), giving a prevalence of 7.4/10,000 births. These included triploidy, other trisomies, marker chromosomes, unbalanced translocations, deletions and duplications. There was a wide variation between the registers in both the overall prenatal diagnosis rate of RCA, an average of 65% (range 5-92%) and the prevalence of RCA (range 2.4-12.9/10,000 births). In all, 49% were liveborn. The data provide the prevalence of families currently requiring specialised genetic counselling services in the perinatal period for these conditions and, for some, long-term care.

Ebstein's anomaly: one and a half ventricular repair.

Patients with Ebstein's anomaly can present after childhood or adolescence with cyanosis, arrhythmias, severe right ventricular dysfunction and frequently with left ventricular dysfunction secondary to the prolonged cyanosis and to the right ventricular interference. At this point conventional repair is accompanied by elevated mortality and morbidity and poor functional results. We report our experience with three patients (8, 16 and 35 years of age) with Ebstein's anomaly, very dilated right atrium, severe tricuspid valve regurgitation (4/4), bi-directional shunt through an atrial septal defect and reduced left ventricular function (mean ejection fraction = 58%, mean shortening fraction = 25%). All underwent one and a half ventricular repair consisting of closure of the atrial septal defect, tricuspid repair with reduction of the atrialised portion of the right ventricle and end-to-side anastomosis of the superior vena cava to the right pulmonary artery. All patients survived, with a mean follow-up of 33 months. In all there was complete regression of the cyanosis and of the signs of heart failure. Postoperative echocardiography showed reduced degree of tricuspid regurgitation (2/4) and improvement of the left ventricular function (mean ejection fraction = 77%, mean shortening fraction = 40%). In patients with Ebstein's anomaly referred late for surgery with severely compromised right ventricular function or even with reduced biventricular function, the presence of a relatively hypoplastic and/or malfunctioning right ventricular chamber inadequate to sustain the entire systemic venous return but capable of managing part of the systemic venous return, permits a one and a half ventricular repair with good functional results.

A hypothesis for the geochemical anomaly in the North Creek Watershed /

A regional geochemical reconnaissance by bottom stream sediment sampling, has delineated an area of high metal content in the north central sector of the North Creek Watershed. Development of a geochemical model, relating to the relative chemical concentrations derived from the chemical analyses of bottom sediments, suspended sediments, stream waters and well waters collected from the north central sector, was designed to discover the source of the anomaly. Samples of each type of material were analysed by the A.R.L. Direct Reading Multi-element Emission Spectrograph Q.A. 137 for elements: Na, K, Ca, Sr, Si, As, Pb, Zn, Cd, Ni, Ti, Ag, Mo, Be, Fe, AI, Mn, Cu, Cr, P and Y. Anomalous results led to the discovery of a spring, the waters of which carried high concentrations of Zn, Cd, Pb, As, Ni, Ti, Ag, Sr and Si. In addition, the spring waters had high concentrations of Na, Ca, Mg, 504 , alkalinity, N03' and low concentrations of K, Cl and NH3. Increased specific conductivity (up to 2500 ~mho/cm.) was noted in the spring waters as well as increased calculated total dissolved solids (up to 2047 mg/l) and increased ionic strength (up to 0.06). On the other hand, decreases were noted in water temperature (8°C), pH (pH 7.2) and Eh (+.154 volts). Piezometer nests were installed in the anomalous north central sector of the watershed. In accordance with the slope of the piezometric surface from wells cased down to the till/bedrock interface, groundwater flow is directed from the recharge area (northwest of the anomaly) towards the artesian spring via the highly fractured dolostone aquifer of the Upper Eramosa Member. The bedrock aquifer is confined by the overlying Halton till and the underlying Lower Eramosa Member (Vinemount Shale). The oxidation of sphalerite and galena and the dissolution of gypsum, celestite, calcite, and dolomite within the Eramosa Member, contributed its highly, dissolved constituents to the circulating groundwaters, the age of which is greater than 20 years as determined by tritium dating. Groundwater is assumed to flow along the Vinemount Shale and discharge as an artesian spring where the shale unit becomes discontinuous. The anomaly is located on a topographic low where bedrock is close to the surface. Thermodynamic evaluation of the major ion speciation from the anomalous spring and surface waters, showed gypsum to be supersaturated in these spring waters. Downstream from the spring, the loss of carbon dioxide from the spring waters resulted in the supersaturation with respect to calcite, aragonite, magnesite and dolomite. This corresponded with increases in Eh (+.304 volts) and pH (pH 8.5) in the anomalous surface waters. In conclusion, the interaction of groundwaters within the highly, mineralized carbonate source (Eramosa Member) resulted in the characteristic Ca*Mg*HC03*S04 spring water at the anomalous site, which appeared to be the principle effect upon controlling the anomalous surface water chemistry.

Anomaly Detection Using System Call Sequence Sets.

This paper discusses our research in developing a generalized and systematic method for anomaly detection. The key ideas are to represent normal program behaviour using system call frequencies and to incorporate probabilistic techniques for classification to detect anomalies and intrusions. Using experiments on the sendmail system call data, we demonstrate that concise and accurate classifiers can be constructed to detect anomalies. An overview of the approach that we have implemented is provided.

Adaptive Real-time Anomaly-based Intrusion Detection using Data Mining and Machine Learning Techniques

Die zunehmende Vernetzung der Informations- und Kommunikationssysteme führt zu einer weiteren Erhöhung der Komplexität und damit auch zu einer weiteren Zunahme von Sicherheitslücken. Klassische Schutzmechanismen wie Firewall-Systeme und Anti-Malware-Lösungen bieten schon lange keinen Schutz mehr vor Eindringversuchen in IT-Infrastrukturen. Als ein sehr wirkungsvolles Instrument zum Schutz gegenüber Cyber-Attacken haben sich hierbei die Intrusion Detection Systeme (IDS) etabliert. Solche Systeme sammeln und analysieren Informationen von Netzwerkkomponenten und Rechnern, um ungewöhnliches Verhalten und Sicherheitsverletzungen automatisiert festzustellen. Während signatur-basierte Ansätze nur bereits bekannte Angriffsmuster detektieren können, sind anomalie-basierte IDS auch in der Lage, neue bisher unbekannte Angriffe (Zero-Day-Attacks) frühzeitig zu erkennen. Das Kernproblem von Intrusion Detection Systeme besteht jedoch in der optimalen Verarbeitung der gewaltigen Netzdaten und der Entwicklung eines in Echtzeit arbeitenden adaptiven Erkennungsmodells. Um diese Herausforderungen lösen zu können, stellt diese Dissertation ein Framework bereit, das aus zwei Hauptteilen besteht. Der erste Teil, OptiFilter genannt, verwendet ein dynamisches "Queuing Concept", um die zahlreich anfallenden Netzdaten weiter zu verarbeiten, baut fortlaufend Netzverbindungen auf, und exportiert strukturierte Input-Daten für das IDS. Den zweiten Teil stellt ein adaptiver Klassifikator dar, der ein Klassifikator-Modell basierend auf "Enhanced Growing Hierarchical Self Organizing Map" (EGHSOM), ein Modell für Netzwerk Normalzustand (NNB) und ein "Update Model" umfasst. In dem OptiFilter werden Tcpdump und SNMP traps benutzt, um die Netzwerkpakete und Hostereignisse fortlaufend zu aggregieren. Diese aggregierten Netzwerkpackete und Hostereignisse werden weiter analysiert und in Verbindungsvektoren umgewandelt. Zur Verbesserung der Erkennungsrate des adaptiven Klassifikators wird das künstliche neuronale Netz GHSOM intensiv untersucht und wesentlich weiterentwickelt. In dieser Dissertation werden unterschiedliche Ansätze vorgeschlagen und diskutiert. So wird eine classification-confidence margin threshold definiert, um die unbekannten bösartigen Verbindungen aufzudecken, die Stabilität der Wachstumstopologie durch neuartige Ansätze für die Initialisierung der Gewichtvektoren und durch die Stärkung der Winner Neuronen erhöht, und ein selbst-adaptives Verfahren eingeführt, um das Modell ständig aktualisieren zu können. Darüber hinaus besteht die Hauptaufgabe des NNB-Modells in der weiteren Untersuchung der erkannten unbekannten Verbindungen von der EGHSOM und der Überprüfung, ob sie normal sind. Jedoch, ändern sich die Netzverkehrsdaten wegen des Concept drif Phänomens ständig, was in Echtzeit zur Erzeugung nicht stationärer Netzdaten führt. Dieses Phänomen wird von dem Update-Modell besser kontrolliert. Das EGHSOM-Modell kann die neuen Anomalien effektiv erkennen und das NNB-Model passt die Änderungen in Netzdaten optimal an. Bei den experimentellen Untersuchungen hat das Framework erfolgversprechende Ergebnisse gezeigt. Im ersten Experiment wurde das Framework in Offline-Betriebsmodus evaluiert. Der OptiFilter wurde mit offline-, synthetischen- und realistischen Daten ausgewertet. Der adaptive Klassifikator wurde mit dem 10-Fold Cross Validation Verfahren evaluiert, um dessen Genauigkeit abzuschätzen. Im zweiten Experiment wurde das Framework auf einer 1 bis 10 GB Netzwerkstrecke installiert und im Online-Betriebsmodus in Echtzeit ausgewertet. Der OptiFilter hat erfolgreich die gewaltige Menge von Netzdaten in die strukturierten Verbindungsvektoren umgewandelt und der adaptive Klassifikator hat sie präzise klassifiziert. Die Vergleichsstudie zwischen dem entwickelten Framework und anderen bekannten IDS-Ansätzen zeigt, dass der vorgeschlagene IDSFramework alle anderen Ansätze übertrifft. Dies lässt sich auf folgende Kernpunkte zurückführen: Bearbeitung der gesammelten Netzdaten, Erreichung der besten Performanz (wie die Gesamtgenauigkeit), Detektieren unbekannter Verbindungen und Entwicklung des in Echtzeit arbeitenden Erkennungsmodells von Eindringversuchen.

Land surface anomaly simulations and predictions with a climate model: an El Nino Southern Oscillation case study

The ability of climate models to reproduce and predict land surface anomalies is an important but little-studied topic. In this study, an atmosphere and ocean assimilation scheme is used to determine whether HadCM3 can reproduce and predict snow water equivalent and soil moisture during the 1997–1998 El Nino Southern Oscillation event. Soil moisture is reproduced more successfully, though both snow and soil moisture show some predictability at 1- and 4-month lead times. This result suggests that land surface anomalies may be reasonably well initialized for climate model predictions and hydrological applications using atmospheric assimilation methods over a period of time.

Can desert dust explain the outgoing longwave radiation anomaly over the Sahara during July 2003?

Measurements of the top‐of‐the‐atmosphere outgoing longwave radiation (OLR) for July 2003 from Meteosat‐7 are used to assess the performance of the numerical weather prediction version of the Met Office Unified Model. A significant difference is found over desert regions of northern Africa where the model emits too much OLR by up to 35 Wm−2 in the monthly mean. By cloud‐screening the data we find an error of up to 50 Wm−2 associated with cloud‐free areas, which suggests an error in the model surface temperature, surface emissivity, or atmospheric transmission. By building up a physical model of the radiative properties of mineral dust based on in situ, and surface‐based and satellite remote sensing observations we show that the most plausible explanation for the discrepancy in OLR is due to the neglect of mineral dust in the model. The calculations suggest that mineral dust can exert a longwave radiative forcing by as much as 50 Wm−2 in the monthly mean for 1200 UTC in cloud‐free regions, which accounts for the discrepancy between the model and the Meteosat‐7 observations. This suggests that inclusion of the radiative effects of mineral dust will lead to a significant improvement in the radiation balance of numerical weather prediction models with subsequent improvements in performance.

Observation and origin of an interannual salinity anomaly in the Mozambique Channel

A positive salinity anomaly of 0.2 PSU was observed between 50 and 200 m over the years 2000–2001 across the Mozambique Channel at a section at 17°S which was repeated in 2003, 2005, 2006, and 2008. Meanwhile, a moored array is continued from 2003 to 2008. This anomaly was most distinct showing an interannual but nonseasonal variation. The possible origin of the anomaly is investigated using output from three ocean general circulation models (Estimating the Circulation and Climate of the Ocean, Ocean Circulation and Climate Advanced Modeling, and Parallel Ocean Program). The most probable mechanism for the salinity anomaly is the anomalous inflow of subtropical waters caused by a weakening of the northern part of the South Equatorial Current by weaker trade winds. This mechanism was found in all three numerical models. In addition, the numerical models indicate a possible salinization of one of the source water masses to the Mozambique Channel as an additional cause of the anomaly. The anomaly propagated southward into the Agulhas Current and northward along the African coast.

Idealized SST anomaly regional climate model experiments: a note of caution

To date, a number of studies have focused on the influence of sea surface temperature (SST) on global and regional rainfall variability, with the majority of these focusing on certain ocean basins e.g. the Pacific, North Atlantic and Indian Ocean. In contrast, relatively less work has been done on the influence of the central South Atlantic, particularly in relation to rainfall over southern Africa. Previous work by the authors, using reanalysis data and general circulation model (GCM) experiments, has suggested that cold SST anomalies in the central southern Atlantic Ocean are linked to an increase in rainfall extremes across southern Africa. In this paper we present results from idealised regional climate model (RCM) experiments forced with both positive and negative SST anomalies in the southern Atlantic Ocean. These experiments reveal an unexpected response of rainfall over southern Africa. In particular it was found that SST anomalies of opposite sign can cause similar rainfall responses in the model experiments, with isolated increases in rainfall over central southern Africa as well as a large region of drying over the Mozambique Channel. The purpose of this paper is to highlight this finding and explore explanations for the behaviour of the climate model. It is suggested that the observed changes in rainfall might result from the redistribution of energy (associated with upper level changes to Rossby waves) or, of more concern, model error, and therefore the paper concludes that the results of idealised regional climate models forced with SST anomalies should be viewed cautiously.