Feature Selection for Anomaly Detection Using Optical Emission Spectroscopy

To maintain the pace of development set by Moore's law, production processes in semiconductor manufacturing are becoming more and more complex. The development of efficient and interpretable anomaly detection systems is fundamental to keeping production costs low. As the dimension of process monitoring data can become extremely high anomaly detection systems are impacted by the curse of dimensionality, hence dimensionality reduction plays an important role. Classical dimensionality reduction approaches, such as Principal Component Analysis, generally involve transformations that seek to maximize the explained variance. In datasets with several clusters of correlated variables the contributions of isolated variables to explained variance may be insignificant, with the result that they may not be included in the reduced data representation. It is then not possible to detect an anomaly if it is only reflected in such isolated variables. In this paper we present a new dimensionality reduction technique that takes account of such isolated variables and demonstrate how it can be used to build an interpretable and robust anomaly detection system for Optical Emission Spectroscopy data.

Anomaly Detection in Noisy Images

Finding rare events in multidimensional data is an important detection problem that has applications in many fields, such as risk estimation in insurance industry, finance, flood prediction, medical diagnosis, quality assurance, security, or safety in transportation. The occurrence of such anomalies is so infrequent that there is usually not enough training data to learn an accurate statistical model of the anomaly class. In some cases, such events may have never been observed, so the only information that is available is a set of normal samples and an assumed pairwise similarity function. Such metric may only be known up to a certain number of unspecified parameters, which would either need to be learned from training data, or fixed by a domain expert. Sometimes, the anomalous condition may be formulated algebraically, such as a measure exceeding a predefined threshold, but nuisance variables may complicate the estimation of such a measure. Change detection methods used in time series analysis are not easily extendable to the multidimensional case, where discontinuities are not localized to a single point. On the other hand, in higher dimensions, data exhibits more complex interdependencies, and there is redundancy that could be exploited to adaptively model the normal data. In the first part of this dissertation, we review the theoretical framework for anomaly detection in images and previous anomaly detection work done in the context of crack detection and detection of anomalous components in railway tracks. In the second part, we propose new anomaly detection algorithms. The fact that curvilinear discontinuities in images are sparse with respect to the frame of shearlets, allows us to pose this anomaly detection problem as basis pursuit optimization. Therefore, we pose the problem of detecting curvilinear anomalies in noisy textured images as a blind source separation problem under sparsity constraints, and propose an iterative shrinkage algorithm to solve it. Taking advantage of the parallel nature of this algorithm, we describe how this method can be accelerated using graphical processing units (GPU). Then, we propose a new method for finding defective components on railway tracks using cameras mounted on a train. We describe how to extract features and use a combination of classifiers to solve this problem. Then, we scale anomaly detection to bigger datasets with complex interdependencies. We show that the anomaly detection problem naturally fits in the multitask learning framework. The first task consists of learning a compact representation of the good samples, while the second task consists of learning the anomaly detector. Using deep convolutional neural networks, we show that it is possible to train a deep model with a limited number of anomalous examples. In sequential detection problems, the presence of time-variant nuisance parameters affect the detection performance. In the last part of this dissertation, we present a method for adaptively estimating the threshold of sequential detectors using Extreme Value Theory on a Bayesian framework. Finally, conclusions on the results obtained are provided, followed by a discussion of possible future work.

Monoamniotic twins discordant for body stalk anomaly

Body stalk anomaly is a rare malformation. This anomaly in monozygotic twins is extremely unusual. We describe a case of monoamniotic pregnancy discordant for body stalk anomaly diagnosed at 11 weeks. Ultrasound showed a fetus with a large anterior abdominal wall defect, anomaly of the spine and no evidence of lower extremities and other with a normal morphology. As far as our concern, only three monoamniotic pregnancies discordant for this malformation were reported. Our case represents the fourth reported monoamniotic pregnancy discordant for body stalk anomaly with diagnosis made by ultrasound and the second diagnosed in the first trimester.

'Introducing Dendritic Cells as a Novel Immune-Inspired Algorithm for Anomaly Detection'

Abstract. Dendritic cells are antigen presenting cells that provide a vital link between the innate and adaptive immune system. Research into this family of cells has revealed that they perform the role of coordinating T-cell based immune responses, both reactive and for generating tolerance. We have derived an algorithm based on the functionality of these cells, and have used the signals and differentiation pathways to build a control mechanism for an artificial immune system. We present our algorithmic details in addition to some preliminary results, where the algorithm was applied for the purpose of anomaly detection. We hope that this algorithm will eventually become the key component within a large, distributed immune system, based on sound immunological concepts.

Low-Volatility Anomaly in Helsinki Stock Exchange

International research shows that low-volatility stocks have beaten high-volatility stocks in terms of returns for decades on multiple markets. This abbreviation from traditional risk-return framework is known as low-volatility anomaly. This study focuses on explaining the anomaly and finding how strongly it appears in NASDAQ OMX Helsinki stock exchange. Data consists of all listed companies starting from 2001 and ending close to 2015. Methodology follows closely Baker and Haugen (2012) by sorting companies into deciles according to 3-month volatility and then calculating monthly returns for these different volatility groups. Annualized return for the lowest volatility decile is 8.85 %, while highest volatility decile destroys wealth at rate of -19.96 % per annum. Results are parallel also in quintiles that represent larger amount of companies and thus dilute outliers. Observation period captures financial crisis of 2007-2008 and European debt crisis, which embodies as low main index annual return of 1 %, but at the same time proves the success of low-volatility strategy. Low-volatility anomaly is driven by multiple reasons such as leverage constrained trading and managerial incentives which both prompt to invest in risky assets, but behavioral matters also have major weight in maintaining the anomaly.

Magnetic Theory and Applications in the Naples Bay (Southern Tyrrhenian Sea, Italy): Magnetic Anomaly Fields and Relationships with Morpho-Structural Lineaments

Magnetic theory and application to a complex volcanic area located in Southern Italy are here discussed showing the example of the Gulf of Naples, located at Southern Italy Tyrrhenian margin. A magnetic anomaly map of the Gulf of Naples has been constructed aimed at highlighting new knowledge on geophysics and volcanology of this area of the Eastern Tyrrhenian margin, characterized by a complex geophysical setting, strongly depending on sea bottom topography. The theoretical aspects of marine magnetometry and multibeam bathymetry have been discussed. Magnetic data processing included the correction of the data for the diurnal variation, the correction of the data for the offset and the leveling of the data as a function of the correction at the cross-points of the navigation lines. Multibeam and single-beam bathymetric data processing has been considered. Magnetic anomaly fields in the Naples Bay have been discussed through a detailed geological interpretation and correlated with main morpho-structural features recognized through morphobathymetric interpretation. Details of magnetic anomalies have been selected, represented and correlated with significant seismic profiles, recorded on the same navigation lines of magnetometry. They include the continental shelf offshore the Somma-Vesuvius volcanic complex, the outer shelf of the Gulf of Pozzuoli offshore the Phlegrean Fields volcanic complex, the relict volcanic banks of Pentapalummo, Nisida and Miseno, the Gaia volcanic bank on the Naples slope, the western slope of the Dohrn canyon, the Magnaghi canyon’s head and the magnetic anomalies among the Ischia and Procida islands.

Information Fusion for Anomaly Detection with the Dendritic Cell Algorithm

Dendritic cells are antigen presenting cells that provide a vital link between the innate and adaptive immune system, providing the initial detection of pathogenic invaders. Research into this family of cells has revealed that they perform information fusion which directs immune responses. We have derived a Dendritic Cell Algorithm based on the functionality of these cells, by modelling the biological signals and differentiation pathways to build a control mechanism for an artificial immune system. We present algorithmic details in addition to experimental results, when the algorithm was applied to anomaly detection for the detection of port scans. The results show the Dendritic Cell Algorithm is successful at detecting port scans.

Introducting Dendritic Cells as a Novel Immune-Inspired Algorithm for Anomaly Detection

Dendritic cells are antigen presenting cells that provide a vital link between the innate and adaptive immune system. Research into this family of cells has revealed that they perform the role of coordinating T-cell based immune responses, both reactive and for generating tolerance. We have derived an algorithm based on the functionality of these cells, and have used the signals and differentiation pathways to build a control mechanism for an artificial immune system. We present our algorithmic details in addition to some preliminary results, where the algorithm was applied for the purpose of anomaly detection. We hope that this algorithm will eventually become the key component within a large, distributed immune system, based on sound imnological concepts.

An Immune Inspired Approach to Anomaly Detection

The immune system provides a rich metaphor for computer security: anomaly detection that works in nature should work for machines. However, early artificial immune system approaches for computer security had only limited success. Arguably, this was due to these artificial systems being based on too simplistic a view of the immune system. We present here a second generation artificial immune system for process anomaly detection. It improves on earlier systems by having different artificial cell types that process information. Following detailed information about how to build such second generation systems, we find that communication between cells types is key to performance. Through realistic testing and validation we show that second generation artificial immune systems are capable of anomaly detection beyond generic system policies. The paper concludes with a discussion and outline of the next steps in this exciting area of computer security.

Dempster-Shafer for Anomaly Detection

In this paper, we implement an anomaly detection system using the Dempster-Shafer method. Using two standard benchmark problems we show that by combining multiple signals it is possible to achieve better results than by using a single signal. We further show that by applying this approach to a real-world email dataset the algorithm works for email worm detection. Dempster-Shafer can be a promising method for anomaly detection problems with multiple features (data sources), and two or more classes.

Dendritic Cells for Real-Time Anomaly Detection

Dendritic Cells (DCs) are innate immune system cells which have the power to activate or suppress the immune system. The behaviour of human DCs is abstracted to form an algorithm suitable for anomaly detection. We test this algorithm on the real-time problem of port scan detection. Our results show a significant difference in artificial DC behaviour for an outgoing portscan when compared to behaviour for normal processes.

Dendritic Cells for Anomaly Detection

Artificial immune systems, more specifically the negative selection algorithm, have previously been applied to intrusion detection. The aim of this research is to develop an intrusion detection system based on a novel concept in immunology, the Danger Theory. Dendritic Cells (DCs) are antigen presenting cells and key to the activation of the human immune system. DCs perform the vital role of combining signals from the host tissue and correlate these signals with proteins known as antigens. In algorithmic terms, individual DCs perform multi-sensor data fusion based on time-windows. The whole population of DCs asynchronously correlates the fused signals with a secondary data stream. The behaviour of human DCs is abstracted to form the DC Algorithm (DCA), which is implemented using an immune inspired framework, libtissue. This system is used to detect context switching for a basic machine learning dataset and to detect outgoing portscans in real-time. Experimental results show a significant difference between an outgoing portscan and normal traffic.

'Introducing Dendritic Cells as a Novel Immune-Inspired Algorithm for Anomaly Detection'

Abstract. Dendritic cells are antigen presenting cells that provide a vital link between the innate and adaptive immune system. Research into this family of cells has revealed that they perform the role of coordinating T-cell based immune responses, both reactive and for generating tolerance. We have derived an algorithm based on the functionality of these cells, and have used the signals and differentiation pathways to build a control mechanism for an artificial immune system. We present our algorithmic details in addition to some preliminary results, where the algorithm was applied for the purpose of anomaly detection. We hope that this algorithm will eventually become the key component within a large, distributed immune system, based on sound immunological concepts.

Anomaly diagnosis in ceramic claddings by thermography - A review

With the increasing importance given to building rehabilitation comes the need to create simple, fast and non-destructive testing methods (NDT) to identify problems and for anomaly diagnosis. Ceramic tiles are one of the most typical kinds of exterior wall cladding in several countries; the earliest known examples are Egyptian dating from 4000 BC. This type of building facade coating, though being quite often used in due to its aesthetic and architectural characteristics, is one of the most complex that can be applied given the several parts from which it is composed; hence, it is also one of the most difficult to correctly diagnose with expeditious methods. The detachment of ceramic wall tiles is probably the most common and difficult to identify anomaly associated with this kind of cladding and it is also definitely the one that can compromise security the most. Thus, it is necessary to study a process of inspection more efficient and economic than the currently used which often consist in semi-destructive methods (the most common is the pull off test), that can only be used in a small part of the building at a time, allowing some assumptions of what can the rest of the cladding be like. Infrared thermography (IRT) is a NDT with a wide variety of applications in building inspection that is becoming commonly used to identify anomalies related with thermal variations in the inspected surfaces. Few authors have studied the application of IRT in anomalies associated with ceramic claddings claiming that the presence of air or water beneath the superficial layer will influence the heat transfer in a way that can be detected in both a qualitative and a quantitative way by the thermal camera, providing information about the state of the wall in a much broad area per trial than other methods commonly used nowadays. This article intends to present a review of the state of art of this NDT and its potentiality in becoming a more efficient way to diagnose anomalies in ceramic wall claddings.