Adaptive Real-time Anomaly-based Intrusion Detection using Data Mining and Machine Learning Techniques

Die zunehmende Vernetzung der Informations- und Kommunikationssysteme führt zu einer weiteren Erhöhung der Komplexität und damit auch zu einer weiteren Zunahme von Sicherheitslücken. Klassische Schutzmechanismen wie Firewall-Systeme und Anti-Malware-Lösungen bieten schon lange keinen Schutz mehr vor Eindringversuchen in IT-Infrastrukturen. Als ein sehr wirkungsvolles Instrument zum Schutz gegenüber Cyber-Attacken haben sich hierbei die Intrusion Detection Systeme (IDS) etabliert. Solche Systeme sammeln und analysieren Informationen von Netzwerkkomponenten und Rechnern, um ungewöhnliches Verhalten und Sicherheitsverletzungen automatisiert festzustellen. Während signatur-basierte Ansätze nur bereits bekannte Angriffsmuster detektieren können, sind anomalie-basierte IDS auch in der Lage, neue bisher unbekannte Angriffe (Zero-Day-Attacks) frühzeitig zu erkennen. Das Kernproblem von Intrusion Detection Systeme besteht jedoch in der optimalen Verarbeitung der gewaltigen Netzdaten und der Entwicklung eines in Echtzeit arbeitenden adaptiven Erkennungsmodells. Um diese Herausforderungen lösen zu können, stellt diese Dissertation ein Framework bereit, das aus zwei Hauptteilen besteht. Der erste Teil, OptiFilter genannt, verwendet ein dynamisches "Queuing Concept", um die zahlreich anfallenden Netzdaten weiter zu verarbeiten, baut fortlaufend Netzverbindungen auf, und exportiert strukturierte Input-Daten für das IDS. Den zweiten Teil stellt ein adaptiver Klassifikator dar, der ein Klassifikator-Modell basierend auf "Enhanced Growing Hierarchical Self Organizing Map" (EGHSOM), ein Modell für Netzwerk Normalzustand (NNB) und ein "Update Model" umfasst. In dem OptiFilter werden Tcpdump und SNMP traps benutzt, um die Netzwerkpakete und Hostereignisse fortlaufend zu aggregieren. Diese aggregierten Netzwerkpackete und Hostereignisse werden weiter analysiert und in Verbindungsvektoren umgewandelt. Zur Verbesserung der Erkennungsrate des adaptiven Klassifikators wird das künstliche neuronale Netz GHSOM intensiv untersucht und wesentlich weiterentwickelt. In dieser Dissertation werden unterschiedliche Ansätze vorgeschlagen und diskutiert. So wird eine classification-confidence margin threshold definiert, um die unbekannten bösartigen Verbindungen aufzudecken, die Stabilität der Wachstumstopologie durch neuartige Ansätze für die Initialisierung der Gewichtvektoren und durch die Stärkung der Winner Neuronen erhöht, und ein selbst-adaptives Verfahren eingeführt, um das Modell ständig aktualisieren zu können. Darüber hinaus besteht die Hauptaufgabe des NNB-Modells in der weiteren Untersuchung der erkannten unbekannten Verbindungen von der EGHSOM und der Überprüfung, ob sie normal sind. Jedoch, ändern sich die Netzverkehrsdaten wegen des Concept drif Phänomens ständig, was in Echtzeit zur Erzeugung nicht stationärer Netzdaten führt. Dieses Phänomen wird von dem Update-Modell besser kontrolliert. Das EGHSOM-Modell kann die neuen Anomalien effektiv erkennen und das NNB-Model passt die Änderungen in Netzdaten optimal an. Bei den experimentellen Untersuchungen hat das Framework erfolgversprechende Ergebnisse gezeigt. Im ersten Experiment wurde das Framework in Offline-Betriebsmodus evaluiert. Der OptiFilter wurde mit offline-, synthetischen- und realistischen Daten ausgewertet. Der adaptive Klassifikator wurde mit dem 10-Fold Cross Validation Verfahren evaluiert, um dessen Genauigkeit abzuschätzen. Im zweiten Experiment wurde das Framework auf einer 1 bis 10 GB Netzwerkstrecke installiert und im Online-Betriebsmodus in Echtzeit ausgewertet. Der OptiFilter hat erfolgreich die gewaltige Menge von Netzdaten in die strukturierten Verbindungsvektoren umgewandelt und der adaptive Klassifikator hat sie präzise klassifiziert. Die Vergleichsstudie zwischen dem entwickelten Framework und anderen bekannten IDS-Ansätzen zeigt, dass der vorgeschlagene IDSFramework alle anderen Ansätze übertrifft. Dies lässt sich auf folgende Kernpunkte zurückführen: Bearbeitung der gesammelten Netzdaten, Erreichung der besten Performanz (wie die Gesamtgenauigkeit), Detektieren unbekannter Verbindungen und Entwicklung des in Echtzeit arbeitenden Erkennungsmodells von Eindringversuchen.

Opportunities and constraints of peri-urban buffalo and dairy cattle systems in Faisalabad, Punjab, Pakistan

In the big cities of Pakistan, peri-urban dairy production plays an important role for household income generation and the supply of milk and meat to the urban population. On the other hand, milk production in general, and peri-urban dairy production in particular, faces numerous problems that have been well known for decades. Peri-urban dairy producers have been especially neglected by politicians as well as non-government-organizations (NGOs). Against this background, a study in Pakistan’s third largest city, Faisalabad (Punjab Province), was carried out with the aims of gathering basic information, determining major constraints and identifying options for improvements of the peri-urban milk production systems. For data collection, 145 peri-urban households (HH) engaged in dairy production were interviewed face to face using a structured and pretested questionnaire with an interpreter. For analyses, HH were classified into three wealth groups according to their own perception. Thus, 38 HH were poor, 95 HH well off and 12 HH rich (26.2%, 65.5% and 8.3%, respectively). The richer the respondents perceived their HH, the more frequently they were actually in possession of high value HH assets like phones, bank accounts, motorbikes, tractors and cars. Although there was no difference between the wealth groups with respect to the number of HH members (about 10, range: 1 to 23), the educational level of the HH heads differed significantly: on average, heads of poor HH had followed education for 3 years, compared to 6 years for well off HH and 8 years for rich HH. About 40% of the poor and well off HH also had off-farm incomes, while the percentage was much higher - two thirds (67%) - for the rich HH. The majority of the HH were landless (62%); the rest (55 HH) possessed agricultural land from 0.1 to 10.1 ha (average 2.8 ha), where they were growing green fodder: maize, sorghum and pearl millet in summer; berseem, sugar cane and wheat were grown in winter. Dairy animals accounted for about 60% of the herds; the number of dairy animals per HH ranged from 2 to 50 buffaloes (Nili-Ravi breed) and from 0 to 20 cows (mostly crossbred, also Sahiwal). About 37% (n=54) of the HH did not keep cattle. About three quarters of the dairy animals were lactating. The majority of the people taking care of the animals were family workers; 17.3% were hired labourers (exclusively male), employed by 11 rich and 32 well off HH; none of the poor HH employed workers, but the percentages were 33.7% for the well off and 91.7% for the rich HH. The total number of workers increased significantly with increasing wealth (poor: 2.0; well off:2.5; rich: 3.4). Overall, 69 female labourers were recorded, making up 16.8% of employed workers and one fourth of the HH’s own labourers. Apparently, their only duty was to clean the animals´ living areas; only one of them was also watering and showering the animals. Poor HH relied more on female workers than the other two groups: 27.1% of the workers of poor HH were women, but only 14.8% and 6.8% of the labour force of well off and rich HH were female. Two thirds (70%) of the HH sold milk to dhodis (middlemen) and one third (35%) to neighbours; three HH (2%) did doorstep delivery and one HH (1%) had its own shop. The 91 HH keeping both species usually sold mixed milk (97%). Clients for mixed and pure buffalo milk were dhodis (78%, respectively 59%) and neighbours (28%, respectively 47%). The highest milk prices per liter (Pakistani Rupees, 100 PKR @ 0.8 Euro) were paid by alternative clients (44 PKR; 4 HH), followed by neighbours (40 PKR, 50 HH); dhodis paid lower prices (36 PKR, 99 HH). Prices for pure buffalo and mixed milk did not differ significantly. However, HH obtaining the maximum price from the respective clients for the respective type of milk got between 20% (mixed milk, alternative clients) and 68% (mixed milk, dhodi) more than HH fetching the minimum price. Some HH (19%) reported 7% higher prices for the current summer than the preceding winter. Amount of milk sold and distance from the HH to the city center did not influence milk prices. Respondents usually named problems that directly affected their income and that were directly and constantly visible to them, such as high costs, little space and fodder shortages. Other constraints that are only influencing their income indirectly, e.g. the relatively low genetic potential of their animals due to neglected breeding as well as the short- and long-term health problems correlated with imbalanced feeding and insufficient health care, were rarely named. The same accounts for problems accompanying improper dung management (storage, disposal, burning instead of recycling) for the environment and human health. Most of the named problems are linked to each other and should be addressed within the context of the entire system. Therefore, further research should focus on systematic investigations and improvement options, taking a holistic and interdisciplinary approach instead of only working in single fields. Concerted efforts of dairy farmers, researchers, NGOs and political decision makers are necessary to create an economic, ecological and social framework that allows dairy production to serve the entire society. For this, different improvement options should be tested in terms of their impact on environment and income of the farmers, as well as feasibility and sustainability in the peri-urban zones of Faisalabad.

Parameter Estimation in Chaotic Systems

This report examines how to estimate the parameters of a chaotic system given noisy observations of the state behavior of the system. Investigating parameter estimation for chaotic systems is interesting because of possible applications for high-precision measurement and for use in other signal processing, communication, and control applications involving chaotic systems. In this report, we examine theoretical issues regarding parameter estimation in chaotic systems and develop an efficient algorithm to perform parameter estimation. We discover two properties that are helpful for performing parameter estimation on non-structurally stable systems. First, it turns out that most data in a time series of state observations contribute very little information about the underlying parameters of a system, while a few sections of data may be extraordinarily sensitive to parameter changes. Second, for one-parameter families of systems, we demonstrate that there is often a preferred direction in parameter space governing how easily trajectories of one system can "shadow'" trajectories of nearby systems. This asymmetry of shadowing behavior in parameter space is proved for certain families of maps of the interval. Numerical evidence indicates that similar results may be true for a wide variety of other systems. Using the two properties cited above, we devise an algorithm for performing parameter estimation. Standard parameter estimation techniques such as the extended Kalman filter perform poorly on chaotic systems because of divergence problems. The proposed algorithm achieves accuracies several orders of magnitude better than the Kalman filter and has good convergence properties for large data sets.

Implementing Distributed Systems Using Linear Naming

Linear graph reduction is a simple computational model in which the cost of naming things is explicitly represented. The key idea is the notion of "linearity". A name is linear if it is only used once, so with linear naming you cannot create more than one outstanding reference to an entity. As a result, linear naming is cheap to support and easy to reason about. Programs can be translated into the linear graph reduction model such that linear names in the program are implemented directly as linear names in the model. Nonlinear names are supported by constructing them out of linear names. The translation thus exposes those places where the program uses names in expensive, nonlinear ways. Two applications demonstrate the utility of using linear graph reduction: First, in the area of distributed computing, linear naming makes it easy to support cheap cross-network references and highly portable data structures, Linear naming also facilitates demand driven migration of tasks and data around the network without requiring explicit guidance from the programmer. Second, linear graph reduction reveals a new characterization of the phenomenon of state. Systems in which state appears are those which depend on certain -global- system properties. State is not a localizable phenomenon, which suggests that our usual object oriented metaphor for state is flawed.

Microscale thermal engineering of electronic systems

The electronics industry is encountering thermal challenges and opportunities with lengthscales comparable to or much less than one micrometer. Examples include nanoscale phonon hotspots in transistors and the increasing temperature rise in onchip interconnects. Millimeter-scale hotspots on microprocessors, resulting from varying rates of power consumption, are being addressed using two-phase microchannel heat sinks. Nanoscale thermal data storage technology has received much attention recently. This paper provides an overview of these topics with a focus on related research at Stanford University.

Keyword Join: Realizing Keyword Search in P2P-based Database Systems

In this paper, we present a P2P-based database sharing system that provides information sharing capabilities through keyword-based search techniques. Our system requires neither a global schema nor schema mappings between different databases, and our keyword-based search algorithms are robust in the presence of frequent changes in the content and membership of peers. To facilitate data integration, we introduce keyword join operator to combine partial answers containing different keywords into complete answers. We also present an efficient algorithm that optimize the keyword join operations for partial answer integration. Our experimental study on both real and synthetic datasets demonstrates the effectiveness of our algorithms, and the efficiency of the proposed query processing strategies.

Moderating effects of management control systems and innovation on performance. Simple methods for correcting the effects of measurement error for interaction effects in small samples

In the accounting literature, interaction or moderating effects are usually assessed by means of OLS regression and summated rating scales are constructed to reduce measurement error bias. Structural equation models and two-stage least squares regression could be used to completely eliminate this bias, but large samples are needed. Partial Least Squares are appropriate for small samples but do not correct measurement error bias. In this article, disattenuated regression is discussed as a small sample alternative and is illustrated on data of Bisbe and Otley (in press) that examine the interaction effect of innovation and style of use of budgets on performance. Sizeable differences emerge between OLS and disattenuated regression

Optimizing the Dynamic Distribution of Data-stream for High Speed Communications

The performances of high-speed network communications frequently rest with the distribution of data-stream. In this paper, a dynamic data-stream balancing architecture based on link information is introduced and discussed firstly. Then the algorithms for simultaneously acquiring the passing nodes and links of a path between any two source-destination nodes rapidly, as well as a dynamic data-stream distribution planning are proposed. Some related topics such as data fragment disposal, fair service, etc. are further studied and discussed. Besides, the performance and efficiency of proposed algorithms, especially for fair service and convergence, are evaluated through a demonstration with regard to the rate of bandwidth utilization. Hoping the discussion presented here can be helpful to application developers in selecting an effective strategy for planning the distribution of data-stream.

A compositional approach to stable isotope data analysis

Isotopic data are currently becoming an important source of information regarding sources, evolution and mixing processes of water in hydrogeologic systems. However, it is not clear how to treat with statistics the geochemical data and the isotopic data together. We propose to introduce the isotopic information as new parts, and apply compositional data analysis with the resulting increased composition. Results are equivalent to downscale the classical isotopic delta variables, because they are already relative (as needed in the compositional framework) and isotopic variations are almost always very small. This methodology is illustrated and tested with the study of the Llobregat River Basin (Barcelona, NE Spain), where it is shown that, though very small, isotopic variations comp lement geochemical principal components, and help in the better identification of pollution sources

LiDAR data filtering with GRASS GIS for the determination of digital terrain models

In the last years, the use of every type of Digital Elevation Models has iimproved. The LiDAR (Light Detection and Ranging) technology, based on the scansion of the territory b airborne laser telemeters, allows the construction of digital Surface Models (DSM), in an easy way by a simple data interpolation

CAC controlled services sharing a link with connectionless data services in an ATM network: a performance evaluation study

In the B-ISDN there is a provision for four classes of services, all of them supported by a single transport network (the ATM network). Three of these services, the connected oriented (CO) ones, permit connection access control (CAC) but the fourth, the connectionless oriented (CLO) one, does not. Therefore, when CLO service and CO services have to share the same ATM link, a conflict may arise. This is because a bandwidth allocation to obtain maximum statistical gain can damage the contracted ATM quality of service (QOS); and vice versa, in order to guarantee the contracted QOS, the statistical gain have to be sacrificed. The paper presents a performance evaluation study of the influence of the CLO service on a CO service (a circuit emulation service or a variable bit-rate service) when sharing the same link

Large-Area Photo-Mosaics Using Global Alignment and Navigation Data

Seafloor imagery is a rich source of data for the study of biological and geological processes. Among several applications, still images of the ocean floor can be used to build image composites referred to as photo-mosaics. Photo-mosaics provide a wide-area visual representation of the benthos, and enable applications as diverse as geological surveys, mapping and detection of temporal changes in the morphology of biodiversity. We present an approach for creating globally aligned photo-mosaics using 3D position estimates provided by navigation sensors available in deep water surveys. Without image registration, such navigation data does not provide enough accuracy to produce useful composite images. Results from a challenging data set of the Lucky Strike vent field at the Mid Atlantic Ridge are reported

Local WMR navigation with monocular data

This article presents recent WMR (wheeled mobile robot) navigation experiences using local perception knowledge provided by monocular and odometer systems. A local narrow perception horizon is used to plan safety trajectories towards the objective. Therefore, monocular data are proposed as a way to obtain real time local information by building two dimensional occupancy grids through a time integration of the frames. The path planning is accomplished by using attraction potential fields, while the trajectory tracking is performed by using model predictive control techniques. The results are faced to indoor situations by using the lab available platform consisting in a differential driven mobile robot

A survey on voltage sag events in power systems

This paper aims to survey the techniques and methods described in literature to analyse and characterise voltage sags and the corresponding objectives of these works. The study has been performed from a data mining point of view

Embedding Emotional Context in Recommender Systems

Emotions are crucial for user's decision making in recommendation processes. We first introduce ambient recommender systems, which arise from the analysis of new trends on the exploitation of the emotional context in the next generation of recommender systems. We then explain some results of these new trends in real-world applications through the smart prediction assistant (SPA) platform in an intelligent learning guide with more than three million users. While most approaches to recommending have focused on algorithm performance. SPA makes recommendations to users on the basis of emotional information acquired in an incremental way. This article provides a cross-disciplinary perspective to achieve this goal in such recommender systems through a SPA platform. The methodology applied in SPA is the result of a bunch of technology transfer projects for large real-world rccommender systems