Monitoring and analysis of internet traffic targeting unused address spaces

Today’s evolving networks are experiencing a large number of different attacks ranging from system break-ins, infection from automatic attack tools such as worms, viruses, trojan horses and denial of service (DoS). One important aspect of such attacks is that they are often indiscriminate and target Internet addresses without regard to whether they are bona fide allocated or not. Due to the absence of any advertised host services the traffic observed on unused IP addresses is by definition unsolicited and likely to be either opportunistic or malicious. The analysis of large repositories of such traffic can be used to extract useful information about both ongoing and new attack patterns and unearth unusual attack behaviors. However, such an analysis is difficult due to the size and nature of the collected traffic on unused address spaces. In this dissertation, we present a network traffic analysis technique which uses traffic collected from unused address spaces and relies on the statistical properties of the collected traffic, in order to accurately and quickly detect new and ongoing network anomalies. Detection of network anomalies is based on the concept that an anomalous activity usually transforms the network parameters in such a way that their statistical properties no longer remain constant, resulting in abrupt changes. In this dissertation, we use sequential analysis techniques to identify changes in the behavior of network traffic targeting unused address spaces to unveil both ongoing and new attack patterns. Specifically, we have developed a dynamic sliding window based non-parametric cumulative sum change detection techniques for identification of changes in network traffic. Furthermore we have introduced dynamic thresholds to detect changes in network traffic behavior and also detect when a particular change has ended. Experimental results are presented that demonstrate the operational effectiveness and efficiency of the proposed approach, using both synthetically generated datasets and real network traces collected from a dedicated block of unused IP addresses.

Use of IP addresses for high rate flooding attack detection

High-rate flooding attacks (aka Distributed Denial of Service or DDoS attacks) continue to constitute a pernicious threat within the Internet domain. In this work we demonstrate how using packet source IP addresses coupled with a change-point analysis of the rate of arrival of new IP addresses may be sufficient to detect the onset of a high-rate flooding attack. Importantly, minimizing the number of features to be examined, directly addresses the issue of scalability of the detection process to higher network speeds. Using a proof of concept implementation we have shown how pre-onset IP addresses can be efficiently represented using a bit vector and used to modify a “white list” filter in a firewall as part of the mitigation strategy.

Proactive detection of high collision concentration locations on highways

In previous research (Chung et al., 2009), the potential of the continuous risk profile (CRP) to proactively detect the systematic deterioration of freeway safety levels was presented. In this paper, this potential is investigated further, and an algorithm is proposed for proactively detecting sites where the collision rate is not sufficiently high to be classified as a high collision concentration location but where a systematic deterioration of safety level is observed. The approach proposed compares the weighted CRP across different years and uses the cumulative sum (CUSUM) algorithm to detect the sites where changes in collision rate are observed. The CRPs of the detected sites are then compared for reproducibility. When high reproducibility is observed, a growth factor is used for sequential hypothesis testing to determine if the collision profiles are increasing over time. Findings from applying the proposed method using empirical data are documented in the paper together with a detailed description of the method.

HMM triangle relative entropy concepts in sequential change detection applied to vision-based dim target manoeuvre detection

The quick detection of abrupt (unknown) parameter changes in an observed hidden Markov model (HMM) is important in several applications. Motivated by the recent application of relative entropy concepts in the robust sequential change detection problem (and the related model selection problem), this paper proposes a sequential unknown change detection algorithm based on a relative entropy based HMM parameter estimator. Our proposed approach is able to overcome the lack of knowledge of post-change parameters, and is illustrated to have similar performance to the popular cumulative sum (CUSUM) algorithm (which requires knowledge of the post-change parameter values) when examined, on both simulated and real data, in a vision-based aircraft manoeuvre detection problem.

Risk modelling in quality clinical registries : Monitoring lesion treatment failure rate in percutaneous coronary interventions

Aims: This paper describes the development of a risk adjustment (RA) model predictive of individual lesion treatment failure in percutaneous coronary interventions (PCI) for use in a quality monitoring and improvement program. Methods and results: Prospectively collected data for 3972 consecutive revascularisation procedures (5601 lesions) performed between January 2003 and September 2011 were studied. Data on procedures to September 2009 (n = 3100) were used to identify factors predictive of lesion treatment failure. Factors identified included lesion risk class (p < 0.001), occlusion type (p < 0.001), patient age (p = 0.001), vessel system (p < 0.04), vessel diameter (p < 0.001), unstable angina (p = 0.003) and presence of major cardiac risk factors (p = 0.01). A Bayesian RA model was built using these factors with predictive performance of the model tested on the remaining procedures (area under the receiver operating curve: 0.765, Hosmer–Lemeshow p value: 0.11). Cumulative sum, exponentially weighted moving average and funnel plots were constructed using the RA model and subjectively evaluated. Conclusion: A RA model was developed and applied to SPC monitoring for lesion failure in a PCI database. If linked to appropriate quality improvement governance response protocols, SPC using this RA tool might improve quality control and risk management by identifying variation in performance based on a comparison of observed and expected outcomes.

Asymptotic minimax robust and misspecified Lorden quickest change detection for dependent stochastic processes

The quick detection of an abrupt unknown change in the conditional distribution of a dependent stochastic process has numerous applications. In this paper, we pose a minimax robust quickest change detection problem for cases where there is uncertainty about the post-change conditional distribution. Our minimax robust formulation is based on the popular Lorden criteria of optimal quickest change detection. Under a condition on the set of possible post-change distributions, we show that the widely known cumulative sum (CUSUM) rule is asymptotically minimax robust under our Lorden minimax robust formulation as a false alarm constraint becomes more strict. We also establish general asymptotic bounds on the detection delay of misspecified CUSUM rules (i.e. CUSUM rules that are designed with post- change distributions that differ from those of the observed sequence). We exploit these bounds to compare the delay performance of asymptotically minimax robust, asymptotically optimal, and other misspecified CUSUM rules. In simulation examples, we illustrate that asymptotically minimax robust CUSUM rules can provide better detection delay performance at greatly reduced computation effort compared to competing generalised likelihood ratio procedures.

Estimation of the time of a linear trend in monitoring survival time

Change point estimation is recognized as an essential tool of root cause analyses within quality control programs as it enables clinical experts to search for potential causes of change in hospital outcomes more effectively. In this paper, we consider estimation of the time when a linear trend disturbance has occurred in survival time following an in-control clinical intervention in the presence of variable patient mix. To model the process and change point, a linear trend in the survival time of patients who underwent cardiac surgery is formulated using hierarchical models in a Bayesian framework. The data are right censored since the monitoring is conducted over a limited follow-up period. We capture the effect of risk factors prior to the surgery using a Weibull accelerated failure time regression model. We use Markov Chain Monte Carlo to obtain posterior distributions of the change point parameters including the location and the slope size of the trend and also corresponding probabilistic intervals and inferences. The performance of the Bayesian estimator is investigated through simulations and the result shows that precise estimates can be obtained when they are used in conjunction with the risk-adjusted survival time cumulative sum control chart (CUSUM) control charts for different trend scenarios. In comparison with the alternatives, step change point model and built-in CUSUM estimator, more accurate and precise estimates are obtained by the proposed Bayesian estimator over linear trends. These superiorities are enhanced when probability quantification, flexibility and generalizability of the Bayesian change point detection model are also considered.

sRNAscanner: A Computational Tool for Intergenic Small RNA Detection in Bacterial Genomes

Background:Bacterial non-coding small RNAs (sRNAs) have attracted considerable attention due to their ubiquitous nature and contribution to numerous cellular processes including survival, adaptation and pathogenesis. Existing computational approaches for identifying bacterial sRNAs demonstrate varying levels of success and there remains considerable room for improvement. Methodology/Principal Findings: Here we have proposed a transcriptional signal-based computational method to identify intergenic sRNA transcriptional units (TUs) in completely sequenced bacterial genomes. Our sRNAscanner tool uses position weight matrices derived from experimentally defined E. coli K-12 MG1655 sRNA promoter and rho-independent terminator signals to identify intergenic sRNA TUs through sliding window based genome scans. Analysis of genomes representative of twelve species suggested that sRNAscanner demonstrated equivalent sensitivity to sRNAPredict2, the best performing bioinformatics tool available presently. However, each algorithm yielded substantial numbers of known and uncharacterized hits that were unique to one or the other tool only. sRNAscanner identified 118 novel putative intergenic sRNA genes in Salmonella enterica Typhimurium LT2, none of which were flagged by sRNAPredict2. Candidate sRNA locations were compared with available deep sequencing libraries derived from Hfq-co-immunoprecipitated RNA purified from a second Typhimurium strain (Sittka et al. (2008) PLoS Genetics 4: e1000163). Sixteen potential novel sRNAs computationally predicted and detected in deep sequencing libraries were selected for experimental validation by Northern analysis using total RNA isolated from bacteria grown under eleven different growth conditions. RNA bands of expected sizes were detected in Northern blots for six of the examined candidates. Furthermore, the 5'-ends of these six Northern-supported sRNA candidates were successfully mapped using 5'-RACE analysis. Conclusions/Significance: We have developed, computationally examined and experimentally validated the sRNAscanner algorithm. Data derived from this study has successfully identified six novel S. Typhimurium sRNA genes. In addition, the computational specificity analysis we have undertaken suggests that similar to 40% of sRNAscanner hits with high cumulative sum of scores represent genuine, undiscovered sRNA genes. Collectively, these data strongly support the utility of sRNAscanner and offer a glimpse of its potential to reveal large numbers of sRNA genes that have to date defied identification. sRNAscanner is available from: http://bicmku.in:8081/sRNAscanner or http://cluster.physics.iisc.ernet.in/sRNAscanner/.

Lumettoman maan routaolojen mallintaminen ja ennustettavuus muuttuvassa ilmastossa

Raportissa on arvioitu ilmastonmuutoksen vaikutusta Suomen maaperän talviaikaiseen jäätymiseen lämpösummien perusteella. Laskelmat kuvaavat roudan paksuutta nimenomaisesti lumettomilla alueilla, esimerkiksi teillä, joilta satanut lumi aurataan pois. Luonnossa lämpöä eristävän lumipeitteen alla routaa on ohuemmin kuin tällaisilla lumettomilla alueilla. Toisaalta luonnollisessa ympäristössä paikalliset erot korostuvat johtuen mm. maalajeista ja kasvillisuudesta. Roudan paksuudet laskettiin ensin perusjakson 1971–2000 ilmasto-oloissa talviaikaisten säähavaintotietoihin pohjautuvien lämpötilojen perusteella. Sen jälkeen laskelmat toistettiin kolmelle tulevalle ajanjaksolle (2010–2039, 2040–2069 ja 2070–2099) kohottamalla lämpötiloja ilmastonmuutosmallien ennustamalla tavalla. Laskelman pohjana käytettiin 19 ilmastomallin A1B-skenaarioajojen keskimäärin simuloimaa lämpötilan muutosta. Tulosten herkkyyden arvioimiseksi joitakin laskelmia tehtiin myös tätä selvästi heikompaa ja voimakkaampaa lämpenemisarviota käyttäen. A1B-skenaarion mukaisen lämpötilan nousun toteutuessa nykyisiä mallituloksia vastaavasti routakerros ohenee sadan vuoden aikana Pohjois-Suomessa 30–40 %, suuressa osassa maan keski- ja eteläosissa 50–70 %. Jo lähivuosikymmeninä roudan ennustetaan ohentuvan 10–30 %, saaristossa enemmän. Mikäli lämpeneminen toteutuisi voimakkaimman tarkastellun vaihtoehdon mukaisesti, roudan syvyys pienenisi tätäkin enemmän. Roudan paksuuden vuosienvälistä vaihtelua ja sen muuttumista tulevaisuudessa pyrittiin myös arvioimaan. Leutoina talvina routa ohenee enemmän kuin normaaleina tai ankarina pakkastalvina. Päivittäistä sään vaihtelua simuloineen säägeneraattorin tuottamassa aineistoissa esiintyi kuitenkin liian vähän hyvin alhaisia ja hyvin korkeita lämpötiloja. Siksi näitten lämpötilatietojen pohjalta laskettu roudan paksuuskin ilmeisesti vaihtelee liian vähän vuodesta toiseen. Kelirikkotilanteita voi esiintyä myös kesken routakauden, jos useamman päivän suojasää ja samanaikainen runsas vesisade pääsevät sulattamaan maata. Tällaiset routakauden aikana sattuvat säätilat näyttävätkin yleistyvän lähivuosikymmeninä. Vuosisadan loppua kohti ne sen sijaan maan eteläosissa jälleen vähenevät, koska routakausi lyhenee oleellisesti. Tulevia vuosikymmeniä koskevien ilmastonmuutosennusteiden ohella routaa ja kelirikon esiintymistä on periaatteessa mahdollista ennustaa myös lähiaikojen sääennusteita hyödyntäen. Pitkät, viikkojen tai kuukausien mittaiset sääennusteet eivät tosin ole ainakaan vielä erityisen luotettavia, mutta myös lyhyemmistä ennusteista voisi olla hyötyä mm. tienpitoa suunniteltaessa.

A novel algorithm for cooperative distributed sequential spectrum sensing in Cognitive Radio

This paper considers cooperative spectrum sensing in Cognitive Radios. In our previous work we have developed DualSPRT, a distributed algorithm for cooperative spectrum sensing using Sequential Probability Ratio Test (SPRT) at the Cognitive Radios as well as at the fusion center. This algorithm works well, but is not optimal. In this paper we propose an improved algorithm- SPRT-CSPRT, which is motivated from Cumulative Sum Procedures (CUSUM). We analyse it theoretically. We also modify this algorithm to handle uncertainties in SNR's and fading.

Application of signal detection methods to fisheries management

The abundance of many commercially important fish stocks are declining and this has led to widespread concern on the performance of traditional approach in fisheries management. Quantitative models are used for obtaining estimates of population abundance and the management advice is based on annual harvest levels (TAC), where only a certain amount of catch is allowed from specific fish stocks. However, these models are data intensive and less useful when stocks have limited historical information. This study examined whether empirical stock indicators can be used to manage fisheries. The relationship between indicators and the underlying stock abundance is not direct and hence can be affected by disturbances that may account for both transient and persistent effects. Methods from Statistical Process Control (SPC) theory such as the Cumulative Sum (CUSUM) control charts are useful in classifying these effects and hence they can be used to trigger management response only when a significant impact occurs to the stock biomass. This thesis explores how empirical indicators along with CUSUM can be used for monitoring, assessment and management of fish stocks. I begin my thesis by exploring various age based catch indicators, to identify those which are potentially useful in tracking the state of fish stocks. The sensitivity and response of these indicators towards changes in Spawning Stock Biomass (SSB) showed that indicators based on age groups that are fully selected to the fishing gear or Large Fish Indicators (LFIs) are most useful and robust across the range of scenarios considered. The Decision-Interval (DI-CUSUM) and Self-Starting (SS-CUSUM) forms are the two types of control charts used in this study. In contrast to the DI-CUSUM, the SS-CUSUM can be initiated without specifying a target reference point (‘control mean’) to detect out-of-control (significant impact) situations. The sensitivity and specificity of SS-CUSUM showed that the performances are robust when LFIs are used. Once an out-of-control situation is detected, the next step is to determine how much shift has occurred in the underlying stock biomass. If an estimate of this shift is available, they can be used to update TAC by incorporation into Harvest Control Rules (HCRs). Various methods from Engineering Process Control (EPC) theory were tested to determine which method can measure the shift size in stock biomass with the highest accuracy. Results showed that methods based on Grubb’s harmonic rule gave reliable shift size estimates. The accuracy of these estimates can be improved by monitoring a combined indicator metric of stock-recruitment and LFI because this may account for impacts independent of fishing. The procedure of integrating both SPC and EPC is known as Statistical Process Adjustment (SPA). A HCR based on SPA was designed for DI-CUSUM and the scheme was successful in bringing out-of-control fish stocks back to its in-control state. The HCR was also tested using SS-CUSUM in the context of data poor fish stocks. Results showed that the scheme will be useful for sustaining the initial in-control state of the fish stock until more observations become available for quantitative assessments.

North Atlantic climatic signals and the plankton of the European Continental Shelf

Climatic variability on the European Continental Shelf is dominated by events over the North Atlantic Ocean, and in particular by the North Atlantic Oscillation (NAO). The NAO is essentially a winter phenomenon, and its effects will be felt most strongly by populations for which winter conditions are critical. One example is the copepod Calanus finmarchicus, whose northern North Sea populations overwinter at depth in the North Atlantic. Its annual abundance in this region is strongly dependent on water transports at the end of the winter, and hence on the NAO index. Variations in the NAO give rise to changes in the circulation of the North Atlantic Ocean, with additional perturbations arising from El Ni (n) over tildeo - Southern Oscillation (ENSO) events in the Pacific, and these changes can be delayed by several years because of the adjustment time of the ocean circulation. One measure of the circulation is the latitude of the north wall of the Gulf Stream (GSNW index). Interannual variations in the plankton of the Shelf Seas show strong correlations with the fluctuations of the GSNW index, which are the result of Atlantic-wide atmospheric processes. These associations imply that the interannual variations are climatically induced rather than due to natural fluctuations of the marine ecosystem, and that the zooplankton populations have not been significantly affected by anthropogenic processes such as nutrient enrichment or fishing pressure. While the GSNW index represents a response to atmospheric changes over two or more years, the zooplankton populations correlated with it have generation times of a few weeks. The simplest explanation for the associations between the zooplankton and the GSNW index is that the plankton are responding to weather patterns propagating downstream from the Gulf Stream system. It seems that these meteorological processes operate in the spring. Although it has been suggested that there was a regime shift in the North Sea in the late 1980s, examination of the time-series by the cumulative sum (CUSUM) technique shows that any changes in the zooplankton of the central and northern North Sea are consistent with the background climatic variability. The abundance of total copepods increased during this period but this change does not represent a dramatic change in ecosystem processes. It is possible some change may have occurred at the end of the time-series in the years 1997 and 1998.

Influence of the Pacific Decadal Oscillation on phytoplankton phenology and community structure in the western North Pacific

Phytoplankton phenology and community structure in the western North Pacific were investigated for 2001–2009, based on satellite ocean colour data and the Continuous Plankton Recorder survey. We estimated the timing of the spring bloom based on the cumulative sum satellite chlorophyll adata, and found that the Pacific Decadal Oscillation (PDO)-related interannual SST anomaly in spring significantly affected phytoplankton phenology. The bloom occurred either later or earlier in years of positive or negative PDO (indicating cold and warm conditions, respectively). Phytoplankton composition in the early summer varied depending on the magnitude of seasonal SST increases, rather than the SST value itself. Interannual variations in diatom abundance and the relative abundance of non-diatoms were positively correlated with SST increases for March–April and May–July, respectively, suggesting that mixed layer environmental factors, such as light availability and nutrient stoichiometry, determine shifts in phytoplankton community structure. Our study emphasised the importance of the interannual variation in climate-induced warm–cool cycles as one of the key mechanisms linking climatic forcing and lower trophic level ecosystems.

Comparison of new and primary production models using SeaWiFS data in contrasting hydrographic zones of the northern North Atlantic.

The accuracy of two satellite models of marine primary (PP) and new production (NP) were assessed against 14C and 15N uptake measurements taken during six research cruises in the northern North Atlantic. The wavelength resolving model (WRM) was more accurate than the Vertical General Production Model (VGPM) for computation of both PP and NP. Mean monthly satellite maps of PP and NP for both models were generated from 1997 to 2010 using SeaWiFS data for the Irminger basin and North Atlantic. Intra- and inter-annual variability of the two models was compared in six hydrographic zones. Both models exhibited similar spatio-temporal patterns: PP and NP increased from April to June and decreased by August. Higher values were associated with the East Greenland Current (EGC), Iceland Basin (ICB) and the Reykjanes Ridge (RKR) and lower values occurred in the Central Irminger Current (CIC), North Irminger Current (NIC) and Southern Irminger Current (SIC). The annual PP and NP over the SeaWiFS record was 258 and 82 gC m-2 yr-1 respectively for the VGPM and 190 and 41 gC m-2 yr-1 for the WRM. Average annual cumulative sum in the anomalies of NP for the VGPM were positively correlated with the North Atlantic Oscillation (NAO) in the EGC, CIC and SIC and negatively correlated with the multivariate ENSO index (MEI) in the ICB. By contrast, cumulative sum of the anomalies of NP for the WRM were significantly correlated with NAO only in the EGC and CIC. NP from both VGPM and WRM exhibited significant negative correlations with Arctic Oscillation (AO) in all hydrographic zones. The differences in estimates of PP and NP in these hydrographic zones arise principally from the parameterisation of the euphotic depth and the SST dependence of photo-physiological term in the VGPM, which has a greater sensitivity to variations in temperature than the WRM. In waters of 0 to 5C PP using the VGPM was 43% higher than WRM, from 5 to 10C the VGPM was 29% higher and from 10 to 15C the VGPM was 27% higher.