Application of rank correlation, clustering and classification in information security

This article is devoted to experimental investigation of a novel application of a clustering technique introduced by the authors recently in order to use robust and stable consensus functions in information security, where it is often necessary to process large data sets and monitor outcomes in real time, as it is required, for example, for intrusion detection. Here we concentrate on a particular case of application to profiling of phishing websites. First, we apply several independent clustering algorithms to a randomized sample of data to obtain independent initial clusterings. Silhouette index is used to determine the number of clusters. Second, rank correlation is used to select a subset of features for dimensionality reduction. We investigate the effectiveness of the Pearson Linear Correlation Coefficient, the Spearman Rank Correlation Coefficient and the Goodman--Kruskal Correlation Coefficient in this application. Third, we use a consensus function to combine independent initial clusterings into one consensus clustering. Fourth, we train fast supervised classification algorithms on the resulting consensus clustering in order to enable them to process the whole large data set as well as new data. The precision and recall of classifiers at the final stage of this scheme are critical for the effectiveness of the whole procedure. We investigated various combinations of several correlation coefficients, consensus functions, and a variety of supervised classification algorithms.

Discriminating DDoS attacks from flash crowds using flow correlation coefficient

Distributed Denial of Service (DDoS) attack is a critical threat to the Internet, and botnets are usually the engines behind them. Sophisticated botmasters attempt to disable detectors by mimicking the traffic patterns of flash crowds. This poses a critical challenge to those who defend against DDoS attacks. In our deep study of the size and organization of current botnets, we found that the current attack flows are usually more similar to each other compared to the flows of flash crowds. Based on this, we proposed a discrimination algorithm using the flow correlation coefficient as a similarity metric among suspicious flows. We formulated the problem, and presented theoretical proofs for the feasibility of the proposed discrimination method in theory. Our extensive experiments confirmed the theoretical analysis and demonstrated the effectiveness of the proposed method in practice.

Document clustering in correlation similarity measure space

This paper presents a new spectral clustering method called correlation preserving indexing (CPI), which is performed in the correlation similarity measure space. In this framework, the documents are projected into a low-dimensional semantic space in which the correlations between the documents in the local patches are maximized while the correlations between the documents outside these patches are minimized simultaneously. Since the intrinsic geometrical structure of the document space is often embedded in the similarities between the documents, correlation as a similarity measure is more suitable for detecting the intrinsic geometrical structure of the document space than euclidean distance. Consequently, the proposed CPI method can effectively discover the intrinsic structures embedded in high-dimensional document space. The effectiveness of the new method is demonstrated by extensive experiments conducted on various data sets and by comparison with existing document clustering methods.

Implications of Kungurian (Early Permian) conodonts from Hatahoko, Japan, for correlation between the Tethyan and international timescales

The correlation between the fusulinid-based Tethyan and the conodont-based international timescales of the Permian System has become one of themost disputed issues among the Permian community during the past two decades. In this paper,we document a conodont fauna consisting of four species including Sweetognathus guizhouensis, Pseudohindeodus augustus, Hindeodus permicus and a new genus Meiognathus pustulus from the lower part of a large exotic limestone block at Hatahoko in the Nyukawa area, Gifu Prefecture, central Japan, which all suggest aKungurian age. The Kungurian age indicated by the conodonts is consistent with the age of the associated brachiopods, but conflicts with the Murgabian age indicated by the associated fusulinids including Cancellina nipponica, Neoschwagerina simplex, Neofusulinella praecursor etc. This co-occurrence of Kungurian conodonts and Murgabian fusulinids in central Japan suggests that previously unrecognized temporal distributions of some key fusulinid or conodont elements need to be clarified and that the intensively-disputed correlation problem between theKungurian containing theMurgabian fusulinids at the Luodian section in Guizhou, South China with the strata containing the ammonoid Waagenoceras in Oman and Sicily was caused by artificial conodont taxonomic discrepancies. The Luodian section in South China could serve as a key reference section for the correlation of the Kungurian Stage (late Early Permian) between the Tethyan and international timescales. 

Network traffic classification using correlation information

Traffic classification has wide applications in network management, from security monitoring to quality of service measurements. Recent research tends to apply machine learning techniques to flow statistical feature based classification methods. The nearest neighbor (NN)-based method has exhibited superior classification performance. It also has several important advantages, such as no requirements of training procedure, no risk of overfitting of parameters, and naturally being able to handle a huge number of classes. However, the performance of NN classifier can be severely affected if the size of training data is small. In this paper, we propose a novel nonparametric approach for traffic classification, which can improve the classification performance effectively by incorporating correlated information into the classification process. We analyze the new classification approach and its performance benefit from both theoretical and empirical perspectives. A large number of experiments are carried out on two real-world traffic data sets to validate the proposed approach. The results show the traffic classification performance can be improved significantly even under the extreme difficult circumstance of very few training samples.

Correlation between a self-administered walk test and a standardised Six Minute Walk Test in adults

This study was undertaken to assess the correlation between a self-administered, adapted Six Minute Walk Test (the Home-Heart-Walk) and the standard Six Minute Walk Test based on the American Thoracic Society guideline. A correlational study was conducted at a university campus in Sydney, Australia. Thirteen healthy volunteers underwent the Home-Heart-Walk and the standard Six Minute Walk Test on a single occasion. The distance that participants walked during the two tests was assessed using Pearson's correlation. The correlation between the Home-Heart-Walk and the Six Minute Walk Test distance was 0.81. The Home-Heart-Walk distance was highly correlated to the standard Six Minute Walk Test distance in this study. This relationship provides confidence for further research in populations to facilitate monitoring and evaluation.

Context-dependent correlation between resting metabolic rate and daily energy expenditure in wild chipmunks

Several empirical studies have shown that variation in daily energy expenditure (DEE) and resting metabolic rate (RMR) is influenced by environmental and individual factors, but whether these shared influences are responsible for, or independent of, relationships between DEE and RMR remains unknown. The objectives of this study were to (i) simultaneously evaluate the effects of environmental and individual variables on DEE and RMR in free-ranging eastern chipmunks (Tamias striatus) and (ii) quantify the correlation between DEE and RMR before and after controlling for common sources of variation. We found that the influence of individual factors on DEE and RMR is most often shared, whereas the influence of environmental factors tends to be distinct. Both raw and mass-adjusted DEE and RMR were significantly correlated, but this correlation vanished after accounting for the shared effect of reproduction on both traits. However, within reproductive individuals, DEE and RMR remained positively correlated after accounting for all other significant covariates. The ratio of DEE to RMR was significantly higher during reproduction than at other times of the year and was negatively correlated with ambient temperature. DEE and RMR appear to be inherently correlated during reproduction, but this correlation does not persist during other, less energy-demanding periods of the annual cycle.

Performance, personality, and energetics : correlation, causation, and mechanism

The study of phenotypic evolution should be an integrative endeavor that combines different approaches and crosses disciplinary and phylogenetic boundaries to consider complex traits and organisms that historically have been studied in isolation from each other. Analyses of individual variation within populations can act to bridge studies focused at the levels of morphology, physiology, biochemistry, organismal performance, behavior, and life history. For example, the study of individual variation recently facilitated the integration of behavior into the concept of a pace-of-life syndrome and effectively linked the field of energetics with research on animal personality. Here, we illustrate how studies on the pace-of-life syndrome and the energetics of personality can be integrated within a physiology-performance-behavior-fitness paradigm that includes consideration of ecological context. We first introduce key concepts and definitions and then review the rapidly expanding literature on the links between energy metabolism and personality traits commonly studied in nonhuman animals (activity, exploration, boldness, aggressiveness, sociability). We highlight some empirical literature involving mammals and squamates that demonstrates how emerging fields can develop in rather disparate ways because of historical accidents and/or particularities of different kinds of organisms. We then briefly discuss potentially interesting avenues for future conceptual and empirical research in relation to motivation, intraindividual variation, and mechanisms underlying trait correlations. The integration of performance traits within the pace-of-life-syndrome concept has the potential to fill a logical gap between the context dependency of selection and how energetics and personality are expected to interrelate. Studies of how performance abilities and/or aspects of Darwinian fitness relate to both metabolic rate and personality traits are particularly lacking.