A distributed spatial-temporal weighted model on MapReduce for short-term traffic flow forecasting

Accurate and timely traffic flow prediction is crucial to proactive traffic management and control in data-driven intelligent transportation systems (D2ITS), which has attracted great research interest in the last few years. In this paper, we propose a Spatial-Temporal Weighted K-Nearest Neighbor model, named STW-KNN, in a general MapReduce framework of distributed modeling on a Hadoop platform, to enhance the accuracy and efficiency of short-term traffic flow forecasting. More specifically, STW-KNN considers the spatial-temporal correlation and weight of traffic flow with trend adjustment features, to optimize the search mechanisms containing state vector, proximity measure, prediction function, and K selection. urthermore, STW-KNN is implemented on a widely adopted Hadoop distributed computing platform with the MapReduce parallel processing paradigm, for parallel prediction of traffic flow in real time. inally, with extensive experiments on real-world big taxi trajectory data, STW-KNN is compared with the state-of-the-art prediction models including conventional K-Nearest Neighbor (KNN), Artificial Neural Networks (ANNs), Naïve Bayes (NB), Random orest (R), and C4.. The results demonstrate that the proposed model is superior to existing models on accuracy by decreasing the mean absolute percentage error (MAPE) value more than 11.9% only in time domain and even achieves 89.71% accuracy improvement with the MAPEs of between 4% and 6.% in both space and time domains, and also significantly improves the efficiency and scalability of short-term traffic flow forecasting over existing approaches.

Real-time traffic flow statistical analysis based on network-constrained moving object trajectories

In this paper, we propose a novel traffic flow analysis method, Network-constrained Moving Objects Database based Traffic Flow Statistical Analysis (NMOD-TFSA) model. By sampling and analyzing the spatial-temporal trajectories of network constrained moving objects, NMOD-TFSA can get the real-time traffic conditions of the transportation network. The experimental results show that, compared with the floating-car methods which are widely used in current traffic flow analyzing systems, NMOD-TFSA provides an improved performance in terms of communication costs and statistical accuracy.

A MapReduce-based nearest neighbor approach for big-data-driven traffic flow prediction

In big-data-driven traffic flow prediction systems, the robustness of prediction performance depends on accuracy and timeliness. This paper presents a new MapReduce-based nearest neighbor (NN) approach for traffic flow prediction using correlation analysis (TFPC) on a Hadoop platform. In particular, we develop a real-time prediction system including two key modules, i.e., offline distributed training (ODT) and online parallel prediction (OPP). Moreover, we build a parallel k-nearest neighbor optimization classifier, which incorporates correlation information among traffic flows into the classification process. Finally, we propose a novel prediction calculation method, combining the current data observed in OPP and the classification results obtained from large-scale historical data in ODT, to generate traffic flow prediction in real time. The empirical study on real-world traffic flow big data using the leave-one-out cross validation method shows that TFPC significantly outperforms four state-of-the-art prediction approaches, i.e., autoregressive integrated moving average, Naïve Bayes, multilayer perceptron neural networks, and NN regression, in terms of accuracy, which can be improved 90.07% in the best case, with an average mean absolute percent error of 5.53%. In addition, it displays excellent speedup, scaleup, and sizeup.

An optimal transportation routing approach using GIS-based dynamic traffic flows

This paper examines the value of real-time traffic information gathered through Geographic Information Systems for achieving an optimal vehicle routing within a dynamically stochastic transportation network. We present a systematic approach in determining the dynamically varying parameters and implementation attributes that were used for the development of a Web-based transportation routing application integrated with real-time GIS services. We propose and implement an optimal routing algorithm by modifying Dijkstra’s algorithm in order to incorporate stochastically changing traffic flows. We describe the significant features of our Web application in making use of the real-time dynamic traffic flow information from GIS services towards achieving total costs savings and vehicle usage reduction. These features help users and vehicle drivers in improving their service levels and productivity as the Web application enables them to interactively find the optimal path and in identifying destinations effectively.

Hierarchical summarization techniques for network traffic

In today’s high speed networks it is becoming increasingly challenging for network managers to understand the nature of the traffic that is carried in their network. A major problem for traffic analysis in this context is how to extract a concise yet accurate summary of the relevant aggregate traffic flows that are present in network traces. In this paper, we present two summarization techniques to minimize the size of the traffic flow report that is generated by a hierarchical cluster analysis tool. By analyzing the accuracy and compaction gain of our approach on a standard benchmark dataset, we demonstrate that our approach achieves more accurate summaries than those of an existing tool that is based on frequent itemset mining.

Detecting deviations from known paths and speeds in a surveillance situation

This paper presents techniques for analysing human behaviour via video surveillance. In known scenes under surveillance, common paths of movement between entry and exit points are obtained and classified. These are used, together with a priori velocity data, to serve as a model of normal traffic flow in the scene. Surveillance sequences are then processed to extract and track the movement of people in the scene, which is compared with the models to enable detection of abnormal movement

Spatial coordinated medium sharing: Optimal access control management in drive-thru internet

Driven by the ever-growing expectation of ubiquitous connectivity and the widespread adoption of IEEE 802.11 networks, it is not only highly demanded but also entirely possible for in-motion vehicles to establish convenient Internet access to roadside WiFi access points (APs) than ever before, which is referred to as Drive-Thru Internet. The performance of Drive-Thru Internet, however, would suffer from the high vehicle mobility, severe channel contentions, and instinct issues of the IEEE 802.11 MAC as it was originally designed for static scenarios. As an effort to address these problems, in this paper, we develop a unified analytical framework to evaluate the performance of Drive-Thru Internet, which can accommodate various vehicular traffic flow states, and to be compatible with IEEE 802.11a/b/g networks with a distributed coordination function (DCF). We first develop the mathematical analysis to evaluate the mean saturated throughput of vehicles and the transmitted data volume of a vehicle per drive-thru. We show that the throughput performance of Drive-Thru Internet can be enhanced by selecting an optimal transmission region within an AP's coverage for the coordinated medium sharing of all vehicles. We then develop a spatial access control management approach accordingly, which ensures the airtime fairness for medium sharing and boosts the throughput performance of Drive-Thru Internet in a practical, efficient, and distributed manner. Simulation results show that our optimal access control management approach can efficiently work in IEEE 802.11b and 802.11g networks. The maximal transmitted data volume per drive-thru can be enhanced by 113.1% and 59.5% for IEEE 802.11b and IEEE 802.11g networks with a DCF, respectively, compared with the normal IEEE 802.11 medium access with a DCF.

A flow control methodology for the rapid generation of vehicle traffic models

This paper presents a conveyor-based methodology to model complex vehicle flows common to factory and distribution warehouse facilities. The AGV and human path modelling techniques available in many commercial discrete event simulation packages require extensive knowledge and time to implement even the simplest flow control rules for multiple vehicle interaction. Although discrete event simulation is accepted as an effective tool to model vehicle delivery movements, human paths and delivery schedules for modern assembly lines, the time to generate accurate models is a significant limitation of existing simulation-based optimisation methodologies. The flow control method has been successfully implemented using two commercial simulation packages. It provides a realistic visual representation, as well as accurate statistical results, and reduces the model development process cost.

An effective network traffic classification method with unknown flow detection

Traffic classification technique is an essential tool for network and system security in the complex environments such as cloud computing based environment. The state-of-the-art traffic classification methods aim to take the advantages of flow statistical features and machine learning techniques, however the classification performance is severely affected by limited supervised information and unknown applications. To achieve effective network traffic classification, we propose a new method to tackle the problem of unknown applications in the crucial situation of a small supervised training set. The proposed method possesses the superior capability of detecting unknown flows generated by unknown applications and utilizing the correlation information among real-world network traffic to boost the classification performance. A theoretical analysis is provided to confirm performance benefit of the proposed method. Moreover, the comprehensive performance evaluation conducted on two real-world network traffic datasets shows that the proposed scheme outperforms the existing methods in the critical network environment.

An automatic application signature construction system for unknown traffic

Identifying applications and classifying network traffic flows according to their source applications are critical for a broad range of network activities. Such a decision can be based on packet header fields, packet payload content, statistical characteristics of traffic and communication patterns of network hosts. However, most present techniques rely on some sort of apriori knowledge, which means they require labor-intensive preprocessing before running and cannot deal with previously unknown applications. In this paper, we propose a traffic classification system based on application signatures, with a novel approach to fully automate the process of deriving signatures from unidentified traffic. The key idea is to integrate statistics-based flow clustering with payload-based signature matching method, so as to eliminate the requirement of pre-labeled training data sets. We evaluate the efficiency of our approach using real-world traffic trace, and the results indicate that signature classifiers built from clustered data and pre-labeled data are able to achieve similar high accuracy better than 99%.

Internet traffic classification using machine learning : a token-based approach

Due to the increasing unreliability of traditional port-based methods, Internet traffic classification has attracted a lot of research efforts in recent years. Quite a lot of previous papers have focused on using statistical characteristics as discriminators and applying machine learning techniques to classify the traffic flows. In this paper, we propose a novel machine learning based approach where the features are extracted from packet payload instead of flow statistics. Specifically, every flow is represented by a feature vector, in which each item indicates the occurrence of a particular token, i.e.; a common substring, in the payload. We have applied various machine learning algorithms to evaluate the idea and used different feature selection schemes to identify the critical tokens. Experimental result based on a real-world traffic data set shows that the approach can achieve high accuracy with low overhead.

A novel semi-supervised approach for network traffic clustering

Network traffic classification is an essential component for network management and security systems. To address the limitations of traditional port-based and payload-based methods, recent studies have been focusing on alternative approaches. One promising direction is applying machine learning techniques to classify traffic flows based on packet and flow level statistics. In particular, previous papers have illustrated that clustering can achieve high accuracy and discover unknown application classes. In this work, we present a novel semi-supervised learning method using constrained clustering algorithms. The motivation is that in network domain a lot of background information is available in addition to the data instances themselves. For example, we might know that flow ƒ1 and ƒ2 are using the same application protocol because they are visiting the same host address at the same port simultaneously. In this case, ƒ1 and ƒ2 shall be grouped into the same cluster ideally. Therefore, we describe these correlations in the form of pair-wise must-link constraints and incorporate them in the process of clustering. We have applied three constrained variants of the K-Means algorithm, which perform hard or soft constraint satisfaction and metric learning from constraints. A number of real-world traffic traces have been used to show the availability of constraints and to test the proposed approach. The experimental results indicate that by incorporating constraints in the course of clustering, the overall accuracy and cluster purity can be significantly improved.

Discriminating DDoS attacks from flash crowds using flow correlation coefficient

Distributed Denial of Service (DDoS) attack is a critical threat to the Internet, and botnets are usually the engines behind them. Sophisticated botmasters attempt to disable detectors by mimicking the traffic patterns of flash crowds. This poses a critical challenge to those who defend against DDoS attacks. In our deep study of the size and organization of current botnets, we found that the current attack flows are usually more similar to each other compared to the flows of flash crowds. Based on this, we proposed a discrimination algorithm using the flow correlation coefficient as a similarity metric among suspicious flows. We formulated the problem, and presented theoretical proofs for the feasibility of the proposed discrimination method in theory. Our extensive experiments confirmed the theoretical analysis and demonstrated the effectiveness of the proposed method in practice.

Internet traffic clustering with constraints

Due to the limitations of the traditional port-based and payload-based traffic classification approaches, the past decade has seen extensive work on utilizing machine learning techniques to classify network traffic based on packet and flow level features. In particular, previous studies have shown that the unsupervised clustering approach is both accurate and capable of discovering previously unknown application classes. In this paper, we explore the utility of side information in the process of traffic clustering. Specifically, we focus on the flow correlation information that can be efficiently extracted from packet headers and expressed as instance-level constraints, which indicate that particular sets of flows are using the same application and thus should be put into the same cluster. To incorporate the constraints, we propose a modified constrained K-Means algorithm. A variety of real-world traffic traces are used to show that the constraints are widely available. The experimental results indicate that the constrained approach not only improves the quality of the resulted clusters, but also speeds up the convergence of the clustering process.