Machine Learning Enhanced Entropy-Based Network Anomaly Detection

Abstract

The advanced development of new technologies and heterogeneous environments relies on the proper processing of large data volumes, and accurate and fast response of real-time applications. Such circumstances provide a fertile ground for the appearance of diverse security concerns, thus challenging the scientific community for building more reliable and efficient Network Anomaly Detection Systems. This research proposes a comprehensive flow-based anomaly detection architecture, which encompasses techniques for entropy-based data processing and machine learning-based attack detection. It encompasses several attack categories and relies on the use of modelled and synthetically generated traffic patterns for Port Scan, Network Scan, DDoS amplification, flood, and dictionary attacks. The entropy-based analysis is used for easier detection of the hidden traffic patterns, as it can capture the behaviour of the biggest contributors, and of a large number of minor appearances in the feature distribution. The unusual traffic is then processed by the use of unsupervised machine learning algorithms. The approach is verified with datasets based on real network traffic, synthetically generated attack traffic instances and botnet traffic. The architecture is an original solution, planned for further real-network application, targeting the possible support for a range of different use cases.