The analysis of network flows for security is not new and has been adopted in both network and security industries for more than a decade. It was originally invented for high speed switching but has also been used in Distributed Denial-of-Service (DDoS) attack detection.


Recently, with the help of technological advancements in machine learning and streaming analytics, it is getting renewed attention as a countermeasure to rapidly evolving cyber attacks by globally syndicated adversaries beyond DDoS attacks. Additionally, as internet traffics are encrypted at an accelerated rate, the meta information, such as network flow, is becoming the only available data for the analysis anyway.


New large-scale network analytics technology


NTT Security is staying ahead of the curve and has successfully launched new capabilities that leverage the analysis of network flow data from its world leading ICT infrastructures. By applying machine learning to the internet scale network analytics, we are able to produce a large amount of high quality blacklists of Command and Control (C&C) servers that are detected up to two weeks earlier than major vendors. 


Another benefit of network flow analytics is the quick detection of targeted attacks. As you can imagine, flow patterns tell us if a given attack is random (i.e. it flows to many organizations) or is targeted (i.e. flows to a specific organization). NTT Group customers can immediately enjoy the benefits through NTT Security’s Managed Security Services.


The internet scale network analytics provides us with a more complete understanding of botnet infrastructures as they are being formed in real-time. This includes the location and nature of C&C servers, bots under the control, and ultimately who is behind the infrastructure. The analysis enables us to block and ultimately take down the botnet infrastructure from which attackers launch many types of attacks, such as DDoS and ransomware distributions. 


The analysis results are represented as real-time enriched blacklists. Applications of such blacklists are limitless, including threat feeds to third party security solutions for example Intrusion Detection System (IDS), Intrusion Prevention System (IPS), and secure web gateways.


Advancements in network flow analysis 


We have been developing and deploying technologies for network flow analysis at enterprise scale, while accumulating skills and knowledge of them. The breakthrough has been made possible with the combination of these experiences, access to large amount of flow data, and advancements we have made in these two areas:


1. Big streaming analytics platform


Recently, open source projects in streaming analytics, such as Apache Kafka, Spark and Flink, are becoming very active and producing innovative software. NTT Group is actively participating in those projects and has built a massively scalable, very fast streaming platform by leveraging the best of breeds of open source software. Our platform can easily handle the pipeline processing of over a hundred of thousands of flows per second, and enables us to apply advanced analytics to a large amount of data streams in a massively scalable manner. The platform is designed to be modular and immutable by adopting modern software architecture, such as containerization. Together with continuous integration and delivery process, the architecture brings low maintenance costs, agility and resilience.


2. Data quality improvement

Arguably, for machine learning, the quality of data used in training is the most important factor that determines the overall performance. The training data is just like textbook in education – you cannot learn things very well when your textbook is poorly written or even wrong. We have invented a patent pending algorithm and applied it to our original data set to improve and expand the data set. Our original data set itself is one of the most comprehensive in the industry, which is collected from multiple sources, including those gained through our global delivery of security services as well as validated, unique data from NTT R&D proprietary honey clients.


At NTT Security, we strive to protect our NTT Group customers from new and existing threats by expanding the borders of network flow analysis in many fronts. For example, we are working on correlation with results from passive Domain Name System (DNS) data and flow analysis to improve accuracy and coverage. We are also exploring the possibility of integrating with internet scale active scanning (and doing it in the IPv6 space). Lastly, we plan to expand our frontier to Operation Technology (OT). In OT environments, non-IP proprietary protocols are often used but flow patterns are statistically predictable. Because of these characteristics, OT environments lend themselves well to flow based anomaly detection.


For more information on our new botnet infrastructure detection capabilities, click here.