• Madhuri Kanojiya
• Lokesh Chouhan

Cybersecurity experts predict that the cost of damage from cyber attacks will rise to $9.2 billion in 2019, with a new attack occurring every few seconds. Managing the vast amount of data generated daily presents a significant challenge for traditional intrusion detection systems. Protecting sensitive information is a priority for both governments and businesses, emphasizing the need for a real-time, large-scale, and robust intrusion detection system (IDS). This paper introduces a distributed, fault-tolerant, and scalable IDS that leverages Apache Spark's Structured Streaming and machine learning capabilities to detect intrusions in real time. The system is implemented on Microsoft Azure, which offers both processing power and storage capabilities. A decision tree algorithm is employed to classify incoming data. By using a machine learning dataset as the data source, the system gains enhanced insights into its ability to respond to cyber attacks. Experimental results demonstrated a high accuracy of 99.95% and processed over 55,175 events per second using a small cluster.

interruption location framework; ML; Apache Spark; Streaming Structured; Data; Decision Trees; Azure Cloud of Microsoft

"Real-Time Cybersecurity: Leveraging Apache Spark and Machine Learning for Effective Intrusion Detection in Azure Cloud Environment", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.6, Issue 1, page no.505-510, January-2019, Available :http://www.jetir.org/papers/JETIR1901I68.pdf

"Real-Time Cybersecurity: Leveraging Apache Spark and Machine Learning for Effective Intrusion Detection in Azure Cloud Environment", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.6, Issue 1, page no. pp505-510, January-2019, Available at : http://www.jetir.org/papers/JETIR1901I68.pdf