• Ashitosh Chitnis
• Shishir Tewari

Abstract Modern applications dependent on data need to focus on data quality and reliability because it maintains reliable performance and trustworthy operations of machine learning models and analytical systems. Data observability systems and drift detection mechanisms operated by computers serve as essential elements for tracking incoming data stream accuracy and consistency. The usage of traditional methods to discover data drift and inconsistencies depends on manual human involvement and rule-based methods while showing inefficiency when applied to big and real-time data processes. The paper presents a study about how machine learning methods can perform autonomous drift detection and data observability to spot data distribution abnormalities beforehand. Our system based on machine learning uses statistical models together with anomaly detection algorithms and explainable AI (XAI) features which strengthens decision quality and interpretation in data monitoring processes. This paper evaluates the scalability issues along with false positives and describes real-time processing limitations before discussing AI-based adaptive drift correction systems of the future. The evaluation focuses on different drift detection approaches through a real-life example which proves that AI-powered tools help maintain data reliability.

Data Observability, Drift Detection, Machine Learning Automation, Anomaly Detection, Real-Time Data Monitoring

"AUTOMATED DATA OBSERVABILITY AND DRIFT DETECTION USING MACHINE LEARNING", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.9, Issue 8, page no.g282-g295, August-2022, Available :http://www.jetir.org/papers/JETIR2208636.pdf

"AUTOMATED DATA OBSERVABILITY AND DRIFT DETECTION USING MACHINE LEARNING", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.9, Issue 8, page no. ppg282-g295, August-2022, Available at : http://www.jetir.org/papers/JETIR2208636.pdf