• Mukesh Kumar Saini
• Arun Saini

There are currently many pipeline tools available for data engineering, which data scientists use to tackle data wrangling issues and perform tasks ranging from data ingestion and preparation to its use as input for machine learning (ML). These tools either come with essential built-in components or can be integrated with others to achieve the desired data engineering outcomes. While some of these tools are fully or partially commercial, there is also a range of open-source options capable of handling expert-level data engineering tasks. This survey explores various categories and examples of these pipeline tools based on their design and intended data engineering functions. The categories covered include Extract Transform Load/Extract Load Transform (ETL/ELT), Data Integration, Ingestion, and Transformation pipelines, Data Pipeline Orchestration and Workflow Management, and Machine Learning Pipelines. Additionally, the survey outlines how these tools are used within these categories, providing examples and discussing case studies that showcase real-world applications. These case studies illustrate user experiences with sample data, the complexities involved in the pipelines, and summarize approaches for preparing data for machine learning.

Artificial Intelligence (AI), Data Engineering, Structured Data, Unstructured Data ETL, Machine Learning, Data Integration, Data Pipeline etc.

"Optimizing Data Engineering Through a Comprehensive Exploration of Modern Pipeline Tools ", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.7, Issue 2, page no.925-936, February-2020, Available :http://www.jetir.org/papers/JETIR2002539.pdf

"Optimizing Data Engineering Through a Comprehensive Exploration of Modern Pipeline Tools ", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.7, Issue 2, page no. pp925-936, February-2020, Available at : http://www.jetir.org/papers/JETIR2002539.pdf