• praveen j
• Dr . Rajesh Kumar B

Cloud-based Spark data pipelines in product-based companies process large volumes of financial and customer data on Amazon EMR, and inefficiencies in storage layout, executor configuration, and pre-production validation translate directly into recurring infrastructure cost. This paper presents a practical, config-first approach to cost optimization for data engineering pipelines, built around three complementary techniques: S3 small-file compaction through partition-aware REPARTITION hints, Spark job tuning through executor sizing, Adaptive Query Execution (AQE), and dynamic allocation, and pre-production verification of every change through an automated EMR trigger-and-test workflow before the change reaches production.

Cost optimisation

"Cost Optimization in Data Engineering", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.13, Issue 9, page no.c884-c888, September-2026, Available :http://www.jetir.org/papers/JETIR2609299.pdf

"Cost Optimization in Data Engineering", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.13, Issue 9, page no. ppc884-c888, September-2026, Available at : http://www.jetir.org/papers/JETIR2609299.pdf