• Harshada Chandrakant Gore
• Akash Ambadas Bhagwat
• Shailesh Pandharinath Dhome

Gradient Descent (GD) is a foundational optimization technique widely used in machine learning for minimizing objective functions, particularly in training neural networks and other models. This paper provides a mathematical exploration of the gradient descent algorithm, covering its basic principles and essential variants, such as Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent, Momentum-based methods, AdaGrad, RMSprop, and Adam. By examining each variant’s mathematical formulation, convergence properties, and practical impact on optimization speed and accuracy, we highlight the trade-offs involved in their application. The comparative analysis reveals how these variants address limitations of traditional gradient descent, such as slow convergence and sensitivity to local minima, thereby enhancing model performance and efficiency in large-scale and complex learning tasks. This paper aims to equip machine learning practitioners with insights into choosing appropriate optimization strategies based on theoretical understanding, problem requirements, and empirical considerations.

Gradient Descent, Machine Learning, Optimization, SGD, Mini-Batch, Convergence, Momentum, AdaGrad, RMSprop, Adam, Neural Networks, Loss Minimization, Learning Rate, Large-scale, Non-convex.

"Mathematical approaches to gradient descent and its variants in Machine Learning", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.11, Issue 11, page no.d823-d828, November-2024, Available :http://www.jetir.org/papers/JETIR2411396.pdf

"Mathematical approaches to gradient descent and its variants in Machine Learning", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.11, Issue 11, page no. ppd823-d828, November-2024, Available at : http://www.jetir.org/papers/JETIR2411396.pdf