ADAM Optimizer Vs Gradient Descent
Adam also know as Adaptive momentum Estimation is an important Optimization Algorithm commonly used and most prefferable for Machine Learning and Deep Learning tasks
It combines combining Momentum calculation and effective learning rate calculation
Momentum is the method of smooth gradients update by using memorised previous gradients, and as of for Adam it uses previous two gradients value
Get Akshat Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Gradient Descent is an iterative algorithm that updates model parameter with each iteration to find best/optimal set of model parameters, and hence is typically slower for larger and complex algorithms.
But Adam calculates exponential weighted average of both gradient and it’s squared value
Former value is used for smooth and effective gradient update, meanwhile Latter is used for effective updation of learning rate. Also Bias correction is done at each stage, that is important for early training of model
Further Adam is preffered over other algorithms as it quickly converges toward global minimum, Works fine with noisy small batches of data, Initial hyparameter tuning, and many more reasons
But considering Adam as the best algorithm is also not always the right choice, We should Always consider other algorithms like SGD , RMS and others according to the results we get from comparing all these algorithms for different type of multiple models, trained on same dataset, with same hyperparameters, data spilliting, data cleaning and other methods
