Optimization Algorithms Visualization

1. Gradient Descent

a. Visualization in 3d :

b. Visualization in 2d :

2. Momentum Gradient Descent

a. Visualization in 3d :

b. Visualization in 2d :

3. Nesterov Accelerated Gradient Descent ( NAG )

a. Visualization in 3d :

b. Visualization in 2d :

4. AdaGrad

a. Visualization in 3d :

b. Visualization in 2d :

5. RMSprop

a. Visualization in 3d :

b. Visualization in 2d :

6. Adam

a. Visualization in 3d :

b. Visualization in 2d :

Experiaments

1. Try out with dffernet sets of data
2. Try out with differnet learning rates , epochs, weight Initializations, and visualize the plots.

Resources

1. Paper - An overview of gradient descent optimization algorithms
2. Blog - Overview of Optimization Algorithms