Gradient Descent Explained: SGD, Adam & Optimization Algorithms

Gradient descent minimizes loss by stepping along the negative gradient. Optimizers add momentum, adaptive steps, and regularization to make this efficient and stable.

Core algorithms: SGD (+ momentum), RMSProp, Adam/AdamW. Prefer AdamW (decoupled weight decay) for transformers; use SGD+momentum for vision models when data is large.

Learning rate: warm up for a few hundred to a few thousand steps; decay with cosine schedule. Use per‑parameter groups to set lower LR for embeddings and higher for heads.

Regularization: weight decay (avoid applying to biases/norms), dropout, data augmentation.

Practical defaults: batch size 32–256, AdamW β1=0.9 β2=0.95, weight decay 0.01, cosine decay with 5% warmup. Tune LR first; it dominates performance.

Debugging tips: plot loss vs steps; watch for divergence (NaN); reduce LR or gradient clip; verify data normalization.