Shashank Agarwal·2d agoFrom SGD to AdamW: Evolution of Optimizers in Machine LearningA short tour of the optimizers you use every day and why each one exists.
Jennifer Wei·Oct 25Taming Orthogonal Optimizers at Scale (Part 2): The Anatomy of a Silent Tensor-Aliasing BugIn Part 1, I set out to integrate $O(n³)$ orthogonal optimizers — specifically Muon and Microsoft Research’s Dion3 — into AI2’s olmo-core…
InTowards AIbyJAIGANESAN·Oct 4Muon: The Orthogonalized Optimizer Explained Through Equations, Intuition, Code and ResultsFrom SGD, to Adam’s per-coordinate scaling, to the five matrix multiplications that turn a gradient into an orthogonal updateA response icon2
Ojas Arora·Sep 8DLS Part 3 : SGD Se Adam Tak — Ek Optimizer Doosre Ki Galti Se Paida HuaSGD slow tha, Momentum ne speed di lekin overshoot kiya, NAG ne thoda sambhala, AdaGrad ne learning rate tune kiya lekin mar gaya, RMSProp…
Muhammad Tabish Nazir·Aug 12Can Adam Be Made More Memory Efficient? Exploring Rank-Projected Momentum, a One-State OptimizerA research investigation into whether useful adaptive optimization can survive without storing a persistent second-moment tensor
