The optimizer revolution: how training neural networks is changing faster than the networks themselves

If training a neural network were driving a car, we used to just have one pedal (Adam). Now we have steering wheels that work on multiple levels, cruise control that adjusts itself, and even special fuel for different terrain.

This means the real innovation in AI isn't just bigger models — it's smarter, more efficient ways to teach them, which could make AI cheaper and faster to deploy at scale.