¿Qué es Adam, weight decay?

Adam is stochastic gradient descent (Book 4, chapter 24) with a per-parameter step: with gradient $g_k$ at step $k$, it keeps averages $m_k = \beta_1m_{k-1} + (1-\beta_1)g_k$ and $v_k = \beta_2v_{k-1} + (1-\beta_2)g_k^2$ and moves $\theta$ by $-\eta\,\hat m_k/(\sqrt{\hat v_k} + \epsilon)$, where hats undo the averages’ bias towards zero (Kingma and Ba). Weight decay shrinks every weight by $\eta\lambda\theta$ at each step, apart from the gradient step (AdamW, Loshchilov and Hutter); for plain gradient descent it is a ridge penalty.