O que é Algorithmic differentiation, forward and reverse modes, dual numbers?

Algorithmic differentiation computes derivatives of a function given as a program by applying the chain rule to its elementary operations, exactly up to rounding. Forward mode propagates, with each value, its derivative along one input direction; dual numbers $a + b\epsilon$ with $\epsilon^2 = 0$ implement it, since $f(a + b\epsilon) = f(a) + f'(a)b\epsilon$ (Wengert, 1964). Reverse mode records the operations and then propagates adjoints $\bar v = \partial y/\partial v$ from the output back to every input (Linnainmaa, 1976, from his 1970 master’s thesis on rounding errors; Griewank, 2012, recounts several independent discoveries); in finance it is called adjoint mode, or AAD.