Attention mechanism, transformer क्या है?

An attention mechanism lets each position of a sequence form its output as a weighted average of all positions: with queries $\mathsf Q = ZA_{\mathsf Q}$, keys $\mathsf K = ZA_{\mathsf K}$ and values $\mathsf V = ZA_{\mathsf V}$ computed from the sequence $Z$ ($T\times h$), the output is $\mathrm{softmax}(\mathsf Q\mathsf K^\top/\sqrt h)\mathsf V$. A transformer layer combines several attention heads, a position-wise network, residual connections and layer normalisation; positions are encoded by adding learned or fixed vectors to the inputs (Vaswani et al., 2017).