What is Pinball loss, continuous ranked probability score?

The pinball loss of a forecast $q$ of the $\alpha$-quantile is $\ell_\alpha(y, q) = \max\{\alpha(y - q), (\alpha - 1)(y - q)\}$. The continuous ranked probability score of a forecast distribution $F$ is $\mathrm{CRPS}(F, y) = \int(F(z) - \mathbf 1\{z\ge y\})^2\,dz = 2\int_0^1\ell_\alpha(y, F^{-1}(\alpha))\,d\alpha$, in the units of the target; for a Gaussian forecast it has a closed form.