O que é Entropy, mutual information?

The entropy of a discrete random variable $X$ is $H(X) = -\sum_xp(x)\log_2p(x)$ bits (Shannon, 1948). The mutual information of $X$ and $Y$ is $I(X; Y) = \sum_{x,y}p(x, y)\log_2\frac{p(x, y)}{p(x)p(y)} = H(X) - H(X \mid Y)$, the Kullback–Leibler divergence of the joint law from the product of its marginals.