Computational Statistics & Data Analysis (MVComp2)

Lecture 14: Information theory

Tristan Bereau

Institute for Theoretical Physics, Heidelberg University

Intuition

Entropy

Measure of uncertainty, or lack of predictability, associated with a random variable drawn from a given distribution.

Information content: Observe a sequence $X_{n} \sim p$ generated from distribution $p$ . If $p$ has high entropy, it will be hard to predict the value of each observation $X_{n}$ . Hence the dataset has high information content. On the other hand, a distribution with 0 entropy will always yield the same $X_{n}$ , so the dataset does not contain much information. Link to data compression.

Entropy for discrete random variables¹

Consider a discrete random variable $X$ with distribution $p$ over $K$ states. The entropy is defined by² ³ $H (X) = - \sum_{k = 1}^{K} p (X = k) \log_{2} p (X = k) = - E_{X} [\log_{2} p (X)]$

Maximum entropy: The discrete distribution with maximum entropy is the uniform distribution (see information content in previous slide). Maximum uncertainty.
Minimum entropy: The distribution with minimum entropy (which is zero) is any delta function, putting all its mass on one state. This distribution has no uncertainty.

Cross entropy¹

Cross entropy

The cross entropy between distribution $p$ and $q$ is defined by $H_{ce} (p, q) = - \sum_{k = 1}^{K} p_{k} \log q_{k}$

Intuition: Expected number of bits needed to compress some data samples drawn from distribution $p$ using a code based on distribution $q$ . This can be minimized by setting $q = p$ , in which case the expected number of bits of the optimal code is $H_{ce} (p, p) = H (p)$ .

Joint entropy¹

Joint entropy

The joint entropy of two random variables $X$ and $Y$ is defined as $H (X, Y) = - \sum_{x, y} p (x, y) \log_{2} p (x, y)$

Example: Consider choosing an integer from 1 to 8. Let $X (n) = 1$ if $n$ is even and $Y (n) = 1$ is $n$ is prime, i.e.,

$n$	1	2	3	4	5	6	7	8
$X$	0	1	0	1	0	1	0	1
$Y$	0	1	1	0	1	0	1	0

The joint distribution is

$p (X, Y)$	$Y = 0$	$Y = 1$
$X = 0$	$\frac{1}{8}$	$\frac{3}{8}$
$X = 1$	$\frac{3}{8}$	$\frac{1}{8}$

So the joint entropy yields $H (X, Y) = - [\frac{1}{8} \log_{2} \frac{1}{8} + \dots] = 1.81 bits$ .

On the other hand, the marginal probabilities are uniform, $p (X = 0) = p (X = 1) = p (Y = 0) = p (Y = 1) = 0.5$ , so $H (X) = H (Y) = 1$ . As such we have $H (X, Y) < H (X) + H (Y)$ which serves as an upper bond on the joint entropy (for when $X$ and $Y$ are independent).

Lower bound on $H (X, Y)$ : If $Y$ is a deterministic function of $X$ , then $H (X, Y) = H (X)$ .

Conditional entropy¹

Conditional entropy

The conditional entropy of $Y$ given $X$ is the uncertainty we have in $Y$ after seeing $X$ , averaged over possible values for $X$ $\begin{aligned} H (Y | X) & = E_{p (X)} [H (p (Y | X))] \\ = \sum_{x} p (x) H (Y | X = x) \\ = - \sum_{x} p (x) \sum_{y} p (y | x) \log_{2} p (y | x) \\ = - \sum_{x, y} p (x) p (y | x) \log_{2} p (y | x) \\ = - \sum_{x, y} p (x, y) \log_{2} \frac{p (x, y)}{p (x)} \\ = - \sum_{x, y} p (x, y) \log_{2} p (x, y) + \sum_{x} p (x) \log_{2} p (x) \\ = H (X, Y) - H (X) \end{aligned}$

If $Y$ is a deterministic function of $X$ , then knowing $X$ completely determines $Y$ , so that $H (Y | X) = 0$ .
If $X$ and $Y$ are independent, knowing $X$ tells us nothing about $Y$ and $H (Y | X) = H (Y)$ .

KL divergence and MLE¹

Objective: Find the distribution $q$ that is as close as possible to $p$ , as measured by the KL divergence: $\begin{aligned} q^{*} & = \arg min_{q} D_{KL} (p ‖ q) \\ = \arg min_{q} \int d x p (x) \log p (x) - \int d x p (x) \log q (x) \end{aligned}$

Suppose that $p$ is the empirical distribution, i.e., place probability mass only on the training data and zero mass everywhere else $p_{D} (x) = \frac{1}{N} \sum_{n = 1}^{N} δ (x - x_{n})$

Replace $p (x)$ by $p_{D} (x)$ $\begin{aligned} D_{KL} (p_{D} ‖ q) & = - \int d x p (x) \log q (x) + C \\ = - \int d x [\frac{1}{N} \sum_{n} δ (x - x_{n})] \log q (x) + C \\ = - \frac{1}{N} \sum_{n} \log q (x_{n}) + C \end{aligned}$ where $C = \int d x p (x) \log p (x)$ is a constant independent of $q$ . This is called the cross-entropy objective, and is equal to the average negative log likelihood of $q$ on the training set.

Minimizing KL divergence to the empirical distribution is equivalent to maximizing likelihood.

Forward and reverse KL¹

Definitions

Forward KL

$D_{KL} (p ‖ q) = \int d x p (x) \log \frac{p (x)}{q (x)}$ Minimize wrt $q$ is known as a moment projection. If $p (x) > 0$ but $q (x) = 0$ , the $\log$ term will be infinite. Will force $q$ to include areas where $p$ is non-zero. $q$ will be mode-covering. It will typically over-estimate the support of $p$ . See panel (a).

Reverse KL

$D_{KL} (q ‖ p) = \int d x q (x) \log \frac{q (x)}{p (x)}$ Minimize wrt $q$ is known as an information projection. In any region where $p (x) = 0$ but $q (x) > 0$ , the $\log$ term will be infinite. Will force $q$ to exclude all areas where $p$ has zero probability. Will place probability mass in very few parts of space—called mode-seeking behavior. It will typically under-estimate the support of $p$ . See panels (b) and (c).

1 / 30

Computational Statistics & Data Analysis (MVComp2) Lecture 14: Information theory Tristan Bereau Institute for Theoretical Physics, Heidelberg University

Computational Statistics & Data Analysis (MVComp2)
Table of contents
Commercial break
Course evaluation
Soft Matter Physics
Machine learning for the biomolecular world
Introduction
Literature
Recap from last time
Entropy
Intuition
Entropy for discrete random variables1
Entropy for a binary random variables
Cross entropy1
Joint entropy1
Conditional entropy1
Relative entropy
Relative entropy1
Interpretation of the KL divergence
Example: KL divergence between two gaussians1
KL divergence and MLE1
KL divergence and MLE1
Forward and reverse KL1
Mutual information
Mutual information1
Interpretation1
Summary
Summary
References
Murphy, Kevin P....

Computational Statistics & Data Analysis (MVComp2)

Commercial break

Course evaluation

Soft Matter Physics

Machine learning for the biomolecular world

Introduction

Literature

Recap from last time

Entropy

Intuition

Entropy for discrete random variables¹

Entropy for a binary random variables

Cross entropy¹

Joint entropy¹

Conditional entropy¹

Relative entropy

Relative entropy¹

Relative entropy

Kullback-Leibler (KL) divergence

Interpretation of the KL divergence

Example: KL divergence between two gaussians¹

KL divergence and MLE¹

KL divergence and MLE¹

Forward and reverse KL¹

Forward KL

Reverse KL

Mutual information

Mutual information¹

Interpretation¹

Summary

Summary

References