Optimal decision

Pick the alternative hypothesis, \(M_1\), iff \[ p(M_1 | \mathcal{D}) > p(M_0 | \mathcal{D}), \quad \Leftrightarrow \frac{p(M_1 | \mathcal{D})}{p(M_0 | \mathcal{D})} > 1 \]

Uniform prior

Using a uniform prior, \(p(M_0) = p(M_1) = 0.5\), we have (from Bayes theorem): \[ \text{pick} \ M_1 \ \text{iff} \ \frac{p(\mathcal{D} | M_1)}{p(\mathcal{D} | M_0)} > 1 \]

Bayes factor

This quantity is the ratio of the marginal likelihoods,² also known as Bayes factor: \[ \boxed{ B_{1,0} = \frac{p(\mathcal{D} | M_1)}{p(\mathcal{D} | M_0)} } \]

Two outcomes: Bernoulli² likelihood and conjugate prior \(\text{Beta}(\alpha_1, \alpha_0)\) (i.e., the beta-Bernoulli model).

Marginal likelihood under \(M_0\) is simple because the integral over \(\theta\) yields a single case \(\theta=\frac 12\) \[ p(\mathcal{D} | M_0) = \left(\frac 12\right)^{N_1}\left(1-\frac 12\right)^{N-N_1}= \left( \frac 12 \right)^N \]

For \(M_1\) it’s a bit more complicated \[ p(\mathcal{D} | M_1) = \int {\rm d}\theta \, p(\mathcal{D} | \theta) p(\theta) = \frac{B(\alpha_1 + N_1, \alpha_0 + N_0)}{B(\alpha_1, \alpha_0)} \]

Interpretation

Would probably choose \(M_0\) for 2 or 3 Heads, \(M_1\) for 0 or 5 Heads, case 1 or 4 is unclear.

Decision theory

The action space requires choosing one model, \(m \in \mathcal{M}\). Pick the most probable \[ \hat m = \underset{m \in \mathcal{M}}{\arg \max} \, p(m | \mathcal{D}) = \underset{m \in \mathcal{M}}{\arg \max} \, \frac{p(\mathcal{D} | m)p(m)}{\sum_{m \in \mathcal{M}} p(\mathcal{D} | m) p(m)} \] i.e., maximize the posterior² over models.

MAP model for uniform priors

For unform priors, \(p(m) = \frac 1{\vert\mathcal{M}\vert}\), the MAP simplifies to \[ \hat m = \underset{m \in \mathcal{M}}{\arg \max} \, p(\mathcal{D} | m) = \underset{m \in \mathcal{M}}{\arg \max} \, \int {\rm d}\pmb \theta \, p(\mathcal{D} | \pmb \theta, m) p(\pmb \theta | m) \] is known as the marginal likelihood, or the evidence for model \(m\).

If all settings of \(\pmb \theta\) assign high probability to the data, this is probably a good model.

We write the individual terms \[ p(y_n | \pmb x , \mathcal{D}_{1:n-1}, m) = \int {\rm d}\pmb \theta \, p(y | \pmb x, \pmb \theta) p(\pmb \theta | \mathcal{D}_{1:n-1}, m) \] Use a plug-in approximation (i.e., replacing an expression by its empirical estimate) \[ \begin{align*} p(y_n | \pmb x , \mathcal{D}_{1:n-1}, m) &\approx \int {\rm d}\pmb \theta p(y | \pmb x, \pmb \theta) \delta ( \pmb \theta - \pmb{\hat\theta}_m(\mathcal{D}_{1:n-1})) \\ &= p(y | \pmb x, \pmb{\hat\theta}_m(\mathcal{D}_{1:n-1})) \end{align*} \]

Then we get \[ \log p(\mathcal{D} | m) \approx \sum_{n=1}^N \log p(y_n | \pmb x_n, \pmb{\hat \theta}_m(\mathcal{D}_{1:n-1})) \] which is similar to a leave-one-out cross-validation estimate of the likelihood!

Intuition: overly complex model will overfit the “early” examples and predict the remaining ones poorly, and thus also get a low cross-validation score.

Now Taylor expand around the mode \(\pmb{\hat \theta}\) (i.e., the lowest energy state) \[ \varepsilon(\pmb \theta) \approx \varepsilon(\pmb{\hat \theta}) + {\color{grey}(\pmb \theta - \pmb{\hat\theta})^\intercal \pmb g } + \frac 12 (\pmb \theta - \pmb{\hat\theta})^\intercal {\bf H}(\pmb \theta - \pmb{\hat\theta}) \] (\({\bf H}\) is the Hessian of the negative log joint, \(-\log p(\pmb \theta, \mathcal{D})\)) and so we get for both the joint distribution and the posterior: \[ \begin{align*} \hat p(\pmb \theta, \mathcal{D}) &= \frac 1Z {\rm e}^{-\varepsilon(\pmb{\hat\theta})} \exp\left[- \frac 12 (\pmb \theta - \pmb{\hat\theta})^\intercal {\bf H}(\pmb \theta - \pmb{\hat\theta})\right] \\ \hat p(\pmb \theta | \mathcal{D}) &= \frac 1Z \hat p(\pmb \theta, \mathcal{D}) = \mathcal{N}(\pmb \theta | \pmb{\hat\theta}, {\bf H}^{-1}) \end{align*} \] which means that the posterior approximates a multivariate Gaussian³

Make use of the quadratic approximation to the posterior, \(\hat p(\pmb\theta | \mathcal{D})\), and the MAP plug-in approximation when integrating \(\hat p(\pmb\theta | \mathcal{D}) = \mathcal{N}(\pmb{\theta} | \hat{\pmb{\theta}}, \textbf{H}^{-1})\) \[ \log p(\mathcal{D} | m) \approx \log p(\mathcal{D} | \pmb{\hat\theta}_\text{map}) + \log p(\pmb{\hat \theta_\text{map}}) - \frac 12 \log \vert {\bf H} \vert \]

Assume uniform prior and replace MAP estimate with the MLE, \(\pmb{\hat \theta}\) \[ \log p(\mathcal{D} | m) \approx \log p(\mathcal{D} | \pmb{\hat\theta}) - \frac 12 \log \vert {\bf H}\vert \]

We define the BIC loss² \[ \mathcal{L}_\text{BIC}(m) = -2 \log p(\mathcal{D} | \pmb{\hat\theta}, m) + k \log N \]