KNN classification¹

Curse of dimensionality

Linear classifier¹

• Sigmoid: probability that the class label is \(y=1\).
• Optimal decision: predict \(y=1\) iff class 1 is more likely than class 0

\[ \begin{align*} f(\pmb{x}) &= \mathbb{I}\left[p(y=1| \pmb{x}) > p(y=0 | \pmb{x})\right] \\ &= \mathbb{I} \left( \log{\frac{p(y=1 | \pmb{x})}{p(y=0 | \pmb{x})} > 0} \right) \\ &= \mathbb{I}(a > 0) \end{align*} \]

Maximum likelihood estimation

Absorb the bias term \(b\) into the weight vector \(\pmb{w}\) \[ \begin{align*} \text{NLL}\pmb{w} &= - \frac{1}{N} \log p(\mathcal{D} | \pmb{w}) = -\frac{1}{N} \log \prod_{n=1}^N \text{Ber}(y_n | \mu_n) \\ &= \frac 1N \sum_{n=1}^N [y_n \log \mu_n + (1-y_n)\log(1-\mu_n)] \\ &= \frac 1N \sum_{n=1}^N \mathbb{H}(y_n, \mu_n) \end{align*} \] where \(\mathbb{H}\) is the binary cross entropy.¹

MLE: Optimizing the objective

Finding the MLE corresponds to solving \[ \nabla_{\pmb{w}} \text{NLL}(\pmb{w}) = 0. \] To start, note that for \(\mu_n = \sigma(a_n)\) and \(a_n = \pmb{w}^\intercal\pmb{x}_n\), \[ \frac{\text{d}\mu_n}{\text{d}a_n} = \frac{\text{d}\sigma(a_n)}{\text{d}a_n} = \frac{\text{d}}{\text{d}a_n}\left( \frac 1{1 + \text{e}^{-a_n}}\right) = \sigma(a_n)\left(1-\sigma(a_n)\right). \]

Nonlinear classifier¹

Same trick as for nonlinear regression

preprocess the inputs in a suitable way. Transform input feature vector using \(\pmb \phi(\pmb x)\)

Gaussian discriminant analysis¹

Consider multivariate Gaussians as class conditional densities \[ p(\pmb x | y = c, \pmb \theta) = \mathcal{N}(\pmb x | \pmb \mu_c, {\bf \Sigma}_c) \] which leads to the class posterior \[ p(y = c | \pmb x, \pmb \theta) \propto \pi_c \mathcal{N}(\pmb x | \pmb \mu_c, {\bf \Sigma}_c) \] where \(\pi_c = p(y = c | \pmb \theta)\) is the prior probability of label \(c\).

Quadratic decision boundaries

The log posterior over class labels \[ \begin{align*} \log p(y=c | \pmb{x}, \pmb{\theta}) =& \log \pi_c - \frac 12 \log \vert 2\pi \Sigma_c \vert \\ &- \frac 12 (\pmb{x} - \pmb{\mu}_c)^\intercal \Sigma_c^{-1}(\pmb{x}-\pmb{\mu}_c) + \text{const} \end{align*} \] is called the discriminant function. The decision boundary between any two classes is quadratic in \(\pmb{x}\). This is known as a quadratic discriminant analysis (QDA).

Linear decision boundaries

Start from log posterior of a Gaussian discriminant analysis model \[ \begin{align*} \log p(y = c | \pmb x, \pmb \theta) =& \log \pi_c - \frac 12 \log \vert 2\pi {\bf \Sigma}_c \vert \\ &- \frac 12 (\pmb x - \pmb \mu_c)^\intercal {\bf \Sigma}_c^{-1} (\pmb x - \pmb \mu_c) \end{align*} \] Now assume that the covariance matrices are tied or shared across classes, \({\bf \Sigma}_c = {\bf \Sigma}\).¹ \[ \begin{align*} \log p(y = c | \pmb x, \pmb \theta) =& \underbrace{\log \pi_c - \frac 12 \pmb \mu_c^\intercal{\bf \Sigma}^{-1} \pmb \mu_c}_{\gamma_c} + \pmb x^\intercal \underbrace{{\bf \Sigma}^{-1} \pmb \mu_c}_{\beta_c} \\ & + \underbrace{\text{const} - \frac 12 \pmb x^\intercal {\bf \Sigma}^{-1} \pmb x}_\kappa \\ =& \gamma_c + \pmb x^\intercal \beta_c - \kappa \end{align*} \] Importantly, the \(\kappa\) term is independent of \(c\), and thus an irrelevant additive constant that can be dropped. So the discriminant function is a linear function of \(\pmb x\). Hence it is an LDA.²

Naive Bayes

Use a class conditional density of the form \[ p(\pmb{x} | y=c, \pmb{\theta}) = \prod_{d=1}^D p(x_d | y=c, \pmb{\theta}_{dc}) \] where \(\pmb{\theta}_{dc}\) are the parameters for the class conditional density for class \(c\) and feature \(d\). The posterior over class labels yields \[ p(y=c| \pmb{x}, \pmb{\theta}) = \frac {p(y = c| \pmb \pi) \prod_{d=1}^D p(x_d | y = c, \pmb \theta_{dc})} {\sum_{c'} p(y = c' | \pmb \pi) \prod_{d=1}^D p(x_d | y = c', \pmb \theta_{dc})} \] where \(\pi_c\) is the prior probability of class \(c\) and \(\pmb{\theta} = (\pi, \{ \pmb{\theta}_{dc}\})\) are all the parameters. This is a naive Bayes classifier.