Introduction: Linear regression¹

Linear regression

Model of the following form \[ p(y | {\bf x}, {\bf \theta}) = \mathcal{N}(y | w_0 + {\bf w}^\intercal {\bf x}, \sigma^2) \] where \({\bf \theta} = (w_0, {\bf w}, \sigma^2)\) are the model parameters.²

Ordinary least squares¹

Weighted least squares¹

Sometimes the variance may depend on the input.² In this case we may want to associate a specific weight to each example \[ \begin{align*} p(y | {\bf x}, {\bf \theta}) &= \mathcal{N}(y | {\bf w}^\intercal {\bf x}, \sigma^2({\bf x})) \\ &= \frac 1{\sqrt{2\pi\sigma^2({\bf x})}} \exp\left(-\frac 1{2\sigma^2({\bf x})}(y - {\bf w}^\intercal {\bf x})^2\right) \\ &= \mathcal{N}(y | {\bf Xw}, {\bf \Lambda}^{-1}) \end{align*} \] where \({\bf \Lambda} = \text{diag}(1/\sigma^2({\bf x}_n))\).

The maximum likelihood estimate yields the weighted least-squares estimate \[ \boxed{ \hat{\bf w} = ({\bf X^\intercal \Lambda X})^{-1} {\bf X^\intercal \Lambda y} } \]

Residual and parity plots¹

Residual plot

Residuals \(r_n = y_n - \hat y_n\) vs the input \(x_n\). Should follow \(\mathcal{N}(0, \sigma^2)\).

Parity plot

Predictions \(\hat y_n\) vs the true output \(y_n\). Points should be on the diagonal.

Prediction accuracy and \(R^2\)¹

Quantify accuracy

Compute the residual sum of squares on the dataset (the smaller the better) \[ \text{RSS}({\bf w}) = \sum_{n=1}^N (y_n - {\bf w^\intercal x}_n)^2 \]

Root mean-squared error (RMSE)

RMSE is a more common measure to quantify accuracy \[ \text{RMSE}({\bf w}) = \sqrt{\frac 1N \text{RSS}({\bf w})} = \sqrt{\frac 1N \sum_{n=1}^N (y_n - {\bf w^\intercal x}_n)^2} \]

Coefficient of determination (\(R^2\))

More interpretable measure \[ R^2 = 1 - \frac{\sum_{n=1}^N (\hat y_n - y_n)^2}{\sum_{n=1}^N (\bar y_n - y_n)^2} = 1 - \frac{\text{RSS}}{\text{TSS}} \] where \(\bar y = \frac 1N \sum_{n=1}^N y_n\) is the empirical mean, RSS is the residual sum of squares, and TSS is the total sum of squares. \(R^2\) measures the variance in the predictions relative to a simple constant prediction, \(\bar y\).

Robust linear regression: Multiple strategies

Laplace likelihood

\[ p(y | {\bf x}, {\bf w}, b) = \text{Laplace}(y | {\bf w}^\intercal{\bf x}, b) \propto \exp\left( -\frac 1b |y - {\bf w}^\intercal{\bf x}|\right) \] MLE can be computed using linear programming.¹

Student-\(t\) likelihood

\[ p(y | {\bf x}, {\bf w}, \sigma^2, \nu) = \left[ 1 + \frac 1\nu \left(\frac{y-\mu}\sigma \right)^2 \right]^{-\frac{\nu+1}2} \] Model can be fit using stochastic gradient descent.

Huber loss

Mix of \(\ell_2\) and \(\ell_1\) regularizations at short and long ranges, respectively.²

Bayesian linear regression

Recall multivariate normal likelihood (iid) with known variance \[ p(\mathcal{D} | {\bf w}, \sigma^2) = \prod_{n=1}^N p(y_n | {\bf w}^\intercal{\bf x}, \sigma^2) = \mathcal{N}({\bf y} | {\bf Xw}, \sigma^2 {\bf I}_N) \] Compute the posterior over the parameters, \(p({\bf w} | \mathcal{D}, \sigma^2)\). Assume further a Gaussian prior \[ p({\bf w}) = \mathcal{N}({\bf w} | {\bf \breve{w}}, {\bf \breve{\Sigma}}) \] Use Bayes’ rule \[ \begin{align*} p({\bf w} | {\bf X}, {\bf y}, \sigma^2) &\propto \mathcal{N}({\bf y} | {\bf Xw}, \sigma^2 {\bf I}_N) \mathcal{N}({\bf w} | {\bf \breve{w}}, {\bf \breve{\Sigma}}) = \mathcal{N}({\bf w} | \tilde{\bf w}, \tilde{\bf \Sigma}) \\ \tilde{\bf w} &= \tilde{\bf \Sigma} ({\bf \breve{\Sigma}}^{-1}{\bf \breve{w}} + \frac 1{\sigma^2} {\bf X}^\intercal{\bf y}) \\ \tilde{\bf \Sigma} &= \left({\bf \breve{\Sigma}}^{-1} + \frac 1{\sigma^2} {\bf X}^\intercal{\bf X}\right)^{-1} \end{align*} \] where \(\tilde{\bf w}\) and \(\tilde{\bf \Sigma}\) are the posterior mean and covariance, respectively.¹

Introduction¹

Linear regression

Recall \(p(y | {\bf x}, {\bf w}) = \mathcal{N}(y | {\bf w^\intercal x}, \sigma^2)\). Useful property: the mean of the output, \(E[y | {\bf x}, {\bf w}]\), is a linear function of the inputs \({\bf x}\).

Moments of GLMs

For GLMs, \(T(y) = y\). One can then show \[ \begin{align*} E[y_n | {\bf x}_n, {\bf w}, \sigma^2] &= A'(\eta_n) = \ell^{-1}(\eta_n) \\ \text{Var}[y_n | {\bf x}_n, {\bf w}, \sigma^2] &= A''(\eta_n) \sigma^2 \end{align*} \] where \(\ell^{-1}\) is called the mean function.¹

Example: linear regression

where \(\eta_n = {\bf w}^\intercal {\bf x}_n\), can be rewritten in GLM form \[ \log p(y_n | {\bf x}_n, {\bf w}, \sigma^2) = \frac{y_n \eta_n - \frac{\eta_n^2}{2}}{\sigma^2} - \frac 12\left(\frac{y_n^2}{\sigma^2} + \log(2\pi\sigma^2) \right) \] We see that \(A(\eta_n) = \eta_n^2/2\) and thus \[ \begin{align*} E[y_n | {\bf x}_n, {\bf w}, \sigma^2] &= \eta_n = {\bf w^\intercal x}_n \\ \text{Var}[y_n | {\bf x}_n, {\bf w}, \sigma^2] &= \sigma^2 \end{align*} \]

Example: binomial regression

Maximum likelihood estimation¹

Gradient

\[ \begin{align*} \frac{\partial}{\partial {\bf w}} \text{NLL}({\bf w}) &= - \frac 1{\sigma^2} \sum_{n=1}^N \frac{\partial}{\partial \eta_n} (\eta_n y_n - A(\eta_n)) \frac{\partial \eta_n}{\partial {\bf w}} \\ &= -\frac 1{\sigma^2} \sum_{n=1}^N (y_n - A'(\eta_n)) {\bf x}_n \end{align*} \]

Hessian

\[ {\bf H} = \frac{\partial}{\partial{\bf w}\partial{\bf w}^\intercal}\text{NLL}({\bf w}) = \frac 1{\sigma^2}\sum_{n=1}^N A''(\eta_n){\bf x}_n {\bf x}_n^\intercal \] is positive definite, so the MLE for a GLM is unique!

Strategy

Fit GLMs using gradient-based solvers to reach the global minimum.