Literature

• Chapter 11 from Murphy (2022): Basis expansion
• Chapter 13 from Murphy (2022): Neural networks

Recap from last time

Generalization

Dangers of overfitting and underfitting are always present

Maximum-a-posteriori (MAP) estimate

Regularizes the negative log-likelihood by the log of the prior

Cross-validation

Common technique to choose regularization parameter

Regularization in linear regression

Ridge (\(\ell_2\)) and Lasso (\(\ell_0\) or \(\ell_1\)) constrain the strength and the number of weights, respectively

Example: Polynomial fitting¹

\[ f(x; {\bf w}) = \sum_{d=0}^D w_d x^d = {\bf w}^\intercal[1, x, x^2, \dots, x^D] \]

Regression splines¹

• Using polynomial basis functions creates a nonlinear mapping from input to output, even though the model remains linear in the parameters.
• Problem with polynomials: they form a global approximation to the function.
• More flexibility with a series of local approximations (i.e., local support)

Consider the 1D case with a set of basis functions: \[ f(x; {\theta}) = \sum_{i=1}^m w_i B_i(x) \]

B-spline basis functions¹

B-spline basis functions: Properties¹

• Piecewise polynomial of degree \(D\), where the locations of the pieces are defined by knots \(t_1<\dots<t_m\).
• Function is continuous and has continuous derivatives of orders \(1, \dots, D-1\)
• Common to use cubic splines, i.e., \(D=3\): will have continuous first and second derivatives.

Fitting with splines¹

Example: classification of 2d data into 2 categories

https://playground.tensorflow.org/

The importance of depth

Universal function approximator

Possible to show that an MLP with one hidden layer is a universal function approximator, i.e., it can model any suitably smooth function, given enough hidden units, to any desired level of accuracy.

Intuition

Each hidden unit can specify a half plane. With enough of these you can “carve up” any region of space.

In practice

Deep networks do perform better than shallow ones. They can leverage the features that are learned by earlier layers. See XOR example.

Connections with biology

Simple example in Pytorch

import torch
import torch.nn as nn
import torch.optim as optim

# Ensure reproducibility
torch.manual_seed(0)

# 1. Generate a toy dataset
# Let's generate some random data with shape (100, 1)
inputs = torch.rand(100, 1)
# Let's use a simple linear relation with added noise for targets
targets = 2.5 * inputs + 3 + torch.randn(100, 1) * 0.1

# 2. Define the neural network
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(1, 10)  # First fully-connected layer
        self.fc2 = nn.Linear(10, 1)  # Second fully-connected layer
        self.relu = nn.ReLU()
        
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)  # Activation function
        return self.fc2(x)

# Create an instance of the network
model = SimpleNN()

# 3. Define the loss and optimizer
criterion = nn.MSELoss()  # Mean squared error
optimizer = optim.SGD(model.parameters(), lr=0.01)

# 4. Train the network
num_epochs = 500
for epoch in range(num_epochs):
    # Forward pass
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    
    # Zero gradients, backward pass, optimizer step
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    # Print the loss every 50 epochs
    if (epoch + 1) % 100 == 0:
        print(f"Epoch [{epoch + 1}/{num_epochs}], Loss: {loss.item():.4f}")

Epoch [100/500], Loss: 0.0366
Epoch [200/500], Loss: 0.0167
Epoch [300/500], Loss: 0.0105
Epoch [400/500], Loss: 0.0086
Epoch [500/500], Loss: 0.0080

Summary

Regression splines

provide a local approximation to the function. Used together with a linear combination of parameters, so as to retain convex optimization

B-spline basis functions

Offer a degree \(D\) of continuous derivatives. Cubic (\(D=3\)) often used.

Neural networks

composition of linear transformations connected by differentiable activation functions

Differentiability

key property to ensure efficient learning. Enabled/optimized by means of automatic differentiation (chain rule).