• Step A: Hidden layers learn new features from the raw input — like splines or polynomial features, but discovered from data rather than pre-specified

• Step B: The output layer classifies or predicts using those learned features — same as logistic regression or linear regression, but on top of the new representation

• The full pipeline: \(\mathbf{x} \to\) hidden layers \(\to\) learned features \(\to\) output

• Nonlinearity is essential: without it, any number of layers collapses to a single linear transformation (the collapse theorem)

• Training = minimize loss via SGD + backprop — the loss surface is non-convex, but SGD navigates it effectively

Today: We formalize all of this — the math behind representation learning, backpropagation, and the practical machinery that makes training work.

The core mechanism is autograd:

• requires_grad=True tells PyTorch to track every operation on a tensor
• .backward() traverses the computational graph backward to compute \(\partial y / \partial x\)
• The gradient is stored in x.grad

Example: For \(f(x) = x^2 + 3x + 1\) at \(x = 2\):

• y = x**2 + 3*x + 1 builds the computational graph
• y.backward() computes \(f'(x) = 2x + 3\), so \(f'(2) = 7\)
• x.grad returns 7.0

This is the same chain rule we will formalize in Block 5 — PyTorch just does it automatically.

ESL’s deeper insight: Neural networks are a form of parametric projection pursuit regression (PPR).

PPR models:

\[f(\mathbf{x}) = \sum_{m=1}^{M} g_m(\boldsymbol{\omega}_m^T \mathbf{x})\]

where each \(g_m\) is estimated nonparametrically (smoothing splines, etc.).

Neural networks fix \(g_m = \sigma\) (a known parametric function):

\[f(\mathbf{x}) = \sum_{m=1}^{M} \beta_m \, \sigma(\boldsymbol{\alpha}_m^T \mathbf{x})\]

Why this matters: PPR can use only 5–10 terms (nonparametric \(g_m\) is expensive to estimate). Neural networks, with the cheap parametric \(\sigma\), can use 20–100+ terms — vastly more representational power.

Layer \(\ell\) (hidden): \(\quad \mathbf{z}^{(\ell)} = \sigma\!\left(\mathbf{W}^{(\ell)}\mathbf{z}^{(\ell-1)} + \mathbf{b}^{(\ell)}\right), \quad \ell = 1, \ldots, L-1\)

Output layer: \(\quad f(\mathbf{x}) = g\!\left(\mathbf{W}^{(L)}\mathbf{z}^{(L-1)} + \mathbf{b}^{(L)}\right)\), where \(g\) = identity (regression), sigmoid (binary), or softmax (multi-class)

Composition: \(\quad f = f_L \circ f_{L-1} \circ \cdots \circ f_1\)

Each layer learns progressively more abstract features — this is hierarchical feature learning:

• Layer 1: edges, local patterns
• Layer 2: combinations of edges — textures, shapes
• Layer 3: parts of objects
• Deeper layers: full concepts

This hierarchical composition is the fundamental reason deep networks outperform shallow ones.

Recall from 9A: 60,000 training images but 235,146 parameters — 4x overparameterized. How this works is a central question for Block 6.

• Wide (single layer, \(M\) units): \(O(M)\) linear regions

• Deep (\(L\) layers, \(M\) units each): up to \(O(M^L)\) linear regions — exponential in depth! (Montufar et al. 2014)

Same number of neurons, vastly different expressive power.

• Sigmoid: \(\sigma(z) = \frac{1}{1+e^{-z}}\) — maps to \((0, 1)\). Historical default; saturates at extremes.
• Tanh: \(\tanh(z) = 2\sigma(2z) - 1\) — maps to \((-1, 1)\). Zero-centered, but still saturates.
• ReLU: \(\max(0, z)\) — simple, no saturation for \(z > 0\). The modern default.
• Leaky ReLU: \(\max(0.01z, z)\) — fixes “dying ReLU” with a small negative slope.
• GELU: \(z \cdot \Phi(z)\) where \(\Phi\) is the standard normal CDF — smooth, non-monotonic gate used in Transformers (GPT, BERT).

Loss:

\[R(\boldsymbol{\theta}) = \sum_{i=1}^{n} \bigl(y_i - f_{\boldsymbol{\theta}}(\mathbf{x}_i)\bigr)^2\]

Statistical interpretation: This assumes \(Y | \mathbf{X} \sim N\!\left(f(\mathbf{x}),\, \sigma^2\right)\).

Minimizing MSE is equivalent to maximizing the Gaussian log-likelihood:

\[\ell(\boldsymbol{\theta}) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - f_{\boldsymbol{\theta}}(\mathbf{x}_i))^2\]

The second term is just \(-\frac{1}{2\sigma^2} R(\boldsymbol{\theta})\), so \(\arg\min R = \arg\max \ell\).

Multi-class classification (\(K > 2\)):

• Output: softmax — \(f_k(\mathbf{x}) = \dfrac{e^{T_k}}{\sum_{j=1}^K e^{T_j}}\), ensuring \(\sum_k f_k = 1\)

• Loss: \(R(\boldsymbol{\theta}) = -\sum_{i=1}^{n} \sum_{k=1}^{K} y_{ik} \log f_k(\mathbf{x}_i)\)

Week 6 callback: This is EXACTLY the multinomial logistic regression criterion — but applied to the learned hidden features \(\mathbf{z}\) instead of the raw inputs \(\mathbf{x}\).

Neural networks do not abandon statistics — they embrace it at the output layer while learning features at the hidden layers.

Naive approach: perturb each parameter, use finite differences:

\[\frac{\partial R}{\partial \theta_j} \approx \frac{R(\theta_j + \epsilon) - R(\theta_j)}{\epsilon}\]

Cost: \(O(P)\) forward passes for \(P\) parameters — with \(P = 235{,}146\) (MNIST), this is impractical.

Backpropagation: Compute ALL gradients in a single backward pass — cost \(\approx O(P)\), roughly the same as one forward pass.

Backpropagation is not a separate optimization algorithm — it IS the chain rule, applied systematically to the computational graph.

Forward pass for a single observation \((\mathbf{x}_i, y_i)\):

Step 1 — Pre-activations:

\[v_1 = \alpha_{01} + \alpha_{11}x_1 + \alpha_{21}x_2, \qquad v_2 = \alpha_{02} + \alpha_{12}x_1 + \alpha_{22}x_2\]

Step 2 — Hidden activations:

\[Z_1 = \sigma(v_1), \qquad Z_2 = \sigma(v_2)\]

Step 3 — Output:

\[f = \beta_0 + \beta_1 Z_1 + \beta_2 Z_2\]

Step 4 — Loss (for one observation):

\[R_i = \tfrac{1}{2}(y_i - f)^2\]

To find \(\frac{\partial R}{\partial \alpha_{11}}\), the chain rule gives:

\[\frac{\partial R}{\partial \alpha_{11}} = \underbrace{\frac{\partial R}{\partial f}}_{\text{output error}} \cdot \underbrace{\frac{\partial f}{\partial Z_1}}_{\text{output weight}} \cdot \underbrace{\frac{\partial Z_1}{\partial v_1}}_{\sigma'(v_1)} \cdot \underbrace{\frac{\partial v_1}{\partial \alpha_{11}}}_{x_1}\]

Each factor corresponds to one edge in the computational graph.

Many of these factors are shared across different parameter gradients. Backpropagation exploits this sharing.

Output error signal (gradient of loss w.r.t. prediction):

\[\delta_i \;=\; \frac{\partial R_i}{\partial f} \;=\; -(y_i - f)\]

Gradients w.r.t. output-layer weights (ESL eq 11.14, ISLR eq 10.29):

\[\frac{\partial R_i}{\partial \beta_m} = \delta_i \cdot Z_{mi}, \qquad \frac{\partial R_i}{\partial \beta_0} = \delta_i\]

These are simple: each gradient is the error signal times the corresponding hidden activation.

For multi-output networks (\(K\) outputs): \(\delta_{ki} = -(y_{ik} - f_k(\mathbf{x}_i))\) for regression (identity output). For classification with softmax + cross-entropy, the gradient w.r.t. the logit \(T_k\) takes the same clean form: \(\delta_{ki} = -(y_{ik} - p_k(\mathbf{x}_i))\), thanks to a special cancellation between softmax and cross-entropy.

Hidden error signal (ESL eq 11.15):

\[s_{mi} = \sigma'(v_{mi}) \sum_{k=1}^{K} \beta_{km}\, \delta_{ki}\]

Gradients w.r.t. input-layer weights (ESL eq 11.14, ISLR eq 10.30):

\[\frac{\partial R_i}{\partial \alpha_{jm}} = s_{mi} \cdot x_{ji}\]

The key: hidden-layer errors \(s_{mi}\) are computed by back-propagating output errors \(\delta_{ki}\) through the weights \(\beta_{km}\) and the activation derivative \(\sigma'(v_{mi})\).

This is where the name “back-propagation” comes from — errors flow backward through the network, from output to input.

Efficiency: Each gradient is computed exactly once. Total cost \(\approx\) 2x the cost of a single forward pass.

Interactive Backprop Walkthrough — step through a 2→2→1 network with concrete numbers, forward and backward

\(\eta\) = learning rate — the single most important hyperparameter in deep learning.

• Too large: diverges (overshoots minima)
• Too small: converges too slowly, gets trapped in sharp local minima

Problem: Computing \(\nabla R\) requires summing over all \(n\) observations — for ImageNet (\(n = 1.2\) million), one gradient step = one full pass through the data.

Properties:

• Unbiased: \(\mathbb{E}_{\mathcal{B}}\!\left[\frac{1}{|\mathcal{B}|}\sum_{i \in \mathcal{B}} \nabla R_i\right] = \nabla R\) (in expectation, equals the full gradient)
• Typical batch sizes: 32, 64, 128, 256
• Epoch = one complete pass through the training set (\(n / |\mathcal{B}|\) updates)

Full-batch GD takes clean, deterministic steps toward the nearest minimum. SGD adds noise from minibatch sampling — the path is jittery but still converges. The noise is not a bug; it is a feature.

• Sharp minima generalize poorly; flat minima generalize well. Sharp minima are sensitive to small perturbations in the data — high test error. Flat minima are robust.

• SGD noise bounces the parameters out of sharp minima. The random fluctuations from minibatch sampling make it hard to stay in narrow, sharp basins.

• Noise magnitude scales with \(\eta\) and inversely with \(|\mathcal{B}|\). Larger learning rate or smaller batch = more noise = stronger preference for flat minima.

• This is implicit regularization — SGD naturally prefers flat minima without an explicit penalty term.

ISLR: “SGD naturally enforces its own form of approximately quadratic regularization.”

Empirically: solutions found by SGD generalize better than those found by full-batch GD, even when both reach the same training loss.

\(R(\boldsymbol{\theta})\) is highly non-convex — many local minima, saddle points, and plateaus.

• Saddle points dominate in high dimensions. In a \(P\)-dimensional parameter space, a critical point requires all \(P\) eigenvalues of the Hessian to be positive for a local minimum. Saddle points (mixed signs) are exponentially more common.

• Local minima cluster near the global minimum. Choromanska et al. (2015) showed that loss surfaces of large networks resemble spin glass models — exponentially many local minima, but they concentrate near the same loss value.

• SGD noise helps escape saddle points. At a saddle point the gradient is zero, but minibatch noise pushes parameters off the saddle along directions of negative curvature.

• The practical implication: Finding ANY local minimum is usually good enough. The obsession with the global minimum is a low-dimensional intuition that does not transfer to overparameterized networks.

In backpropagation, each layer multiplies by \(\sigma'(v)\):

\[\boldsymbol{\delta}^{(\ell)} = \underbrace{\sigma'(\mathbf{v}^{(\ell)})}_{\leq 0.25} \odot \left[(\mathbf{W}^{(\ell+1)})^T \boldsymbol{\delta}^{(\ell+1)}\right]\]

For a network with \(L\) layers, the gradient reaching layer 1 is multiplied by \(\sigma'\) at each of \(L - 1\) hidden layers:

\[\left|\frac{\partial R}{\partial \mathbf{W}^{(1)}}\right| \sim (0.25)^{L-1} \cdot \left|\frac{\partial R}{\partial \mathbf{W}^{(L)}}\right|\]

For \(L = 10\): \((0.25)^9 \approx 3.8 \times 10^{-6}\) — the gradient has effectively vanished.

ReLU alleviates this: \(\text{ReLU}'(z) = 1\) for \(z > 0\) — no multiplicative shrinkage. Gradients flow through ReLU layers without attenuation.

Sigmoid (left): gradient norms decay exponentially from the output layer toward the input — early layers stop learning.

ReLU (right): gradient norms remain roughly constant across layers — all layers learn effectively.

Random near zero (ESL): typical starting values \(\sim U[-0.7, 0.7]\) with standardized inputs.

ESL: “Starting with large weights typically leads to poor solutions.”

ESL: “With small weights, the network starts out approximately linear, and becomes nonlinear as the weights increase.”

But HOW small? Random draws from \(N(0, 0.01)\) work for shallow networks, but for deep networks the variance compounds across layers — we need a principled approach.

For one layer: \(z = \sum_{i=1}^{n_{\text{in}}} w_i x_i\). If \(w_i\) and \(x_i\) are independent with \(\mathbb{E}[w] = 0\) and \(\mathbb{E}[x] = 0\) (this is why we standardize inputs):

\[\text{Var}(z) = n_{\text{in}} \cdot \text{Var}(w) \cdot \text{Var}(x)\]

Setting \(\text{Var}(z) = \text{Var}(x)\) (preserve activation variance across layers):

\[\text{Var}(w) = \frac{1}{n_{\text{in}}}\]

The factor of 2 in He initialization accounts for ReLU zeroing out approximately half the activations.

Principle: Choose weights so variance of activations AND gradients is approximately constant across layers — prevents both vanishing and exploding.

• ESL: “analogous to ridge regression”
• The gradient of the penalty is \(2\lambda\theta_j\); in practice the factor of 2 is often absorbed into \(\lambda\)
• Controls model complexity by shrinking weights toward zero

The penalty applies to all weights in the network — input-to-hidden, hidden-to-hidden, and hidden-to-output. Bias terms are typically excluded (just as in ridge regression, we do not penalize the intercept).

Typical values: \(\lambda \sim 10^{-4}\) to \(10^{-2}\), tuned via validation loss.

• Stop training when validation loss begins to increase — the simplest and most effective regularizer.

• ESL: “Starting from weights near zero (approximately linear), early stopping shrinks the final model toward a linear model.”

• Controls complexity through the number of gradient steps, not an explicit penalty. More steps = more nonlinearity = more capacity.

Strategy: Train \(K\) networks with different random seeds, then average their predictions:

\[\hat{f}_{\text{avg}}(\mathbf{x}) = \frac{1}{K}\sum_{k=1}^K \hat{f}_k(\mathbf{x})\]