Week 9A: Neural Networks — The Intuition

Stat 577 — Statistical Learning Theory

Soumojit Das

Spring 2026

Act 1: Why Do We Need Something New?

The Story So Far

All semester we have been estimating \(f\) in:

\[Y = f(\mathbf{X}) + \varepsilon\]

Weeks 2–3 (Linear Models): We assumed \(f\) is linear:

\[\hat{f}(\mathbf{x}) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p\]

One hyperplane. Valid everywhere. Interpretable. But rigid.

Weeks 7–8 (Trees): We let the data decide, via recursive partitions:

\[\hat{f}(\mathbf{x}) = \sum_j c_j \, \mathbf{1}(\mathbf{x} \in R_j)\]

Flexible. Handles interactions. But discontinuous, axis-aligned, high variance.

Today: A third paradigm that sits between these two.

The Limitation of Linear Models

Suppose we want to classify these two groups:

Logistic regression draws a straight line (hyperplane). It cannot solve this.

The Old Fix: Feature Engineering

Your move in Weeks 2–3: Manually create new features.

Maybe add \(x_1^2\), \(x_1 x_2\), \(x_2^2\), or spline basis functions.

\[\hat{f}(\mathbf{x}) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1^2 + \beta_4 x_1 x_2 + \beta_5 x_2^2\]

The model is still linear in the new features, but the decision boundary in the original space is now curved.

This works. But you have to know in advance which transformations to use.

For 2 features, you can guess. For 784 features (pixels in an image)? You cannot possibly hand-engineer the right polynomial and interaction terms.

The Fundamental Question

What if we didn’t hand-pick the new features?

What if we added a learnable preprocessing step that creates new features from the raw inputs, and then did logistic regression on those?

The Neural Network Idea in One Sentence

The Core Idea

A neural network is a learnable feature extractor glued onto a classifier you already know.

That’s it. Everything else — depth, activation functions, backpropagation, SGD, initialization — is machinery for making this idea work at scale.

Before we write any equations, let’s learn the vocabulary.

Act 2: The Vocabulary

Glossary, Part 1: The Building Blocks

Neuron (or “unit” or “node”):

A single computational element that does two things: (1) compute a weighted sum of its inputs plus a bias, then (2) apply a nonlinear function to the result.

Weight (\(w\)):

A learnable number that multiplies an input. Analogous to \(\beta_j\) in linear regression. Controls how much a particular input matters to this neuron.

Bias (\(b\)):

The intercept term. Without bias, the neuron’s “decision boundary” must pass through the origin. Exactly analogous to \(\beta_0\) in regression.

Pre-activation (\(v\)):

The weighted sum before applying the nonlinear function: \(v = w_1 x_1 + w_2 x_2 + \cdots + b\).

Glossary, Part 2: The Nonlinearity

Activation function (\(\sigma\)):

The nonlinear function applied to the pre-activation. Without it, a neural network collapses to a linear model (we will prove this). Common choices:

Name Formula Range When to use
Sigmoid \(\sigma(v) = 1/(1 + e^{-v})\) \((0, 1)\) Output layer for binary classification
ReLU \(\sigma(v) = \max(0, v)\) \([0, \infty)\) Hidden layers (modern default)
Activation value (\(Z\)):

The output of a neuron after applying the activation function: \(Z = \sigma(v)\).

Ambiguity Warning

The word “activation” can mean the function \(\sigma\) or the output value \(Z\). This is a common source of confusion. Context will usually make it clear which one is meant.

Glossary, Part 3: The Architecture

Layer:

A group of neurons that receive input from the same source and send output to the same destination. Computationally: one matrix multiplication + one bias addition + one activation function.

Input layer:

Not a real “layer” — just the raw data \(\mathbf{x}\). No computation happens here.

Hidden layer:

Any layer between input and output. “Hidden” because its values are not directly observed — they are learned internal representations.

Output layer:

The final layer. Its design depends on the task: sigmoid for binary classification, softmax for multi-class, identity (no activation) for regression.

Architecture:

The structural choices: how many layers, how many neurons per layer, which activation functions. Chosen by you before training. Analogous to choosing “degree-3 polynomial” vs “cubic spline with 5 knots.”

Glossary, Part 4: The Training Process

Parameters (\(\boldsymbol{\theta}\)):

The complete set of all learnable weights and biases. For a network with \(L\) layers, this is \(\boldsymbol{\theta} = \{\) all weights, all biases \(\}\).

Loss function (\(R\)):

Measures how wrong the network’s prediction is. Squared error for regression, cross-entropy for classification. Same loss functions as Weeks 2–6.

Epoch:

One complete pass through all \(n\) training observations. Typically requires many epochs (dozens to hundreds) to train a network.

Depth vs Width:

Depth = number of layers. Width = number of neurons per layer. These are independent architectural choices.

Act 3: Building a Neural Network, Step by Step

Starting Point: Logistic Regression (What You Know)

You have 2 features and a binary response. Logistic regression:

\[\hat{p}(Y = 1 \mid \mathbf{x}) = \sigma(\beta_0 + \beta_1 x_1 + \beta_2 x_2)\]

Feed in raw features → weighted sum → sigmoid → probability.

You learned the \(\beta\)’s by maximizing the log-likelihood. This is your entire model.

The limitation: the decision boundary is a straight line. Classes that aren’t linearly separable cannot be classified correctly.

Step A: Make New Features

Instead of classifying directly from \(x_1, x_2\), let’s first create new features:

\[Z_1 = \sigma(\underbrace{a_{10} + a_{11}x_1 + a_{12}x_2}_{\text{weighted sum of raw inputs}})\]

\[Z_2 = \sigma(a_{20} + a_{21}x_1 + a_{22}x_2)\]

\[Z_3 = \sigma(a_{30} + a_{31}x_1 + a_{32}x_2)\]

Each \(Z_m\) is a new feature. It is a sigmoid-shaped function of the original inputs. The \(a\)’s control:

  • Which direction in input space this feature responds to (the \(a_{m1}, a_{m2}\) values)
  • Where it “turns on” (the \(a_{m0}\) bias)

We have created 3 new features from 2 original ones.

Step B: Classify Using the New Features

Now do logistic regression, but on \(Z_1, Z_2, Z_3\) instead of \(x_1, x_2\):

\[\hat{p}(Y = 1 \mid \mathbf{x}) = \sigma(\beta_0 + \beta_1 Z_1 + \beta_2 Z_2 + \beta_3 Z_3)\]

That’s It. That’s a Neural Network.

  • Step A = the hidden layer (3 neurons)
  • Step B = the output layer (logistic regression on learned features)
  • The \(a\)’s = hidden layer weights
  • The \(\beta\)’s = output layer weights
  • All of them are learned together from data

What Gets Learned?

This is the key point, so let’s be very explicit.

In Weeks 2–3 (basis expansion regression):

  1. You choose the features: \(Z_1 = x_1^2\), \(Z_2 = x_1 x_2\), etc. (fixed formulas)
  2. Then estimate only \(\boldsymbol{\beta}\) by least squares

In a neural network:

  1. The features \(Z_m = \sigma(a_{m0} + a_{m1}x_1 + a_{m2}x_2)\) are parameterized by the \(a\)’s
  2. The \(a\)’s and \(\beta\)’s are estimated simultaneously from data

So the network is figuring out two things at once:

  • What features to create (by adjusting the \(a\)’s)
  • How to combine those features (by adjusting the \(\beta\)’s)

This joint learning is the power. It is also why training is harder — the \(a\)’s enter through the nonlinear \(\sigma\), making the loss function non-convex.

A Single Neuron IS Logistic Regression

Let’s make this concrete. A single neuron with sigmoid activation:

\[f(\mathbf{x}) = \sigma(w_1 x_1 + w_2 x_2 + b)\]

With binary cross-entropy loss, this is identical to logistic regression.

  • \(w_1, w_2\) are the regression coefficients \(\beta_1, \beta_2\)
  • \(b\) is the intercept \(\beta_0\)
  • The loss is the same negative log-likelihood from Week 6

Not “similar to” — literally the same optimization problem.

Key Insight

You have been training one-neuron networks since Week 6. The only new idea today is: what happens when you stack many neurons together?

Act 4: Going Deeper — Two Hidden Layers

Adding a Second Round of Feature-Making

One hidden layer created features from raw inputs. What if we create features of features?

Step A1 — First-round features (from raw inputs):

\[Z_1 = \sigma(a_{10} + a_{11}x_1 + a_{12}x_2)\] \[Z_2 = \sigma(a_{20} + a_{21}x_1 + a_{22}x_2)\] \[Z_3 = \sigma(a_{30} + a_{31}x_1 + a_{32}x_2)\]

Step A2 — Second-round features (from first-round features):

\[Z'_1 = \sigma(c_{10} + c_{11}Z_1 + c_{12}Z_2 + c_{13}Z_3)\] \[Z'_2 = \sigma(c_{20} + c_{21}Z_1 + c_{22}Z_2 + c_{23}Z_3)\]

Notice: the inputs to Step A2 are not \(x_1, x_2\). They are \(Z_1, Z_2, Z_3\). So \(Z'_1\) is a nonlinear function of things that are already nonlinear functions of the raw inputs.

Step B: Classify on Second-Round Features

\[\hat{p}(Y = 1 \mid \mathbf{x}) = \sigma(\beta_0 + \beta_1 Z'_1 + \beta_2 Z'_2)\]

The Full Architecture

\[\underbrace{x_1, x_2}_{\text{raw input}} \;\xrightarrow{\text{Step A1}}\; \underbrace{Z_1, Z_2, Z_3}_{\text{first-round features}} \;\xrightarrow{\text{Step A2}}\; \underbrace{Z'_1, Z'_2}_{\text{second-round features}} \;\xrightarrow{\text{Step B}}\; \hat{p}\]

What gets learned? ALL of it, simultaneously:

Symbol What it controls Count
\(a\)’s How to build first-round features from raw inputs 9
\(c\)’s How to build second-round features from first-round features 8
\(\beta\)’s How to classify using second-round features 3
Total learnable parameters 20

Why Bother with Two Rounds?

The first-round features \(Z_1, Z_2, Z_3\) can each detect one “pattern” — a sigmoid bump in some direction.

But some problems need features that are combinations of patterns:

  • “\(Z_1\) is high AND \(Z_3\) is low”
  • “\(Z_2\) is medium but \(Z_1\) is not high”

No single first-round feature can represent these conjunctions. The second round can.

For vision: Layer 1 detects edges. Layer 2 detects combinations of edges (corners, textures). The output layer classifies based on those.

Each additional layer builds increasingly abstract features on top of the previous layer’s features. This compositional structure is why deep networks are qualitatively different from wide ones.

Do We Always Shrink the Feature Size?

No. The tapering pattern (784 → 256 → 128 → 10) is common but not required.

You can expand (create more features than you started with):

\[\underbrace{x_1, x_2}_{2} \;\xrightarrow{\text{Step A1}}\; \underbrace{Z_1, \ldots, Z_5}_{5} \;\xrightarrow{\text{Step A2}}\; \underbrace{Z'_1, \ldots, Z'_7}_{7} \;\xrightarrow{\text{Step B}}\; \hat{p}\]

Why? The raw 2D space may be too “cramped.” Expanding to higher dimensions makes separation easier (same intuition as the kernel trick in SVMs).

You can go wider then narrower (“diamond”): \(784 \to 1024 \to 512 \to 10\)

You can go narrower then wider (“bottleneck”): \(784 \to 32 \to 512 \to 10\)

The architecture is a modeling decision you make, analogous to choosing the number of knots in a spline.

Act 5: The Full Prediction Pipeline

End-to-End: From Input to Prediction

For \(K\)-class classification with \(L\) hidden layers, the complete pipeline:

Input: \(\;\mathbf{z}^{(0)} = \mathbf{x} \in \mathbb{R}^p\)

Hidden layer \(\ell\) (for \(\ell = 1, \ldots, L\)):

\[\mathbf{z}^{(\ell)} = \sigma\!\left(\mathbf{W}^{(\ell)} \mathbf{z}^{(\ell-1)} + \mathbf{b}^{(\ell)}\right)\]

Each \(\mathbf{W}^{(\ell)}\) is \(M_\ell \times M_{\ell-1}\), mapping from previous layer’s features to this layer’s features.

Logits: \(\;\mathbf{T} = \mathbf{B}\,\mathbf{z}^{(L)} + \mathbf{b}_0, \qquad \mathbf{B}\) is \(K \times M_L\)

Probabilities: \(\;\hat{p}_k = \dfrac{e^{T_k}}{\sum_{j=1}^K e^{T_j}}\) (softmax)

Prediction: \(\;\hat{y} = \arg\max_k\, \hat{p}_k\)

The Pipeline in One Picture

Interactive Pipeline Diagram — click through each stage from \(\mathbf{x}\) to \(\hat{y}\)

Mapping the Intuitive Notation to the Formal Notation

Intuitive (Step A/B) Formal (matrix) What it is
The \(a\)’s (first hidden layer) \(\mathbf{W}^{(1)}, \mathbf{b}^{(1)}\) Weights for building first-round features
The \(c\)’s (second hidden layer) \(\mathbf{W}^{(2)}, \mathbf{b}^{(2)}\) Weights for building second-round features
The \(\beta\)’s (output layer) \(\mathbf{B}, \mathbf{b}_0\) Weights for classifying on final features
\(Z_1, Z_2, Z_3\) \(\mathbf{z}^{(1)}\) First-round learned features
\(Z'_1, Z'_2\) \(\mathbf{z}^{(2)}\) Second-round learned features

The matrix form is just the element-wise equations stacked into matrices for computational efficiency. The math is the same.

Counting Parameters: MNIST Example

MNIST: 784 pixel inputs → 256 hidden → 128 hidden → 10 digit classes

Layer Step Matrix size Parameters
Hidden 1 Step A1: raw inputs → first features \(\mathbf{W}^{(1)}\): \(256 \times 784\), plus 256 biases \(200{,}960\)
Hidden 2 Step A2: first features → second features \(\mathbf{W}^{(2)}\): \(128 \times 256\), plus 128 biases \(32{,}896\)
Output Step B: second features → class scores \(\mathbf{B}\): \(10 \times 128\), plus 10 biases \(1{,}290\)
Total \(235{,}146\)

More Parameters Than Data

MNIST has 60,000 training images but 235,146 parameters — 4× more parameters than observations. How can this possibly generalize? (Answer: regularization + implicit bias of SGD. We’ll get there.)

Act 6: How Training Works

The Training Problem

Before training, all the \(a\)’s, \(c\)’s, and \(\beta\)’s are random numbers. The network outputs garbage.

Training = finding values of ALL parameters that minimize a loss function.

For classification, the loss is cross-entropy (= negative log-likelihood):

\[R(\boldsymbol{\theta}) = -\sum_{i=1}^n \sum_{k=1}^K y_{ik} \log \hat{p}_k(\mathbf{x}_i)\]

This is the same MLE criterion from Week 6. The only difference: \(\hat{p}_k\) is now produced by a neural network instead of a linear model.

The Training Loop

The Complete Training Algorithm

Once, before training:

Initialize all weights to small random values (He or Xavier initialization — covered in Week 9B)

Repeat for many epochs:

For each minibatch of training observations:

  1. Forward pass: run each \(\mathbf{x}_i\) through the pipeline → get \(\hat{\mathbf{p}}_i\)
  2. Compute loss: \(R = -\sum y_{ik} \log \hat{p}_k\) (how wrong are we?)
  3. Backward pass (backprop): compute \(\partial R / \partial \theta\) for EVERY parameter (chain rule)
  4. Update: \(\theta \leftarrow \theta - \eta \cdot \frac{\partial R}{\partial \theta}\) (nudge every parameter to reduce loss)

Check validation loss → stop if it starts rising (early stopping)

The forward pass is the prediction pipeline from Act 5. The backward pass runs it in reverse to compute gradients. SGD uses those gradients to update parameters.

Why Non-Convex?

In logistic regression, the loss is a convex function of \(\boldsymbol{\beta}\). There is one global minimum. Gradient descent always finds it.

In a neural network, the \(a\)’s enter through the nonlinear \(\sigma\):

\[R(\boldsymbol{\theta}) = -\sum_i \left[y_i \log \sigma\!\left(\beta_0 + \beta_1 \sigma(a_{10} + a_{11}x_{1i} + a_{12}x_{2i}) + \cdots\right) + \cdots\right]\]

This is not convex in the \(a\)’s. The loss landscape has:

  • Multiple local minima
  • Saddle points (flat directions)
  • Plateaus

This is the price we pay for learning features instead of hand-picking them. If we fixed the \(a\)’s and only optimized \(\beta\)’s, we’d have convex logistic regression on known features. Joint optimization gives us power but takes away convexity.

Where Does Each Piece of Machinery Plug In?

Machinery Where it acts What it does
Initialization (He/Xavier) All weights, once before training Sets starting values so signals don’t explode or vanish
Forward pass The pipeline, every minibatch Produces predictions
Loss function End of pipeline, every minibatch Scores how wrong we are
Backpropagation Reverse pipeline, every minibatch Computes all gradients
SGD All parameters, every minibatch Updates parameters using gradients
Weight decay SGD update, every minibatch Shrinks weights toward zero (= ridge)
Early stopping Between epochs Halts training when validation loss rises

Act 7: Why Nonlinearity is Essential

The Collapse Theorem

Without a nonlinear activation function \(\sigma\), stacking layers does nothing:

\[f(\mathbf{x}) = \mathbf{W}^{(2)}\bigl(\mathbf{W}^{(1)}\mathbf{x}\bigr) = \underbrace{\bigl(\mathbf{W}^{(2)}\mathbf{W}^{(1)}\bigr)}_{\text{just another matrix}}\mathbf{x}\]

The composition of linear functions is still linear. A 100-layer network with no activations is equivalent to a single linear layer.

Proof: Linear maps are closed under composition. \(f(\mathbf{x}) = \mathbf{Ax}\) composed with \(g(\mathbf{x}) = \mathbf{Bx}\) gives \(g(f(\mathbf{x})) = \mathbf{BAx} = \mathbf{Cx}\), where \(\mathbf{C} = \mathbf{BA}\). No matter how many layers, you get one matrix.

Takeaway

The activation function \(\sigma\) is what gives neural networks their power. The weight matrices provide degrees of freedom. The nonlinearity provides expressive power. Remove either and the model is trivial.

Looking Ahead

What This Lecture Set Up

We now understand:

✓ What a neural network is: a learnable feature extractor + a GLM

✓ Why we need it: manual feature engineering does not scale

✓ What gets learned: all weights simultaneously, via SGD + backprop

✓ The full pipeline: \(\mathbf{x} \to \mathbf{z}^{(1)} \to \cdots \to \mathbf{T} \to \hat{\mathbf{p}} \to \hat{y}\)

After spring break, the main deck (Week 9B) covers:

  • Activation functions: the full zoo (sigmoid, tanh, ReLU, GELU, and why it matters)
  • Backpropagation: the chain rule formalized, computed step by step
  • The vanishing gradient problem and how ReLU solves it
  • Initialization theory: why He and Xavier work (a 60-second derivation)
  • Regularization: weight decay, early stopping, multiple random starts
  • Depth vs width: why \(O(M^L)\) linear regions beats \(O(M)\)