Student Presentations — Weeks 13 (starting 04/13) & 14 (starting 04/20)

STAT 577 — Statistical Learning Theory

Author

Soumojit Das

Published

April 5, 2026

Overview

Each of you will give one 50-minute presentation on an assigned topic in deep learning. You own the full class session: 40–45 minutes of presentation, followed by 5–10 minutes of class discussion and Q&A.

Your job is to teach the class your topic. Present the key ideas, explain the methods, discuss the results, and offer your own critique. You are the expert for your topic — go deep, not wide.

Each topic comes with 2–3 assigned readings. Present all the core readings as a coherent narrative — not as isolated paper summaries, but as a story: what was the problem, what did each paper contribute, and what’s your takeaway?

Note

I will cover the full transformer architecture in detail during Week 15. For topics that touch on attention or transformers (Topics 4–6), focus on the concepts in your assigned papers rather than trying to explain the full transformer architecture from scratch. I will (try to, if time permits) introduce the basic ideas of attention in next week’s lectures to give everyone enough understanding.

Random Assignment

Presentation order was determined by random draw:

set.seed(577)
students <- c("Alex", "Ephraim", "Garrett", "Jackie", "Obrey", "Wooyoung")
order <- sample(students, replace = FALSE)
order
[1] "Obrey"    "Wooyoung" "Alex"     "Jackie"   "Garrett"  "Ephraim"

Topics were assigned in the order they appear below (Topic 1 through Topic 6).

Schedule

Week 13

Day Presenter Topic
Monday Obrey Topic 1: CNNs & the Vision Revolution
Wednesday Wooyoung Topic 2: Residual Networks & the Depth Problem
Friday Alex Topic 3: Double Descent & Overparameterization

Week 14

Day Presenter Topic
Monday Jackie Topic 4: RNNs, LSTMs & the Birth of Attention
Wednesday Garrett Topic 5: Transfer Learning
Friday Ephraim Topic 6: Scaling Laws & Emergent Abilities

Topics & Readings

Topic 1: CNNs & the Vision Revolution (Obrey)

Core readings:

  1. ISLR §10.3 (Convolutional Neural Networks)
  2. Krizhevsky, Sutskever & Hinton (2012). “ImageNet Classification with Deep Convolutional Neural Networks.” NeurIPS. [paper]

Optional background:

  • LeCun, Bottou, Bengio & Haffner (1998). “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE, 86(11), 2278–2324. — This is huge. Treat this as a supplementary reading if you have time. It’s the original LeNet paper that introduced the core ideas of convolution, pooling, and weight sharing.

What to cover: What is convolution and why does weight sharing matter? How did AlexNet change the field? What engineering innovations (ReLU, dropout, GPU training) made it work? Connect back to the fully connected networks we studied in Weeks 9 & 11.

Topic 2: Residual Networks & the Depth Problem (Wooyoung)

Core readings:

  1. He, Zhang, Ren & Sun (2016). “Deep Residual Learning for Image Recognition.” CVPR. [paper]
  2. He, Zhang, Ren & Sun (2016). “Identity Mappings in Deep Residual Networks.” ECCV. [paper]

What to cover: Why do deeper plain networks perform worse (the degradation problem)? How do skip connections fix this? The gradient flow argument. The difference between the original and pre-activation ResNet designs. Why this was the breakthrough that made 100+ layer networks possible.

Topic 3: Double Descent & Overparameterization (Alex)

Core readings:

  1. Belkin, Hsu, Ma & Mandal (2019). “Reconciling Modern Machine Learning Practice and the Classical Bias–Variance Trade-Off.” PNAS. [paper]
  2. Zhang, Bengio, Hardt, Recht & Vinyals (2017). “Understanding Deep Learning Requires Rethinking Generalization.” ICLR. [paper]

What to cover: The classical bias-variance U-curve vs. the modern double descent curve. What is the interpolation threshold and why does test error improve beyond it? Zhang et al.’s shocking result that deep networks can memorize random labels — and what that means for our understanding of generalization. Connect back to bias-variance tradeoff discussion.

Topic 4: RNNs, LSTMs & the Birth of Attention (Jackie)

Core readings:

  1. ISLR §10.4–10.5 (Recurrent Neural Networks)
  2. Hochreiter & Schmidhuber (1997). “Long Short-Term Memory.” Neural Computation, 9(8), 1735–1780. [paper]
  3. Bahdanau, Cho & Bengio (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” ICLR. [paper]

What to cover: Why sequences need a different architecture than images or tabular data. The vanishing gradient problem in RNNs (connect to our Week 11 discussion). How LSTM gates solve this. Then the key transition: Bahdanau’s attention mechanism — the idea that the decoder should look back at all encoder states, not just the last one. This paper planted the seed for transformers.

Topic 5: Transfer Learning (Garrett)

Core readings:

  1. Yosinski, Clune, Bengio & Lipson (2014). “How Transferable Are Features in Deep Neural Networks?” NeurIPS. [paper]
  2. Howard & Ruder (2018). “Universal Language Model Fine-tuning for Text Classification (ULMFiT).” ACL. [paper]

What to cover: Yosinski’s key finding — early layers learn general features (edges, textures), later layers learn task-specific features. What this means for practical deep learning: pretrain on a large dataset, fine-tune on your task. Howard & Ruder’s ULMFiT as the idea that launched modern NLP pretraining. The pretrain → fine-tune paradigm and why it works.

Topic 6: Scaling Laws & Emergent Abilities (Ephraim)

Core readings:

  1. Kaplan, McCandlish, Henighan et al. (2020). “Scaling Laws for Neural Language Models.” OpenAI. [paper]
  2. Wei, Tay, Bommasani et al. (2022). “Emergent Abilities of Large Language Models.” Transactions on Machine Learning Research. [paper]

What to cover: Kaplan et al.’s finding that loss scales as a power law with model size, data, and compute — and what that implies for how to allocate a training budget. Wei et al.’s observation that certain abilities (chain-of-thought reasoning, in-context learning) appear suddenly at scale. Are emergent abilities real or a measurement artifact? This is one of the most actively debated questions in ML right now — present both sides.

Presentation Format

  • Duration: 50 minutes total — aim for 40–45 minutes of presentation, 5–10 minutes of Q&A.
  • Slides: You may use any tool (PowerPoint, Quarto, Beamer, Google Slides, etc.).
  • Depth over breadth: It is better to explain one key result deeply than to rush through everything. If a paper has 10 results, pick the 3 most important ones and teach them well.
  • Critique encouraged: Don’t just summarize — tell us what you think. What’s convincing? What’s weak? What would you do differently?
  • Connect to our course: Whenever possible, relate ideas back to our class’s central concepts (bias-variance, regularization, optimization, the GLM connection, etc.).