Reference topics: Stanford CS231n · independent study notes, not affiliated

Deep Learning for Vision

CNNs, RNNs, attention, transformers, and modern computer vision techniques.

5Lectures
1Assignments
1Exams
42hDuration
1

Introduction to Computer Vision and CNNs

What is Computer Vision? Making computers understand images and video. Tasks: Classification: "this is a cat" Detection: "there's a cat here" (bounding box) Segmentation:...

⏱ 50 min read

What is Computer Vision?

Making computers understand images and video. Tasks:

  • Classification: "this is a cat"
  • Detection: "there's a cat here" (bounding box)
  • Segmentation: pixel-level labels
  • Pose estimation: skeleton of a person
  • Depth estimation: 3D structure
  • Optical flow: motion between frames

CV has exploded with deep learning since 2012.

Why Vision is Hard

An image is a 3D array (H × W × 3). For a 224×224 RGB image: 150,528 numbers.

Naive fully-connected layers would have ~150,000 inputs × thousands of hidden units = billions of parameters.

Images have structure:

  • Locality: nearby pixels are related
  • Translation invariance: a cat is a cat regardless of position
  • Hierarchical features: edges → textures → parts → objects

CNNs exploit this structure.

Convolutional Neural Networks

A CNN uses convolutional layers that slide small filters across the input.

For a 3×3 filter on a H × W × C input:

  • Filter has 3×3×C weights
  • Slides across all positions
  • Produces a H' × W' × 1 feature map (one per filter)

Each filter detects a specific pattern (edge, corner, texture). Multiple filters → multiple feature maps.

Properties of Convolution

Local connectivity: each output depends on a small input region.

Weight sharing: same filter across all positions.

Translation equivariance: shift input → shift output.

Parameter count: 3×3×C per filter. Much smaller than dense.

Pooling

Downsample feature maps to reduce spatial size and add translation invariance.

Max pooling: take max in each 2×2 region.
Average pooling: take average.

Pooling has no learnable parameters. Reduces H and W by factor 2 (for 2×2).

CNN Architecture

Typical pattern:

[Conv → ReLU] × N → Pool → [Conv → ReLU] × M → Pool → ... → Flatten → FC → Softmax

Features get smaller spatially, deeper in channels. Final layers are fully connected.

LeNet (1998)

Yann LeCun's classic for handwritten digit recognition.

  • 2 conv layers
  • 2 pooling layers
  • 3 fully connected
  • ~60,000 parameters

Trained on MNIST. Used in ATMs for check reading.

AlexNet (2012)

The breakthrough. Won ImageNet 2012 by a huge margin.

  • 5 conv layers, 3 FC layers
  • 60M parameters
  • ReLU instead of sigmoid
  • Dropout for regularization
  • Data augmentation
  • Trained on 2 GPUs (GTX 580, 3GB each)

Started the deep learning revolution.

VGG (2014)

Very deep networks using only 3×3 convolutions. VGG-16 has 16 layers.

Simple, uniform architecture. Popular for transfer learning.

Downside: 138M parameters, slow.

ResNet (2015)

Deep residual networks. Introduced skip connections:

y = F(x) + x

Where F is a few conv layers. The +x is the identity shortcut.

Benefits:

  • Trains very deep networks (100-1000 layers)
  • Solves vanishing gradient
  • Won ImageNet 2015

ResNet-50 is a workhorse for vision tasks.

Modern Architectures

  • Inception (GoogLeNet): multi-scale filters
  • DenseNet: dense connections between layers
  • EfficientNet: scaled depth, width, resolution
  • MobileNet: for mobile, depthwise separable convs
  • ViT: vision transformer, patches as tokens
  • ConvNeXt: modernized CNN, competitive with ViT

Training CNNs

  • Loss: cross-entropy for classification
  • Optimizer: SGD with momentum, Adam, AdamW
  • Regularization: dropout, weight decay, data augmentation
  • Learning rate: step decay, cosine, warmup
  • Batch norm: normalize activations per batch, speeds training

Transfer Learning

Pretrain on ImageNet, fine-tune on your task.

  • Freeze early layers (generic features)
  • Fine-tune later layers (task-specific)
  • Works with small datasets (100s of examples)

Standard practice for most CV tasks.

Key Takeaways

  • Images are 3D arrays; CNNs exploit locality
  • Convolution: local, weight-shared, translation-equivariant
  • Pooling: downsample, add invariance
  • AlexNet (2012) started the revolution
  • VGG, ResNet, Inception, EfficientNet, ViT
  • Transfer learning standard
  • Modern: transformers competing with CNNs
2

Training Neural Networks

The Training Loop For each batch: 1. Forward pass: compute predictions 2. Compute loss 3. Backward pass: compute gradients 4. Update weights Repeat until convergence. Loss Functions Classificati...

⏱ 50 min read

The Training Loop

For each batch:

  1. Forward pass: compute predictions
  2. Compute loss
  3. Backward pass: compute gradients
  4. Update weights

Repeat until convergence.

Loss Functions

Classification: cross-entropy.

L = -Σ y_i log(p_i)

Regression: MSE.

L = (1/N) Σ (y_i - ŷ_i)²

Object detection: combined classification + localization.
Segmentation: per-pixel cross-entropy or Dice.

Gradient Descent Variants

Batch GD: use all data per step. Accurate but slow.

Stochastic GD (SGD): one example per step. Noisy but fast.

Mini-batch GD: 32-256 examples per step. Standard.

Momentum: add velocity term.

v = β v + ∇L
θ = θ - α v

Helps escape local minima, faster convergence.

Adam: adaptive per-parameter learning rates. Combines momentum + RMSprop.

m = β₁ m + (1-β₁) ∇L
v = β₂ v + (1-β₂) (∇L)²
θ = θ - α m / (√v + ε)

AdamW decouples weight decay from gradient update. Standard for transformers.

Backpropagation

The chain rule applied to computational graphs. For each layer, compute gradient w.r.t. inputs and weights.

For a linear layer y = Wx:

  • dL/dW = dL/dy × x^T
  • dL/dx = W^T × dL/dy

Automatic differentiation (autograd) handles this in PyTorch, JAX, TensorFlow.

Vanishing and Exploding Gradients

Deep networks suffer from unstable gradients:

  • Vanishing: gradients shrink through layers, early layers barely update.
  • Exploding: gradients blow up, training diverges.

Solutions:

  • ReLU activation: no saturation in positive range.
  • Batch normalization: normalizes each layer's inputs.
  • Residual connections: skip connections provide gradient highway.
  • Careful initialization: He, Xavier.

Activation Functions

  • Sigmoid: σ(x) = 1/(1+e^-x). Saturates; rarely used in hidden layers.
  • Tanh: range [-1, 1]. Still saturates.
  • ReLU: max(0, x). Simple, fast, standard.
  • Leaky ReLU: small slope for x<0.
  • ELU, SELU: smooth variants.
  • GELU: used in BERT, GPT.
  • SiLU/Swish: used in EfficientNet, modern nets.

ReLU is default unless there's a reason.

Weight Initialization

Bad initialization → dead networks.

Xavier/Glorot: for tanh/sigmoid. Var = 2/(fan_in + fan_out).

He: for ReLU. Var = 2/fan_in.

Too small → activations vanish. Too large → explode.

Batch Normalization

Normalize each mini-batch:

x̂ = (x - mean) / √(var + ε)
y = γ x̂ + β

γ, β are learned. Standard in CNNs.

Benefits: faster training, allows higher LR, less sensitive to init.

Layer norm: normalize across features. Used in transformers.

Group norm, instance norm: alternatives for small batch sizes.

Regularization

Dropout: randomly zero activations during training. Forces redundancy.

Weight decay (L2): penalize large weights.

Data augmentation: random crops, flips, color jitter. Cheap and effective.

Early stopping: stop when validation loss increases.

Label smoothing: soft targets prevent overconfidence.

Learning Rate Schedules

  • Step decay: divide LR by 10 every N epochs.
  • Cosine annealing: LR follows cosine curve.
  • Warmup: start small, increase for a few epochs, then decay.
  • One-cycle: fast increase then decrease.

Critical for good convergence.

Hyperparameter Tuning

Key hyperparameters:

  • Learning rate (most important)
  • Batch size
  • Weight decay
  • Dropout rate
  • Number of layers, hidden size
  • Optimizer choice

Approach: random search > grid search. Bayesian optimization for expensive models.

Debugging Training

Loss not decreasing: LR too low, wrong loss, bug in forward.
Loss NaN: LR too high, numerical instability.
Overfitting: regularization, more data, smaller model.
Underfitting: bigger model, train longer.

Sanity checks: overfit a small batch. Verify loss on trivial data.

Key Takeaways

  • Training loop: forward, loss, backward, update
  • Many optimizers; Adam/AdamW standard
  • Backprop via chain rule
  • Vanishing/exploding gradients solved by ReLU, batchnorm, residuals
  • ReLU standard activation
  • He init for ReLU
  • Batch norm speeds training
  • Regularization: dropout, weight decay, augmentation
  • Learning rate schedule matters
  • Random search for hyperparameters
3

Object Detection and Segmentation

From Classification to Detection Classification: &quot;this image contains a cat.&quot; Detection: &quot;there&#39;s a cat at (x, y, w, h).&quot; Detection is harder: multiple objects, varying sizes,...

⏱ 50 min read

From Classification to Detection

Classification: "this image contains a cat."
Detection: "there's a cat at (x, y, w, h)."

Detection is harder: multiple objects, varying sizes, occlusions.

Two-Stage Detectors

R-CNN (2014): region proposals → CNN per region → classify. Slow.

Fast R-CNN (2015): single CNN pass, ROI pooling. Faster.

Faster R-CNN (2015): add Region Proposal Network (RPN). End-to-end.

Still two-stage, still relatively slow (~5-10 FPS).

Used when accuracy matters more than speed.

One-Stage Detectors

YOLO (2016): divide image into grid, predict boxes + classes per cell. Fast.

SSD (2016): multi-scale feature maps for different object sizes.

RetinaNet (2017): focal loss to handle class imbalance.

YOLOv5/v7/v8: modern versions, fast and accurate.

EfficientDet: scaled, efficient.

One-stage is now standard for real-time detection.

Anchor Boxes

Pre-defined boxes of various sizes and aspect ratios at each position.

Each detection predicts:

  • Offset from nearest anchor
  • Class probabilities
  • Objectness score

Recent models (FCOS, CenterNet) are anchor-free.

Non-Maximum Suppression (NMS)

Many overlapping boxes for the same object. NMS:

  1. Sort boxes by score
  2. Keep highest
  3. Remove boxes with IoU > threshold
  4. Repeat

IoU (Intersection over Union) measures overlap.

Evaluation Metrics

IoU: overlap between predicted and ground truth boxes.

mAP (mean Average Precision): average precision across classes and IoU thresholds.

COCO mAP: averages over IoU 0.5 to 0.95, small/medium/large objects.

FPS: frames per second (speed).

Trade-off: accuracy vs speed.

Semantic Segmentation

Classify every pixel.

FCN (2014): replace FC layers with conv layers. Output is a heatmap.

U-Net (2015): encoder-decoder with skip connections. Standard for medical imaging.

DeepLab: atrous convolutions for multi-scale context.

SegFormer, Mask2Former: transformer-based, state-of-the-art.

Loss: per-pixel cross-entropy, Dice, or focal.

Instance Segmentation

Separate objects of the same class.

Mask R-CNN (2017): Faster R-CNN + mask branch. Standard.

YOLACT: real-time instance segmentation.

SOLOv2, CondInst: alternatives.

Used in robotics, autonomous driving.

Panoptic Segmentation

Combine semantic (all pixels) + instance (individual objects).

Each pixel is labeled with a class AND an instance ID (for countable things).

Used in autonomous driving scene understanding.

Keypoint Detection

Find specific points on objects (joints, facial landmarks).

OpenPose: bottom-up, detects all keypoints, groups into people.

HRNet: high-resolution network, accurate.

MediaPipe: Google's, real-time on mobile.

3D Vision

Beyond 2D images:

  • Depth estimation: monocular or stereo
  • 3D object detection: LiDAR + camera
  • NeRF: neural radiance fields for view synthesis
  • Gaussian Splatting: fast 3D scene rendering
  • SLAM: simultaneous localization and mapping

Video Understanding

Beyond single images:

  • Action recognition: classify actions in video
  • Temporal detection: when does action happen
  • Video captioning: describe video
  • Tracking: follow objects across frames

Models: 3D CNNs, two-stream networks, transformers (TimeSformer, VideoMAE).

Key Takeaways

  • Detection: locate + classify objects
  • Two-stage (Faster R-CNN) vs one-stage (YOLO)
  • Anchors, NMS, IoU
  • mAP is the standard metric
  • Segmentation: semantic, instance, panoptic
  • U-Net for medical, Mask R-CNN for general
  • Keypoint detection: OpenPose, MediaPipe
  • 3D vision: depth, NeRF, Gaussian Splatting
  • Video: 3D CNNs, transformers
4

Generative Models: GANs, VAEs, Diffusion

What are Generative Models? Models that learn the distribution of data and generate new samples. Tasks: Image generation (DALL-E, Stable Diffusion, Midjourney) Image editing (inpainting, style...

⏱ 50 min read

What are Generative Models?

Models that learn the distribution of data and generate new samples.

Tasks:

  • Image generation (DALL-E, Stable Diffusion, Midjourney)
  • Image editing (inpainting, style transfer)
  • Super-resolution
  • Data augmentation
  • Anomaly detection

Taxonomy

  • Explicit density: model P(x) directly (VAE, autoregressive)
  • Implicit density: learn to sample without P(x) (GAN)
  • Score-based / diffusion: learn the gradient of log P(x)

Each has strengths.

Autoencoders

Compress input to a latent code, reconstruct:

x → Encoder → z → Decoder → x̂

Trained to minimize ||x - x̂||. Bottleneck forces compact representation.

Used for dimensionality reduction, denoising, feature learning.

But autoencoders are not generative — no way to sample new data.

Variational Autoencoders (VAEs)

Encoder outputs a distribution (μ, σ), not a point. Sample z from it, decode.

Loss = reconstruction loss + KL divergence to prior.

L = ||x - x̂||² + KL(q(z|x) || p(z))

Benefits:

  • Smooth latent space
  • Can sample new data
  • Principled probabilistic framework

Downside: blurry samples (from MSE loss).

GANs (Generative Adversarial Networks)

Goodfellow et al. (2014). Two networks:

  • Generator: produces fake images from noise
  • Discriminator: distinguishes real from fake

Trained adversarially:

min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]

Generator tries to fool discriminator; discriminator tries not to be fooled.

Result: generator produces realistic images.

GAN Variants

  • DCGAN: convolutional GAN, stable architecture
  • CycleGAN: unpaired image translation (horse ↔ zebra)
  • StyleGAN: high-quality faces, controllable style
  • Pix2Pix: paired image translation (edges → photos)
  • BigGAN: class-conditional, high fidelity
  • Progressive GAN: grow resolution during training

GAN Challenges

  • Mode collapse: generator produces limited variety
  • Training instability: hard to balance G and D
  • Evaluation: hard to measure quality (FID, IS)

Diffusion Models

Sohl-Dickstein et al. (2015), popularized by Ho et al. (2020).

Forward process: gradually add noise to image.

x_t = √(1-β_t) x_{t-1} + √β_t ε

After T steps, x_T is pure noise.

Reverse process: learn to denoise step by step.

x_{t-1} = (1/√α_t) (x_t - (1-α_t)/√(1-ᾱ_t) ε_θ(x_t, t))

Train network to predict noise ε at each step.

Why Diffusion Works

  • Stable training (simple MSE loss)
  • High-quality samples
  • Covers the data distribution (no mode collapse)
  • Controllable generation

Downside: slow sampling (many steps). Solved by DDIM, DPM-Solver.

Stable Diffusion

Latent diffusion: apply diffusion in a compressed latent space (VAE encoded).

Benefits: much faster, lower compute.

Text conditioning via CLIP text encoder → cross-attention.

Powers DALL-E 2, Stable Diffusion, Midjourney, Imagen.

Classifier-Free Guidance

Improve sample quality by combining conditional and unconditional predictions:

ε = ε_uncond + w (ε_cond - ε_uncond)

w is the guidance scale. Higher w → more fidelity to prompt, less diversity.

ControlNet

Add spatial conditioning (pose, edges, depth) to diffusion models.

Enables precise control over generation.

Evaluation Metrics

  • FID (Fréchet Inception Distance): distance between real and generated distributions in feature space. Lower is better.
  • IS (Inception Score): quality + diversity. Higher is better.
  • CLIP Score: text-image alignment.

No metric is perfect; human eval still matters.

Applications

  • Text-to-image (DALL-E 3, SDXL, Midjourney)
  • Text-to-video (Sora, Runway)
  • Image editing (inpainting, outpainting)
  • 3D generation (DreamFusion)
  • Drug discovery (molecule generation)
  • Audio generation (MusicGen, AudioLM)

Key Takeaways

  • Generative models learn P(x)
  • Autoencoders compress; VAEs sample
  • GANs: adversarial, sharp but unstable
  • Diffusion: iterative denoising, high quality, stable
  • Stable Diffusion: latent diffusion + text conditioning
  • Guidance, ControlNet for control
  • FID is standard metric
  • Diffusions dominate image generation
5

Advanced Topics and Deployment

Self-Supervised Learning Learn representations without labels. Use pretext tasks: Contrastive: pull similar pairs together, push dissimilar apart Masked prediction: predict masked parts Rotatio...

⏱ 50 min read

Self-Supervised Learning

Learn representations without labels. Use pretext tasks:

  • Contrastive: pull similar pairs together, push dissimilar apart
  • Masked prediction: predict masked parts
  • Rotation: predict how image was rotated

Methods: SimCLR, MoCo, BYOL, DINO, MAE.

Why it matters: unlabeled data is abundant; labels are expensive.

Foundation models (CLIP, DINOv2) trained self-supervised.

Vision Transformers (ViT)

Apply transformer to images: split into patches, treat as tokens.

Image (224×224) → 16×16 patches → 196 tokens → Transformer

Requires more data than CNNs. On ImageNet alone, ViT underperforms ResNet. On larger datasets (JFT-300M), it surpasses.

Modern: Swin, DeiT, BEiT, DINOv2.

Multimodal Models

Combine vision + language:

  • CLIP: contrastive image-text pretraining
  • BLIP: captioning, VQA
  • Flamingo: few-shot visual understanding
  • GPT-4V, Gemini: native multimodal LLMs
  • LLaVA: open-source vision-language

Enables image captioning, VQA, image-grounded dialogue.

Neural Architecture Search (NAS)

Automate architecture design.

  • Reinforcement learning: controller proposes architectures
  • Evolutionary: mutate and select
  • Differentiable (DARTS): relax discrete choices

Found architectures: NASNet, AmoebaNet, EfficientNet.

Expensive to run but produces efficient models.

Model Compression

Deploy large models on small devices.

Pruning: remove weights/neurons. Sparse models run faster.

Quantization: reduce precision.

  • Post-training: quantize trained model (INT8)
  • Quantization-aware training: simulate quantization during training
  • 4-bit, 2-bit for LLMs

Knowledge distillation: train small student to mimic large teacher.

Low-rank factorization: replace layers with smaller matrices.

Combined: 10-100x smaller models with minor quality loss.

Efficient Architectures

  • MobileNet: depthwise separable convolutions
  • ShuffleNet: channel shuffle for efficiency
  • EfficientNet: compound scaling
  • MobileViT: hybrid CNN + transformer

Used in mobile, embedded, edge.

Hardware for Deep Learning

  • GPUs: NVIDIA dominates (A100, H100, RTX 4090)
  • TPUs: Google's tensor processing units
  • NPUs: dedicated neural network accelerators (Apple Neural Engine, Qualcomm Hexagon)
  • FPGAs: reconfigurable, flexible
  • ASICs: custom silicon (Google TPU, Tesla FSD)
  • Neuromorphic: brain-inspired (Intel Loihi)

Each has trade-offs in speed, power, cost, flexibility.

Frameworks

  • PyTorch: research standard, dynamic graphs
  • TensorFlow: production, Keras API
  • JAX: functional, autograd, XLA
  • MXNet: Amazon, legacy
  • ONNX: interchange format
  • TVM: compiler for deployment

PyTorch dominates research; TF/JAX in production.

Deployment

From trained model to production:

  1. Export to ONNX or TorchScript
  2. Optimize (quantization, pruning)
  3. Convert to target (TensorRT, CoreML, TFLite)
  4. Serve (REST API, gRPC)
  5. Monitor (latency, accuracy, drift)

Tools: TorchServe, Triton, KServe, BentoML.

Ethics and Fairness

Models can be biased, unfair, or harmful.

Issues:

  • Bias: racial, gender, age in predictions
  • Privacy: models memorize training data
  • Deepfakes: realistic fake media
  • Misuse: surveillance, manipulation
  • Environmental: energy cost of training

Mitigation:

  • Bias testing and correction
  • Differential privacy
  • Watermarking generated content
  • Regulation (EU AI Act)

Interpretability

Understanding why models predict what they do.

  • Saliency maps: which pixels matter
  • Grad-CAM: class activation maps
  • Attention visualization: what the model attends to
  • LIME, SHAP: local explanations

Important for high-stakes applications (medical, legal, autonomous).

Future Directions

  • Foundation models: one model for many tasks
  • Multimodal: text, image, audio, video
  • Efficient: smaller, faster, cheaper
  • Robust: handle distribution shift, adversarial attacks
  • Aligned: safe, honest, helpful
  • Reasoning: chain-of-thought, planning
  • Agents: LLMs that take actions

The field moves fast. Stay curious.

Key Takeaways

  • Self-supervised learning uses unlabeled data
  • ViT brings transformers to vision
  • Multimodal models combine vision + language
  • NAS automates architecture design
  • Compression: pruning, quantization, distillation
  • Efficient architectures for edge
  • GPUs/TPUs/NPUs/FPGAs for training and inference
  • PyTorch, TF, JAX frameworks
  • Deployment: export, optimize, serve, monitor
  • Ethics, interpretability, and safety are critical

📝 Problem Set 1: CNNs, Training, Backprop

Q1. A conv layer has input 32×32×3, 64 filters of size 3×3, stride 1, no padding. What is the output size and parameter count?

Show Solution

Output: (32 - 3 + 0)/1 + 1 = 30, so 30×30×64. Params per filter: 3×3×3 + 1 = 28. Total: 64 × 28 = 1792.

Q2. Compute the output of a 2×2 max-pool on the matrix [[1,3,2,4],[5,6,7,8],[9,1,2,3],[4,5,6,7]] with stride 2.

Show Solution

Top-left 2×2: max(1,3,5,6)=6. Top-right: max(2,4,7,8)=8. Bottom-left: max(9,1,4,5)=9. Bottom-right: max(2,3,6,7)=7. Output: [[6,8],[9,7]].

Q3. Explain why batch normalization helps training and where the learned parameters γ and β come in.

Show Solution

BatchNorm normalizes activations to zero mean and unit variance per mini-batch, then scales by γ and shifts by β (both learned). This stabilizes activations, allows higher LR, reduces sensitivity to init, and acts as mild regularization. γ/β let the network undo normalization if needed.

🎯 Midterm 1: CNNs, Training, Detection

Q1. Compute gradients for a 2-layer network: h = ReLU(W1 x), y = softmax(W2 h), loss = -log(y_true). Show dL/dW2 and dL/dW1.

Show Solution

dL/dz2 = y - y_true (softmax + CE gradient). dL/dW2 = (y - y_true) h^T. dL/dh = W2^T (y - y_true). dL/dz1 = dL/dh ⊙ (z1 > 0). dL/dW1 = dL/dz1 × x^T.

Q2. Explain the difference between semantic, instance, and panoptic segmentation. Give a use case for each.

Show Solution

Semantic: each pixel classified by class (road, car, person) — no distinction between instances. Used in scene understanding. Instance: each object instance separate (car 1, car 2). Used in counting, robotics. Panoptic: combines both — every pixel labeled with class + instance. Used in autonomous driving.

Q3. A YOLO model outputs 3 boxes at a location: [(0.1, 0.4), (0.2, 0.3), (0.15, 0.35)] with confidences (0.9, 0.8, 0.7). Apply NMS with IoU threshold 0.5.

Show Solution

Sort by confidence: (0.1,0.4)@0.9, (0.2,0.3)@0.8, (0.15,0.35)@0.7. Box 1 kept. Compute IoU with box 2: high overlap → suppress. IoU with box 3: high overlap → suppress. Final: only box 1 remains.