Deep Learning for Vision
CNNs, RNNs, attention, transformers, and modern computer vision techniques.
Introduction to Computer Vision and CNNs
What is Computer Vision? Making computers understand images and video. Tasks: Classification: "this is a cat" Detection: "there's a cat here" (bounding box) Segmentation:...
What is Computer Vision?
Making computers understand images and video. Tasks:
- Classification: "this is a cat"
- Detection: "there's a cat here" (bounding box)
- Segmentation: pixel-level labels
- Pose estimation: skeleton of a person
- Depth estimation: 3D structure
- Optical flow: motion between frames
CV has exploded with deep learning since 2012.
Why Vision is Hard
An image is a 3D array (H × W × 3). For a 224×224 RGB image: 150,528 numbers.
Naive fully-connected layers would have ~150,000 inputs × thousands of hidden units = billions of parameters.
Images have structure:
- Locality: nearby pixels are related
- Translation invariance: a cat is a cat regardless of position
- Hierarchical features: edges → textures → parts → objects
CNNs exploit this structure.
Convolutional Neural Networks
A CNN uses convolutional layers that slide small filters across the input.
For a 3×3 filter on a H × W × C input:
- Filter has 3×3×C weights
- Slides across all positions
- Produces a H' × W' × 1 feature map (one per filter)
Each filter detects a specific pattern (edge, corner, texture). Multiple filters → multiple feature maps.
Properties of Convolution
Local connectivity: each output depends on a small input region.
Weight sharing: same filter across all positions.
Translation equivariance: shift input → shift output.
Parameter count: 3×3×C per filter. Much smaller than dense.
Pooling
Downsample feature maps to reduce spatial size and add translation invariance.
Max pooling: take max in each 2×2 region.
Average pooling: take average.
Pooling has no learnable parameters. Reduces H and W by factor 2 (for 2×2).
CNN Architecture
Typical pattern:
[Conv → ReLU] × N → Pool → [Conv → ReLU] × M → Pool → ... → Flatten → FC → Softmax
Features get smaller spatially, deeper in channels. Final layers are fully connected.
LeNet (1998)
Yann LeCun's classic for handwritten digit recognition.
- 2 conv layers
- 2 pooling layers
- 3 fully connected
- ~60,000 parameters
Trained on MNIST. Used in ATMs for check reading.
AlexNet (2012)
The breakthrough. Won ImageNet 2012 by a huge margin.
- 5 conv layers, 3 FC layers
- 60M parameters
- ReLU instead of sigmoid
- Dropout for regularization
- Data augmentation
- Trained on 2 GPUs (GTX 580, 3GB each)
Started the deep learning revolution.
VGG (2014)
Very deep networks using only 3×3 convolutions. VGG-16 has 16 layers.
Simple, uniform architecture. Popular for transfer learning.
Downside: 138M parameters, slow.
ResNet (2015)
Deep residual networks. Introduced skip connections:
y = F(x) + x
Where F is a few conv layers. The +x is the identity shortcut.
Benefits:
- Trains very deep networks (100-1000 layers)
- Solves vanishing gradient
- Won ImageNet 2015
ResNet-50 is a workhorse for vision tasks.
Modern Architectures
- Inception (GoogLeNet): multi-scale filters
- DenseNet: dense connections between layers
- EfficientNet: scaled depth, width, resolution
- MobileNet: for mobile, depthwise separable convs
- ViT: vision transformer, patches as tokens
- ConvNeXt: modernized CNN, competitive with ViT
Training CNNs
- Loss: cross-entropy for classification
- Optimizer: SGD with momentum, Adam, AdamW
- Regularization: dropout, weight decay, data augmentation
- Learning rate: step decay, cosine, warmup
- Batch norm: normalize activations per batch, speeds training
Transfer Learning
Pretrain on ImageNet, fine-tune on your task.
- Freeze early layers (generic features)
- Fine-tune later layers (task-specific)
- Works with small datasets (100s of examples)
Standard practice for most CV tasks.
Key Takeaways
- Images are 3D arrays; CNNs exploit locality
- Convolution: local, weight-shared, translation-equivariant
- Pooling: downsample, add invariance
- AlexNet (2012) started the revolution
- VGG, ResNet, Inception, EfficientNet, ViT
- Transfer learning standard
- Modern: transformers competing with CNNs
Training Neural Networks
The Training Loop For each batch: 1. Forward pass: compute predictions 2. Compute loss 3. Backward pass: compute gradients 4. Update weights Repeat until convergence. Loss Functions Classificati...
The Training Loop
For each batch:
- Forward pass: compute predictions
- Compute loss
- Backward pass: compute gradients
- Update weights
Repeat until convergence.
Loss Functions
Classification: cross-entropy.
L = -Σ y_i log(p_i)
Regression: MSE.
L = (1/N) Σ (y_i - ŷ_i)²
Object detection: combined classification + localization.
Segmentation: per-pixel cross-entropy or Dice.
Gradient Descent Variants
Batch GD: use all data per step. Accurate but slow.
Stochastic GD (SGD): one example per step. Noisy but fast.
Mini-batch GD: 32-256 examples per step. Standard.
Momentum: add velocity term.
v = β v + ∇L
θ = θ - α v
Helps escape local minima, faster convergence.
Adam: adaptive per-parameter learning rates. Combines momentum + RMSprop.
m = β₁ m + (1-β₁) ∇L
v = β₂ v + (1-β₂) (∇L)²
θ = θ - α m / (√v + ε)
AdamW decouples weight decay from gradient update. Standard for transformers.
Backpropagation
The chain rule applied to computational graphs. For each layer, compute gradient w.r.t. inputs and weights.
For a linear layer y = Wx:
- dL/dW = dL/dy × x^T
- dL/dx = W^T × dL/dy
Automatic differentiation (autograd) handles this in PyTorch, JAX, TensorFlow.
Vanishing and Exploding Gradients
Deep networks suffer from unstable gradients:
- Vanishing: gradients shrink through layers, early layers barely update.
- Exploding: gradients blow up, training diverges.
Solutions:
- ReLU activation: no saturation in positive range.
- Batch normalization: normalizes each layer's inputs.
- Residual connections: skip connections provide gradient highway.
- Careful initialization: He, Xavier.
Activation Functions
- Sigmoid: σ(x) = 1/(1+e^-x). Saturates; rarely used in hidden layers.
- Tanh: range [-1, 1]. Still saturates.
- ReLU: max(0, x). Simple, fast, standard.
- Leaky ReLU: small slope for x<0.
- ELU, SELU: smooth variants.
- GELU: used in BERT, GPT.
- SiLU/Swish: used in EfficientNet, modern nets.
ReLU is default unless there's a reason.
Weight Initialization
Bad initialization → dead networks.
Xavier/Glorot: for tanh/sigmoid. Var = 2/(fan_in + fan_out).
He: for ReLU. Var = 2/fan_in.
Too small → activations vanish. Too large → explode.
Batch Normalization
Normalize each mini-batch:
x̂ = (x - mean) / √(var + ε)
y = γ x̂ + β
γ, β are learned. Standard in CNNs.
Benefits: faster training, allows higher LR, less sensitive to init.
Layer norm: normalize across features. Used in transformers.
Group norm, instance norm: alternatives for small batch sizes.
Regularization
Dropout: randomly zero activations during training. Forces redundancy.
Weight decay (L2): penalize large weights.
Data augmentation: random crops, flips, color jitter. Cheap and effective.
Early stopping: stop when validation loss increases.
Label smoothing: soft targets prevent overconfidence.
Learning Rate Schedules
- Step decay: divide LR by 10 every N epochs.
- Cosine annealing: LR follows cosine curve.
- Warmup: start small, increase for a few epochs, then decay.
- One-cycle: fast increase then decrease.
Critical for good convergence.
Hyperparameter Tuning
Key hyperparameters:
- Learning rate (most important)
- Batch size
- Weight decay
- Dropout rate
- Number of layers, hidden size
- Optimizer choice
Approach: random search > grid search. Bayesian optimization for expensive models.
Debugging Training
Loss not decreasing: LR too low, wrong loss, bug in forward.
Loss NaN: LR too high, numerical instability.
Overfitting: regularization, more data, smaller model.
Underfitting: bigger model, train longer.
Sanity checks: overfit a small batch. Verify loss on trivial data.
Key Takeaways
- Training loop: forward, loss, backward, update
- Many optimizers; Adam/AdamW standard
- Backprop via chain rule
- Vanishing/exploding gradients solved by ReLU, batchnorm, residuals
- ReLU standard activation
- He init for ReLU
- Batch norm speeds training
- Regularization: dropout, weight decay, augmentation
- Learning rate schedule matters
- Random search for hyperparameters
Object Detection and Segmentation
From Classification to Detection Classification: "this image contains a cat." Detection: "there's a cat at (x, y, w, h)." Detection is harder: multiple objects, varying sizes,...
From Classification to Detection
Classification: "this image contains a cat."
Detection: "there's a cat at (x, y, w, h)."
Detection is harder: multiple objects, varying sizes, occlusions.
Two-Stage Detectors
R-CNN (2014): region proposals → CNN per region → classify. Slow.
Fast R-CNN (2015): single CNN pass, ROI pooling. Faster.
Faster R-CNN (2015): add Region Proposal Network (RPN). End-to-end.
Still two-stage, still relatively slow (~5-10 FPS).
Used when accuracy matters more than speed.
One-Stage Detectors
YOLO (2016): divide image into grid, predict boxes + classes per cell. Fast.
SSD (2016): multi-scale feature maps for different object sizes.
RetinaNet (2017): focal loss to handle class imbalance.
YOLOv5/v7/v8: modern versions, fast and accurate.
EfficientDet: scaled, efficient.
One-stage is now standard for real-time detection.
Anchor Boxes
Pre-defined boxes of various sizes and aspect ratios at each position.
Each detection predicts:
- Offset from nearest anchor
- Class probabilities
- Objectness score
Recent models (FCOS, CenterNet) are anchor-free.
Non-Maximum Suppression (NMS)
Many overlapping boxes for the same object. NMS:
- Sort boxes by score
- Keep highest
- Remove boxes with IoU > threshold
- Repeat
IoU (Intersection over Union) measures overlap.
Evaluation Metrics
IoU: overlap between predicted and ground truth boxes.
mAP (mean Average Precision): average precision across classes and IoU thresholds.
COCO mAP: averages over IoU 0.5 to 0.95, small/medium/large objects.
FPS: frames per second (speed).
Trade-off: accuracy vs speed.
Semantic Segmentation
Classify every pixel.
FCN (2014): replace FC layers with conv layers. Output is a heatmap.
U-Net (2015): encoder-decoder with skip connections. Standard for medical imaging.
DeepLab: atrous convolutions for multi-scale context.
SegFormer, Mask2Former: transformer-based, state-of-the-art.
Loss: per-pixel cross-entropy, Dice, or focal.
Instance Segmentation
Separate objects of the same class.
Mask R-CNN (2017): Faster R-CNN + mask branch. Standard.
YOLACT: real-time instance segmentation.
SOLOv2, CondInst: alternatives.
Used in robotics, autonomous driving.
Panoptic Segmentation
Combine semantic (all pixels) + instance (individual objects).
Each pixel is labeled with a class AND an instance ID (for countable things).
Used in autonomous driving scene understanding.
Keypoint Detection
Find specific points on objects (joints, facial landmarks).
OpenPose: bottom-up, detects all keypoints, groups into people.
HRNet: high-resolution network, accurate.
MediaPipe: Google's, real-time on mobile.
3D Vision
Beyond 2D images:
- Depth estimation: monocular or stereo
- 3D object detection: LiDAR + camera
- NeRF: neural radiance fields for view synthesis
- Gaussian Splatting: fast 3D scene rendering
- SLAM: simultaneous localization and mapping
Video Understanding
Beyond single images:
- Action recognition: classify actions in video
- Temporal detection: when does action happen
- Video captioning: describe video
- Tracking: follow objects across frames
Models: 3D CNNs, two-stream networks, transformers (TimeSformer, VideoMAE).
Key Takeaways
- Detection: locate + classify objects
- Two-stage (Faster R-CNN) vs one-stage (YOLO)
- Anchors, NMS, IoU
- mAP is the standard metric
- Segmentation: semantic, instance, panoptic
- U-Net for medical, Mask R-CNN for general
- Keypoint detection: OpenPose, MediaPipe
- 3D vision: depth, NeRF, Gaussian Splatting
- Video: 3D CNNs, transformers
Generative Models: GANs, VAEs, Diffusion
What are Generative Models? Models that learn the distribution of data and generate new samples. Tasks: Image generation (DALL-E, Stable Diffusion, Midjourney) Image editing (inpainting, style...
What are Generative Models?
Models that learn the distribution of data and generate new samples.
Tasks:
- Image generation (DALL-E, Stable Diffusion, Midjourney)
- Image editing (inpainting, style transfer)
- Super-resolution
- Data augmentation
- Anomaly detection
Taxonomy
- Explicit density: model P(x) directly (VAE, autoregressive)
- Implicit density: learn to sample without P(x) (GAN)
- Score-based / diffusion: learn the gradient of log P(x)
Each has strengths.
Autoencoders
Compress input to a latent code, reconstruct:
x → Encoder → z → Decoder → x̂
Trained to minimize ||x - x̂||. Bottleneck forces compact representation.
Used for dimensionality reduction, denoising, feature learning.
But autoencoders are not generative — no way to sample new data.
Variational Autoencoders (VAEs)
Encoder outputs a distribution (μ, σ), not a point. Sample z from it, decode.
Loss = reconstruction loss + KL divergence to prior.
L = ||x - x̂||² + KL(q(z|x) || p(z))
Benefits:
- Smooth latent space
- Can sample new data
- Principled probabilistic framework
Downside: blurry samples (from MSE loss).
GANs (Generative Adversarial Networks)
Goodfellow et al. (2014). Two networks:
- Generator: produces fake images from noise
- Discriminator: distinguishes real from fake
Trained adversarially:
min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]
Generator tries to fool discriminator; discriminator tries not to be fooled.
Result: generator produces realistic images.
GAN Variants
- DCGAN: convolutional GAN, stable architecture
- CycleGAN: unpaired image translation (horse ↔ zebra)
- StyleGAN: high-quality faces, controllable style
- Pix2Pix: paired image translation (edges → photos)
- BigGAN: class-conditional, high fidelity
- Progressive GAN: grow resolution during training
GAN Challenges
- Mode collapse: generator produces limited variety
- Training instability: hard to balance G and D
- Evaluation: hard to measure quality (FID, IS)
Diffusion Models
Sohl-Dickstein et al. (2015), popularized by Ho et al. (2020).
Forward process: gradually add noise to image.
x_t = √(1-β_t) x_{t-1} + √β_t ε
After T steps, x_T is pure noise.
Reverse process: learn to denoise step by step.
x_{t-1} = (1/√α_t) (x_t - (1-α_t)/√(1-ᾱ_t) ε_θ(x_t, t))
Train network to predict noise ε at each step.
Why Diffusion Works
- Stable training (simple MSE loss)
- High-quality samples
- Covers the data distribution (no mode collapse)
- Controllable generation
Downside: slow sampling (many steps). Solved by DDIM, DPM-Solver.
Stable Diffusion
Latent diffusion: apply diffusion in a compressed latent space (VAE encoded).
Benefits: much faster, lower compute.
Text conditioning via CLIP text encoder → cross-attention.
Powers DALL-E 2, Stable Diffusion, Midjourney, Imagen.
Classifier-Free Guidance
Improve sample quality by combining conditional and unconditional predictions:
ε = ε_uncond + w (ε_cond - ε_uncond)
w is the guidance scale. Higher w → more fidelity to prompt, less diversity.
ControlNet
Add spatial conditioning (pose, edges, depth) to diffusion models.
Enables precise control over generation.
Evaluation Metrics
- FID (Fréchet Inception Distance): distance between real and generated distributions in feature space. Lower is better.
- IS (Inception Score): quality + diversity. Higher is better.
- CLIP Score: text-image alignment.
No metric is perfect; human eval still matters.
Applications
- Text-to-image (DALL-E 3, SDXL, Midjourney)
- Text-to-video (Sora, Runway)
- Image editing (inpainting, outpainting)
- 3D generation (DreamFusion)
- Drug discovery (molecule generation)
- Audio generation (MusicGen, AudioLM)
Key Takeaways
- Generative models learn P(x)
- Autoencoders compress; VAEs sample
- GANs: adversarial, sharp but unstable
- Diffusion: iterative denoising, high quality, stable
- Stable Diffusion: latent diffusion + text conditioning
- Guidance, ControlNet for control
- FID is standard metric
- Diffusions dominate image generation
Advanced Topics and Deployment
Self-Supervised Learning Learn representations without labels. Use pretext tasks: Contrastive: pull similar pairs together, push dissimilar apart Masked prediction: predict masked parts Rotatio...
Self-Supervised Learning
Learn representations without labels. Use pretext tasks:
- Contrastive: pull similar pairs together, push dissimilar apart
- Masked prediction: predict masked parts
- Rotation: predict how image was rotated
Methods: SimCLR, MoCo, BYOL, DINO, MAE.
Why it matters: unlabeled data is abundant; labels are expensive.
Foundation models (CLIP, DINOv2) trained self-supervised.
Vision Transformers (ViT)
Apply transformer to images: split into patches, treat as tokens.
Image (224×224) → 16×16 patches → 196 tokens → Transformer
Requires more data than CNNs. On ImageNet alone, ViT underperforms ResNet. On larger datasets (JFT-300M), it surpasses.
Modern: Swin, DeiT, BEiT, DINOv2.
Multimodal Models
Combine vision + language:
- CLIP: contrastive image-text pretraining
- BLIP: captioning, VQA
- Flamingo: few-shot visual understanding
- GPT-4V, Gemini: native multimodal LLMs
- LLaVA: open-source vision-language
Enables image captioning, VQA, image-grounded dialogue.
Neural Architecture Search (NAS)
Automate architecture design.
- Reinforcement learning: controller proposes architectures
- Evolutionary: mutate and select
- Differentiable (DARTS): relax discrete choices
Found architectures: NASNet, AmoebaNet, EfficientNet.
Expensive to run but produces efficient models.
Model Compression
Deploy large models on small devices.
Pruning: remove weights/neurons. Sparse models run faster.
Quantization: reduce precision.
- Post-training: quantize trained model (INT8)
- Quantization-aware training: simulate quantization during training
- 4-bit, 2-bit for LLMs
Knowledge distillation: train small student to mimic large teacher.
Low-rank factorization: replace layers with smaller matrices.
Combined: 10-100x smaller models with minor quality loss.
Efficient Architectures
- MobileNet: depthwise separable convolutions
- ShuffleNet: channel shuffle for efficiency
- EfficientNet: compound scaling
- MobileViT: hybrid CNN + transformer
Used in mobile, embedded, edge.
Hardware for Deep Learning
- GPUs: NVIDIA dominates (A100, H100, RTX 4090)
- TPUs: Google's tensor processing units
- NPUs: dedicated neural network accelerators (Apple Neural Engine, Qualcomm Hexagon)
- FPGAs: reconfigurable, flexible
- ASICs: custom silicon (Google TPU, Tesla FSD)
- Neuromorphic: brain-inspired (Intel Loihi)
Each has trade-offs in speed, power, cost, flexibility.
Frameworks
- PyTorch: research standard, dynamic graphs
- TensorFlow: production, Keras API
- JAX: functional, autograd, XLA
- MXNet: Amazon, legacy
- ONNX: interchange format
- TVM: compiler for deployment
PyTorch dominates research; TF/JAX in production.
Deployment
From trained model to production:
- Export to ONNX or TorchScript
- Optimize (quantization, pruning)
- Convert to target (TensorRT, CoreML, TFLite)
- Serve (REST API, gRPC)
- Monitor (latency, accuracy, drift)
Tools: TorchServe, Triton, KServe, BentoML.
Ethics and Fairness
Models can be biased, unfair, or harmful.
Issues:
- Bias: racial, gender, age in predictions
- Privacy: models memorize training data
- Deepfakes: realistic fake media
- Misuse: surveillance, manipulation
- Environmental: energy cost of training
Mitigation:
- Bias testing and correction
- Differential privacy
- Watermarking generated content
- Regulation (EU AI Act)
Interpretability
Understanding why models predict what they do.
- Saliency maps: which pixels matter
- Grad-CAM: class activation maps
- Attention visualization: what the model attends to
- LIME, SHAP: local explanations
Important for high-stakes applications (medical, legal, autonomous).
Future Directions
- Foundation models: one model for many tasks
- Multimodal: text, image, audio, video
- Efficient: smaller, faster, cheaper
- Robust: handle distribution shift, adversarial attacks
- Aligned: safe, honest, helpful
- Reasoning: chain-of-thought, planning
- Agents: LLMs that take actions
The field moves fast. Stay curious.
Key Takeaways
- Self-supervised learning uses unlabeled data
- ViT brings transformers to vision
- Multimodal models combine vision + language
- NAS automates architecture design
- Compression: pruning, quantization, distillation
- Efficient architectures for edge
- GPUs/TPUs/NPUs/FPGAs for training and inference
- PyTorch, TF, JAX frameworks
- Deployment: export, optimize, serve, monitor
- Ethics, interpretability, and safety are critical
📝 Problem Set 1: CNNs, Training, Backprop
Q1. A conv layer has input 32×32×3, 64 filters of size 3×3, stride 1, no padding. What is the output size and parameter count?
Show Solution
Output: (32 - 3 + 0)/1 + 1 = 30, so 30×30×64. Params per filter: 3×3×3 + 1 = 28. Total: 64 × 28 = 1792.
Q2. Compute the output of a 2×2 max-pool on the matrix [[1,3,2,4],[5,6,7,8],[9,1,2,3],[4,5,6,7]] with stride 2.
Show Solution
Top-left 2×2: max(1,3,5,6)=6. Top-right: max(2,4,7,8)=8. Bottom-left: max(9,1,4,5)=9. Bottom-right: max(2,3,6,7)=7. Output: [[6,8],[9,7]].
Q3. Explain why batch normalization helps training and where the learned parameters γ and β come in.
Show Solution
BatchNorm normalizes activations to zero mean and unit variance per mini-batch, then scales by γ and shifts by β (both learned). This stabilizes activations, allows higher LR, reduces sensitivity to init, and acts as mild regularization. γ/β let the network undo normalization if needed.
🎯 Midterm 1: CNNs, Training, Detection
Q1. Compute gradients for a 2-layer network: h = ReLU(W1 x), y = softmax(W2 h), loss = -log(y_true). Show dL/dW2 and dL/dW1.
Show Solution
dL/dz2 = y - y_true (softmax + CE gradient). dL/dW2 = (y - y_true) h^T. dL/dh = W2^T (y - y_true). dL/dz1 = dL/dh ⊙ (z1 > 0). dL/dW1 = dL/dz1 × x^T.
Q2. Explain the difference between semantic, instance, and panoptic segmentation. Give a use case for each.
Show Solution
Semantic: each pixel classified by class (road, car, person) — no distinction between instances. Used in scene understanding. Instance: each object instance separate (car 1, car 2). Used in counting, robotics. Panoptic: combines both — every pixel labeled with class + instance. Used in autonomous driving.
Q3. A YOLO model outputs 3 boxes at a location: [(0.1, 0.4), (0.2, 0.3), (0.15, 0.35)] with confidences (0.9, 0.8, 0.7). Apply NMS with IoU threshold 0.5.
Show Solution
Sort by confidence: (0.1,0.4)@0.9, (0.2,0.3)@0.8, (0.15,0.35)@0.7. Box 1 kept. Compute IoU with box 2: high overlap → suppress. IoU with box 3: high overlap → suppress. Final: only box 1 remains.