Reference topics: Stanford CS224n ยท independent study notes, not affiliated

Natural Language Processing

Word vectors, RNNs, transformers, and modern NLP with deep learning.

5Lectures
1Assignments
1Exams
40hDuration
1

Introduction to NLP and Word Vectors

What is Natural Language Processing? NLP is the field of AI that deals with understanding and generating human language. It spans: Syntax: structure of sentences (parsing, part-of-speech tagging)...

โฑ 50 min read

What is Natural Language Processing?

NLP is the field of AI that deals with understanding and generating human language. It spans:

  • Syntax: structure of sentences (parsing, part-of-speech tagging)
  • Semantics: meaning of words and sentences
  • Pragmatics: meaning in context (dialogue, implicature)
  • Applications: translation, summarization, question answering, sentiment analysis, chatbots

NLP is hard because language is ambiguous, context-dependent, and continually evolving.

Why is Language Hard for Computers?

Consider: "I saw the man with the telescope."

Two meanings:

  1. I used a telescope to see the man.
  2. I saw a man who had a telescope.

The same words, different syntactic structures, different meanings.

Add pragmatics: "Can you pass the salt?" is a request, not a question. Sarcasm, idioms, cultural references make it even harder.

Classical NLP vs Neural NLP

Classical (pre-2013): hand-crafted features, statistical models (HMMs, CRFs, SVMs). Rule-based systems.

Neural (2013+): word embeddings, RNNs, then transformers. Learned representations end-to-end.

The shift happened because neural methods scaled with data and compute.

Word Representations

How do we represent a word as a vector?

One-hot: vector of size |V|, one 1 for the word, rest 0s. Simple, but:

  • No notion of similarity
  • Huge vectors (|V| = 50,000+)
  • No generalization

Word embeddings: dense, low-dimensional vectors (50-300 dims) where similar words are close.

Word2Vec

Mikolov et al. (2013) introduced Word2Vec. Two variants:

Skip-gram: predict context words from center word.

P(w_{t+j} | w_t) for j = -m..m

CBOW: predict center word from context.

Both train a neural network on a large corpus. The learned weights become the embeddings.

The Skip-gram Objective

Maximize the log probability of context words given center words:

J(ฮธ) = (1/T) ฮฃ_t ฮฃ_{j=-m, jโ‰ 0}^m log P(w_{t+j} | w_t; ฮธ)

Where P uses softmax:

P(o | c) = exp(u_o^T v_c) / ฮฃ_w exp(u_w^T v_c)

Softmax over the entire vocabulary is expensive. Solutions:

  • Negative sampling: sample a few "negative" words, train binary classifier
  • Hierarchical softmax: tree-structured computation

Properties of Word2Vec Embeddings

Learned embeddings capture semantic relationships:

king - man + woman โ‰ˆ queen
Paris - France + Italy โ‰ˆ Rome
walking - walk + swim โ‰ˆ swimming

Analogies work in the embedding space. This surprised everyone.

GloVe

Global Vectors (Pennington et al., 2014). Combines:

  • Global matrix factorization (like LSA)
  • Local context windows (like Word2Vec)

Factorizes the co-occurrence matrix directly. Often works as well as Word2Vec.

FastText

Facebook's extension. Represents words as bags of character n-grams.

Benefits:

  • Handles out-of-vocabulary words (morphology)
  • Better for morphologically rich languages
  • Robust to typos

Contextual Embeddings

Word2Vec and GloVe give one vector per word, regardless of context. But "bank" means different things in "river bank" and "bank account".

ELMo (2018): bidirectional LSTM, context-dependent embeddings. First big win.

BERT (2018): transformer-based, deep bidirectional. State-of-the-art on many tasks.

GPT (2018+): transformer decoder, generative. Powers ChatGPT.

Contextual embeddings replaced static ones almost entirely.

Subword Tokenization

Modern NLP uses subword units (BPE, WordPiece, SentencePiece) instead of whole words.

Benefits:

  • Fixed vocabulary size
  • Handles rare words
  • Language-agnostic

"unhappiness" โ†’ "un", "happi", "ness"

This is what GPT/BERT use.

Key Takeaways

  • NLP spans syntax, semantics, pragmatics
  • Language is ambiguous and context-dependent
  • One-hot encoding is simple but weak
  • Word2Vec/GloVe: static embeddings, capture analogies
  • FastText: handles morphology
  • Contextual embeddings (ELMo, BERT, GPT) are current standard
  • Subword tokenization handles open vocabulary
2

Recurrent Neural Networks and Language Models

Language Modeling A language model assigns probabilities to sequences: P(w_1, w_2, ..., w_n) = P(w_1) ร— P(w_2 | w_1) ร— ... ร— P(w_n | w_1..w_{n-1}) Used for: Speech recognition (pick most likely...

โฑ 50 min read

Language Modeling

A language model assigns probabilities to sequences:

P(w_1, w_2, ..., w_n) = P(w_1) ร— P(w_2 | w_1) ร— ... ร— P(w_n | w_1..w_{n-1})

Used for:

  • Speech recognition (pick most likely transcript)
  • Machine translation
  • Text generation
  • Autocomplete

N-gram Models

Approximate P(w_n | w_1..w_{n-1}) โ‰ˆ P(w_n | w_{n-k}..w_{n-1}).

For bigrams (k=1), trigrams (k=2). Estimate from counts:

P(w_n | w_{n-1}) = count(w_{n-1}, w_n) / count(w_{n-1})

Simple but:

  • Sparse (many n-grams never seen)
  • Curse of dimensionality (exponential in n)
  • No generalization (can't relate similar contexts)

Neural Language Models

Bengio et al. (2003): feed-forward NN with embeddings.

Input: previous n-1 words (embeddings)
Hidden: nonlinear transformation
Output: softmax over vocabulary

Benefits: shared parameters across contexts, smooth probabilities.

Recurrent Neural Networks

An RNN processes a sequence one element at a time, maintaining a hidden state.

h_t = tanh(W_hh ร— h_{t-1} + W_xh ร— x_t + b_h)
y_t = W_hy ร— h_t + b_y

The hidden state h_t summarizes everything seen so far.

Unrolled through time, an RNN shares weights across timesteps.

RNN for Language Modeling

At each step t:

  • Input: word embedding x_t
  • Hidden: h_t
  • Output: distribution over next word

P(w_{t+1} | w_1..w_t) = softmax(W ร— h_t)

Backpropagation Through Time (BPTT)

Train RNN by unrolling and backpropagating. Gradients flow from output at time t back through all earlier steps.

Problem: vanishing/exploding gradients.

  • Vanishing: gradients shrink exponentially with sequence length. Network can't learn long-term dependencies.
  • Exploding: gradients grow exponentially. Training diverges.

Solutions:

  • Gradient clipping: cap gradient magnitude
  • Better activations: ReLU helps
  • LSTM, GRU: gated architectures

LSTM (Long Short-Term Memory)

Hochreiter & Schmidhuber (1997). Introduces:

  • Cell state c_t: long-term memory
  • Forget gate: what to forget
  • Input gate: what to add
  • Output gate: what to output

Gate equations (simplified):

f_t = ฯƒ(W_f [h_{t-1}, x_t])
i_t = ฯƒ(W_i [h_{t-1}, x_t])
o_t = ฯƒ(W_o [h_{t-1}, x_t])
c_t = f_t โŠ™ c_{t-1} + i_t โŠ™ tanh(W_c [h_{t-1}, x_t])
h_t = o_t โŠ™ tanh(c_t)

Gates let the network learn to preserve or reset information. Solves vanishing gradient for long sequences.

GRU (Gated Recurrent Unit)

Simpler than LSTM. Two gates:

  • Update gate: combine forget and input
  • Reset gate: how much past to forget

Fewer parameters, similar performance. Popular when compute-limited.

Bidirectional RNNs

For tasks where the entire sequence is available (e.g., encoding), process forward and backward, concatenate hidden states.

h_t = [h_t_forward; h_t_backward]

Captures context from both directions. Used in ELMo, BERT-style encoders.

Sequence-to-Sequence Models

Sutskever et al. (2014). Encoder RNN โ†’ context vector โ†’ decoder RNN.

Encoder reads input, produces final hidden state.
Decoder generates output, conditioned on that state.

Applications: translation, summarization, dialogue.

Problem: bottleneck โ€” the entire input compressed to one vector.

Attention Mechanism

Bahdanau et al. (2014). Decoder can "look at" all encoder states, not just the final one.

At each decoder step, compute attention weights over encoder states:

ฮฑ_t = softmax(score(h_dec, h_enc))
context = ฮฃ ฮฑ_t ร— h_enc

Decoder uses context plus its own state.

Revolutionized NLP. Now transformers use self-attention everywhere.

Key Takeaways

  • Language models predict next word
  • N-gram models are simple but limited
  • RNNs process sequences with shared weights
  • Vanishing/exploding gradients are key issues
  • LSTM, GRU solve long-term dependency
  • Bidirectional for encoding tasks
  • Seq2seq for translation
  • Attention removes the bottleneck
3

Transformers and Attention

The Transformer Revolution Vaswani et al. (2017): "Attention Is All You Need." Replaced RNNs entirely. Now the standard for NLP, vision, speech, and more. Key innovation: self-attention โ€”...

โฑ 50 min read

The Transformer Revolution

Vaswani et al. (2017): "Attention Is All You Need." Replaced RNNs entirely. Now the standard for NLP, vision, speech, and more.

Key innovation: self-attention โ€” every position attends to every other position in parallel.

Benefits:

  • Parallelizable (no sequential recurrence)
  • Captures long-range dependencies
  • Scales to billions of parameters

Self-Attention

Given input embeddings X (n ร— d), compute:

Q = X W_Q (queries)
K = X W_K (keys)
V = X W_V (values)

Attention:

Attention(Q, K, V) = softmax(Q K^T / โˆšd_k) V

Interpretation:

  • Q K^T: similarity between each pair of positions
  • Softmax: attention weights
  • Multiply by V: weighted average of values

Every position gets a new representation that's a weighted sum of all positions.

Why โˆšd_k?

Without scaling, dot products grow with d_k, pushing softmax into saturation (near 0 or 1). Scaling by โˆšd_k keeps them in a reasonable range.

Multi-Head Attention

Instead of one attention operation, run h parallel "heads," each with its own W_Q, W_K, W_V.

MultiHead = Concat(head_1, ..., head_h) W_O

Each head can focus on different relationships (syntactic, semantic, positional).

Typical h = 8 or 16.

Positional Encoding

Self-attention is permutation-invariant โ€” it doesn't know word order. Add positional information:

PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Or learnable position embeddings (BERT, GPT).

The Transformer Block

Each layer:

  1. Multi-head self-attention
  2. Add & Norm (residual + layer norm)
  3. Feed-forward (2 linear layers with ReLU)
  4. Add & Norm

Stack N such blocks (6 in original, 12-96 in modern models).

Encoder-Decoder Architecture

Encoder: stack of self-attention blocks. Each position attends to all positions.

Decoder: masked self-attention (each position attends only to earlier positions) + cross-attention to encoder outputs.

Used in translation (T5, BART).

BERT (Bidirectional Encoder Representations)

Devlin et al. (2018). Encoder-only transformer.

Pre-trained with two tasks:

  • Masked language modeling (MLM): predict randomly masked words
  • Next sentence prediction (NSP): predict if two sentences are consecutive

Fine-tuned per task (classification, QA, NER).

BERT-Base: 12 layers, 110M params.
BERT-Large: 24 layers, 340M params.

GPT (Generative Pre-trained Transformer)

OpenAI. Decoder-only transformer. Trained to predict the next token on massive text.

GPT-1 (2018): 117M params.
GPT-2 (2019): 1.5B.
GPT-3 (2020): 175B.
GPT-4 (2023): ~1.7T (estimated, MoE).

Zero-shot and few-shot learning via prompting.

Scaling Laws

Kaplan et al. (2020): model performance follows a power law in compute, parameters, and data.

Chinchilla (2022): models were undertrained. Optimal: ~20 tokens per parameter.

Modern models train on trillions of tokens.

Beyond NLP

Transformers now dominate:

  • Vision (ViT): patches as tokens
  • Speech (Whisper): spectrograms as tokens
  • Protein folding (AlphaFold): amino acids as tokens
  • Reinforcement learning (Decision Transformer): states as tokens

A single architecture for everything.

Key Takeaways

  • Transformers replaced RNNs entirely
  • Self-attention: every position attends to every other
  • Multi-head for diverse relationships
  • Positional encoding for order
  • BERT (encoder) for understanding
  • GPT (decoder) for generation
  • Scaling laws guide model size
  • Transformers dominate NLP, vision, speech, biology
4

Pretraining, Fine-tuning, and Transfer Learning

The Pretrain-Finetune Paradigm Training large models from scratch is expensive. Instead: 1. Pretrain on massive unlabeled text (months of GPU time) 2. Fine-tune on task-specific data (hours-days) 3....

โฑ 50 min read

The Pretrain-Finetune Paradigm

Training large models from scratch is expensive. Instead:

  1. Pretrain on massive unlabeled text (months of GPU time)
  2. Fine-tune on task-specific data (hours-days)
  3. Deploy on the target task

Transfer learning is now standard in NLP.

Why Does Pretraining Work?

The model learns general language understanding:

  • Syntax (grammatical structure)
  • Semantics (word meanings)
  • World knowledge (facts about the world)
  • Reasoning patterns

This knowledge transfers to any downstream NLP task.

Pretraining Objectives

Causal LM (GPT): predict next token.

P(x_t | x_1, ..., x_{t-1})

Masked LM (BERT): predict masked tokens.

P(x_masked | context)

Span corruption (T5): replace spans with sentinel tokens, predict them.

Prefix LM (PaLM, UL2): hybrid.

Each has different strengths.

Fine-tuning

Add a task-specific head and train on labeled data:

[CLS] The movie was great [SEP]
โ†“
Encoder
โ†“
[CLS] representation โ†’ classifier โ†’ positive

Small dataset: 1k-100k examples. Training: hours.

Few-shot and Zero-shot Learning

Modern large LMs can perform tasks without fine-tuning:

Zero-shot: just a prompt.

"Translate to French: Hello"

Few-shot: include a few examples in the prompt.

"cat: chat. dog: chien. bird:"

GPT-3 popularized this. No gradient updates needed.

Instruction Tuning

Fine-tune on (instruction, response) pairs:

Instruction: "Summarize the following..."
Response: "..."

Models like InstructGPT, FLAN-T5, Alpaca are instruction-tuned.

Result: models follow instructions much better.

RLHF (Reinforcement Learning from Human Feedback)

Align model with human preferences:

  1. Supervised fine-tuning: train on demonstrations
  2. Reward model: learn to predict human preference
  3. RL (PPO): optimize model to maximize reward

Used for ChatGPT, Claude, Gemini.

The reward model captures what humans want (helpful, harmless, honest).

LoRA (Low-Rank Adaptation)

Fine-tuning large models is expensive. LoRA freezes the base model and adds small trainable matrices:

W' = W + B ร— A

Where A and B are low-rank (rank 4-64). Only these are trained.

Benefits: 10,000x fewer parameters, similar performance, easy to swap.

QLoRA: quantized base + LoRA โ†’ fine-tune 65B on a single 48GB GPU.

Prompt Engineering

Crafting prompts to get desired outputs:

  • Zero-shot: "Classify sentiment: This product is amazing."
  • Chain-of-thought: "Let's think step by step..."
  • Role prompting: "You are a helpful assistant..."
  • Few-shot: include examples
  • ReAct: reason + act (for tool use)

Prompt engineering is now a skill, but its importance may fade as models improve.

Adapters

Add small trainable modules between frozen layers. Alternative to LoRA.

Multi-task adapters: one adapter per task, switch at inference.

Distillation

Train a small "student" model to mimic a large "teacher."

Student is faster, cheaper, sometimes nearly as good.

Examples:

  • DistilBERT: 40% smaller, 60% faster, 97% of BERT performance
  • TinyLlama, Phi-2: small but capable

In-Context Learning

The model learns from examples in the prompt without parameter updates.

Emergent behavior: only appears in large models (>1B params).

Why it works is still debated โ€” implicit gradient descent? Pattern matching?

Retrieval-Augmented Generation (RAG)

Combine language model with a retrieval system:

  1. Query a database for relevant documents
  2. Feed documents + question into the LLM
  3. LLM generates answer grounded in the documents

Benefits:

  • Up-to-date knowledge (no retraining)
  • Cite sources
  • Reduce hallucination

Used in customer support, legal, medical.

Key Takeaways

  • Pretrain-finetune is standard
  • Pretraining learns general language understanding
  • Fine-tuning adapts to tasks
  • Few-shot and zero-shot via prompting
  • Instruction tuning improves following
  • RLHF aligns with human preferences
  • LoRA makes fine-tuning affordable
  • Prompt engineering is a new skill
  • Distillation compresses large models
  • RAG combines retrieval with generation
5

Modern NLP Applications

Machine Translation Translate from source to target language. Classical: rule-based โ†’ statistical (phrase-based) โ†’ neural (seq2seq). Neural MT: encoder-decoder transformer. State-of-the-art since...

โฑ 50 min read

Machine Translation

Translate from source to target language.

Classical: rule-based โ†’ statistical (phrase-based) โ†’ neural (seq2seq).

Neural MT: encoder-decoder transformer. State-of-the-art since 2016.

Challenges:

  • Low-resource languages (limited parallel data)
  • Long documents (coherence)
  • Idioms, cultural references
  • Morphologically rich languages

Modern systems: Google Translate, DeepL, Meta NLLB.

Question Answering

Given a question and context, produce an answer.

Extractive: select a span from context (BERT fine-tuned on SQuAD).

Generative: produce answer from scratch (GPT).

Retrieval-based: find answer in a knowledge base.

Open-domain: combine retrieval + reader (DPR, RAG).

Examples: ChatGPT, Google Search's "People also ask."

Text Summarization

Condense long text into a shorter version.

Extractive: select important sentences.

Abstractive: generate new text (transformer seq2seq).

Multi-document: summarize many documents.

Evaluation: ROUGE (n-gram overlap), BERTScore (semantic similarity), human eval.

Challenges: factual consistency (hallucination), length control, style.

Sentiment Analysis

Classify text as positive, negative, neutral. Or with fine-grained labels (1-5 stars).

Applications: product reviews, social media monitoring, customer feedback.

Simple with BERT fine-tuning. ~95% accuracy on standard benchmarks.

Beyond binary: aspect-based sentiment ("The food was great but the service was slow").

Named Entity Recognition (NER)

Identify people, organizations, locations, dates, etc.

Classical: CRF, HMM.
Neural: BiLSTM-CRF, BERT.
Modern: LLM with prompting.

Standard datasets: CoNLL-2003, OntoNotes.

Text Classification

Any task that assigns labels to text:

  • Spam detection
  • Topic classification
  • Intent recognition (chatbots)
  • Hate speech detection
  • Fake news detection

BERT + softmax head is the standard approach.

Dialogue Systems

Task-oriented: book flights, order food. Slot filling + intent.

Open-domain: chat about anything. GPT-based.

Retrieval-based: select response from a database.

Generative: produce response from scratch.

Modern: LLMs are open-domain dialogue systems.

Information Extraction

Turn unstructured text into structured data:

  • Relation extraction: (Obama, born_in, Hawaii)
  • Event extraction: who, what, when, where
  • Coreference resolution: "He" โ†’ "Obama"

Foundation for knowledge graphs.

Text Generation

Generate coherent, fluent text:

  • Story generation: given prompt, write story
  • Code generation: given description, write code
  • Dialogue: chatbot responses
  • Data-to-text: describe a table in natural language

Modern: GPT-4, Claude, Gemini generate human-quality text.

Ethics and Bias

NLP models learn from human text, which contains bias.

Issues:

  • Gender bias: "doctor" โ†’ male, "nurse" โ†’ female
  • Racial bias: sentiment differs by dialect
  • Political bias: training data skews
  • Toxicity: models generate harmful content

Mitigation: data filtering, debiasing techniques, RLHF, red-teaming.

Evaluation

BLEU: n-gram overlap, machine translation.
ROUGE: n-gram overlap, summarization.
METEOR: synonyms and stemming.
BERTScore: semantic similarity via embeddings.
Perplexity: language model quality.
Human eval: gold standard, but expensive.

No metric is perfect. Multiple metrics + human eval is best.

Deployment

Large models are expensive to serve:

  • Quantization: 8-bit, 4-bit reduce memory
  • Distillation: train smaller student
  • Pruning: remove weights
  • Batching: process multiple requests
  • Caching: store frequent responses
  • Edge deployment: small models on device

Trade-offs between latency, cost, and quality.

Future Directions

  • Multimodal: text + image + audio (GPT-4V, Gemini)
  • Agents: LLMs that use tools, take actions
  • Long context: million-token windows
  • Efficiency: sparse models, MoE
  • Reasoning: chain-of-thought, tree search
  • Alignment: ensuring helpful, harmless, honest

Rapidly evolving field.

Key Takeaways

  • Translation, QA, summarization are mature
  • NER, classification, sentiment standard
  • Dialogue systems revolutionized by LLMs
  • Text generation approaching human quality
  • Bias and ethics are serious concerns
  • Multiple evaluation metrics needed
  • Deployment requires efficiency techniques
  • Future: multimodal, agents, long context

๐Ÿ“ Problem Set 1: Word Embeddings, Attention

Q1. Given embeddings king = [0.2, 0.5], man = [0.1, 0.3], woman = [0.4, 0.6], compute the nearest word to 'king - man + woman'.

Show Solution

king - man + woman = [0.2-0.1+0.4, 0.5-0.3+0.6] = [0.5, 0.8]. Compare to queen = [0.5, 0.7]. Close! Nearest word in vocab is likely queen.

Q2. Compute self-attention for tokens x1 = [1, 0], x2 = [0, 1], with W_Q = W_K = W_V = I (identity).

Show Solution

Q = X, K = X, V = X. Scores: QK^T = [[1, 0], [0, 1]]. Scaled by 1/โˆš2: [[0.707, 0], [0, 0.707]]. Softmax row 1: [0.67, 0.33]. Row 2: [0.33, 0.67]. Output row 1: 0.67ร—[1,0] + 0.33ร—[0,1] = [0.67, 0.33]. Row 2: [0.33, 0.67].

Q3. Explain why transformer attention is O(nยฒ) in sequence length and how recent models address this.

Show Solution

Attention matrix is nร—n (every token attends to every other). n=10k โ†’ 100M entries. Solutions: (1) Sparse attention (Longformer, BigBird), (2) Linear attention (Performer), (3) Recurrent memory (Transformer-XL), (4) State-space models (Mamba, S4), (5) Flash attention โ€” same complexity but IO-efficient.

๐ŸŽฏ Midterm 1: Embeddings, RNNs, Transformers

Q1. Compute the output of a 2-unit RNN at time t=2, given x1=[1,0], x2=[0,1], h0=[0,0], W_hh = [[0.5, 0], [0, 0.5]], W_xh = I, b_h = 0, tanh activation.

Show Solution

t=1: h1 = tanh(W_hh h0 + W_xh x1) = tanh([1, 0]) = [0.762, 0]. t=2: h2 = tanh(W_hh h1 + W_xh x2) = tanh([0.381+0, 0+1]) = tanh([0.381, 1]) = [0.363, 0.762].

Q2. Explain why BERT is bidirectional but GPT is not. How does this affect their use cases?

Show Solution

BERT: MLM objective โ€” some tokens masked, model sees context on both sides. Bidirectional. Good for understanding (classification, QA, NER). GPT: causal LM โ€” each token only sees earlier tokens. Unidirectional. Good for generation (text completion, dialogue). Bidirectional is stronger for encoding; unidirectional matches generation's autoregressive nature.

Q3. A language model assigns probability 0.01 to each of the 100 tokens in a sentence. What is the perplexity?

Show Solution

PPL = exp(-(1/N) ฮฃ log P(w_i)) = exp(-log(0.01)) = exp(4.605) = 100. Perplexity = 100 means the model is as uncertain as choosing uniformly among 100 options.