Natural Language Processing
Word vectors, RNNs, transformers, and modern NLP with deep learning.
Introduction to NLP and Word Vectors
What is Natural Language Processing? NLP is the field of AI that deals with understanding and generating human language. It spans: Syntax: structure of sentences (parsing, part-of-speech tagging)...
What is Natural Language Processing?
NLP is the field of AI that deals with understanding and generating human language. It spans:
- Syntax: structure of sentences (parsing, part-of-speech tagging)
- Semantics: meaning of words and sentences
- Pragmatics: meaning in context (dialogue, implicature)
- Applications: translation, summarization, question answering, sentiment analysis, chatbots
NLP is hard because language is ambiguous, context-dependent, and continually evolving.
Why is Language Hard for Computers?
Consider: "I saw the man with the telescope."
Two meanings:
- I used a telescope to see the man.
- I saw a man who had a telescope.
The same words, different syntactic structures, different meanings.
Add pragmatics: "Can you pass the salt?" is a request, not a question. Sarcasm, idioms, cultural references make it even harder.
Classical NLP vs Neural NLP
Classical (pre-2013): hand-crafted features, statistical models (HMMs, CRFs, SVMs). Rule-based systems.
Neural (2013+): word embeddings, RNNs, then transformers. Learned representations end-to-end.
The shift happened because neural methods scaled with data and compute.
Word Representations
How do we represent a word as a vector?
One-hot: vector of size |V|, one 1 for the word, rest 0s. Simple, but:
- No notion of similarity
- Huge vectors (|V| = 50,000+)
- No generalization
Word embeddings: dense, low-dimensional vectors (50-300 dims) where similar words are close.
Word2Vec
Mikolov et al. (2013) introduced Word2Vec. Two variants:
Skip-gram: predict context words from center word.
P(w_{t+j} | w_t) for j = -m..m
CBOW: predict center word from context.
Both train a neural network on a large corpus. The learned weights become the embeddings.
The Skip-gram Objective
Maximize the log probability of context words given center words:
J(ฮธ) = (1/T) ฮฃ_t ฮฃ_{j=-m, jโ 0}^m log P(w_{t+j} | w_t; ฮธ)
Where P uses softmax:
P(o | c) = exp(u_o^T v_c) / ฮฃ_w exp(u_w^T v_c)
Softmax over the entire vocabulary is expensive. Solutions:
- Negative sampling: sample a few "negative" words, train binary classifier
- Hierarchical softmax: tree-structured computation
Properties of Word2Vec Embeddings
Learned embeddings capture semantic relationships:
king - man + woman โ queen
Paris - France + Italy โ Rome
walking - walk + swim โ swimming
Analogies work in the embedding space. This surprised everyone.
GloVe
Global Vectors (Pennington et al., 2014). Combines:
- Global matrix factorization (like LSA)
- Local context windows (like Word2Vec)
Factorizes the co-occurrence matrix directly. Often works as well as Word2Vec.
FastText
Facebook's extension. Represents words as bags of character n-grams.
Benefits:
- Handles out-of-vocabulary words (morphology)
- Better for morphologically rich languages
- Robust to typos
Contextual Embeddings
Word2Vec and GloVe give one vector per word, regardless of context. But "bank" means different things in "river bank" and "bank account".
ELMo (2018): bidirectional LSTM, context-dependent embeddings. First big win.
BERT (2018): transformer-based, deep bidirectional. State-of-the-art on many tasks.
GPT (2018+): transformer decoder, generative. Powers ChatGPT.
Contextual embeddings replaced static ones almost entirely.
Subword Tokenization
Modern NLP uses subword units (BPE, WordPiece, SentencePiece) instead of whole words.
Benefits:
- Fixed vocabulary size
- Handles rare words
- Language-agnostic
"unhappiness" โ "un", "happi", "ness"
This is what GPT/BERT use.
Key Takeaways
- NLP spans syntax, semantics, pragmatics
- Language is ambiguous and context-dependent
- One-hot encoding is simple but weak
- Word2Vec/GloVe: static embeddings, capture analogies
- FastText: handles morphology
- Contextual embeddings (ELMo, BERT, GPT) are current standard
- Subword tokenization handles open vocabulary
Recurrent Neural Networks and Language Models
Language Modeling A language model assigns probabilities to sequences: P(w_1, w_2, ..., w_n) = P(w_1) ร P(w_2 | w_1) ร ... ร P(w_n | w_1..w_{n-1}) Used for: Speech recognition (pick most likely...
Language Modeling
A language model assigns probabilities to sequences:
P(w_1, w_2, ..., w_n) = P(w_1) ร P(w_2 | w_1) ร ... ร P(w_n | w_1..w_{n-1})
Used for:
- Speech recognition (pick most likely transcript)
- Machine translation
- Text generation
- Autocomplete
N-gram Models
Approximate P(w_n | w_1..w_{n-1}) โ P(w_n | w_{n-k}..w_{n-1}).
For bigrams (k=1), trigrams (k=2). Estimate from counts:
P(w_n | w_{n-1}) = count(w_{n-1}, w_n) / count(w_{n-1})
Simple but:
- Sparse (many n-grams never seen)
- Curse of dimensionality (exponential in n)
- No generalization (can't relate similar contexts)
Neural Language Models
Bengio et al. (2003): feed-forward NN with embeddings.
Input: previous n-1 words (embeddings)
Hidden: nonlinear transformation
Output: softmax over vocabulary
Benefits: shared parameters across contexts, smooth probabilities.
Recurrent Neural Networks
An RNN processes a sequence one element at a time, maintaining a hidden state.
h_t = tanh(W_hh ร h_{t-1} + W_xh ร x_t + b_h)
y_t = W_hy ร h_t + b_y
The hidden state h_t summarizes everything seen so far.
Unrolled through time, an RNN shares weights across timesteps.
RNN for Language Modeling
At each step t:
- Input: word embedding x_t
- Hidden: h_t
- Output: distribution over next word
P(w_{t+1} | w_1..w_t) = softmax(W ร h_t)
Backpropagation Through Time (BPTT)
Train RNN by unrolling and backpropagating. Gradients flow from output at time t back through all earlier steps.
Problem: vanishing/exploding gradients.
- Vanishing: gradients shrink exponentially with sequence length. Network can't learn long-term dependencies.
- Exploding: gradients grow exponentially. Training diverges.
Solutions:
- Gradient clipping: cap gradient magnitude
- Better activations: ReLU helps
- LSTM, GRU: gated architectures
LSTM (Long Short-Term Memory)
Hochreiter & Schmidhuber (1997). Introduces:
- Cell state c_t: long-term memory
- Forget gate: what to forget
- Input gate: what to add
- Output gate: what to output
Gate equations (simplified):
f_t = ฯ(W_f [h_{t-1}, x_t])
i_t = ฯ(W_i [h_{t-1}, x_t])
o_t = ฯ(W_o [h_{t-1}, x_t])
c_t = f_t โ c_{t-1} + i_t โ tanh(W_c [h_{t-1}, x_t])
h_t = o_t โ tanh(c_t)
Gates let the network learn to preserve or reset information. Solves vanishing gradient for long sequences.
GRU (Gated Recurrent Unit)
Simpler than LSTM. Two gates:
- Update gate: combine forget and input
- Reset gate: how much past to forget
Fewer parameters, similar performance. Popular when compute-limited.
Bidirectional RNNs
For tasks where the entire sequence is available (e.g., encoding), process forward and backward, concatenate hidden states.
h_t = [h_t_forward; h_t_backward]
Captures context from both directions. Used in ELMo, BERT-style encoders.
Sequence-to-Sequence Models
Sutskever et al. (2014). Encoder RNN โ context vector โ decoder RNN.
Encoder reads input, produces final hidden state.
Decoder generates output, conditioned on that state.
Applications: translation, summarization, dialogue.
Problem: bottleneck โ the entire input compressed to one vector.
Attention Mechanism
Bahdanau et al. (2014). Decoder can "look at" all encoder states, not just the final one.
At each decoder step, compute attention weights over encoder states:
ฮฑ_t = softmax(score(h_dec, h_enc))
context = ฮฃ ฮฑ_t ร h_enc
Decoder uses context plus its own state.
Revolutionized NLP. Now transformers use self-attention everywhere.
Key Takeaways
- Language models predict next word
- N-gram models are simple but limited
- RNNs process sequences with shared weights
- Vanishing/exploding gradients are key issues
- LSTM, GRU solve long-term dependency
- Bidirectional for encoding tasks
- Seq2seq for translation
- Attention removes the bottleneck
Transformers and Attention
The Transformer Revolution Vaswani et al. (2017): "Attention Is All You Need." Replaced RNNs entirely. Now the standard for NLP, vision, speech, and more. Key innovation: self-attention โ...
The Transformer Revolution
Vaswani et al. (2017): "Attention Is All You Need." Replaced RNNs entirely. Now the standard for NLP, vision, speech, and more.
Key innovation: self-attention โ every position attends to every other position in parallel.
Benefits:
- Parallelizable (no sequential recurrence)
- Captures long-range dependencies
- Scales to billions of parameters
Self-Attention
Given input embeddings X (n ร d), compute:
Q = X W_Q (queries)
K = X W_K (keys)
V = X W_V (values)
Attention:
Attention(Q, K, V) = softmax(Q K^T / โd_k) V
Interpretation:
- Q K^T: similarity between each pair of positions
- Softmax: attention weights
- Multiply by V: weighted average of values
Every position gets a new representation that's a weighted sum of all positions.
Why โd_k?
Without scaling, dot products grow with d_k, pushing softmax into saturation (near 0 or 1). Scaling by โd_k keeps them in a reasonable range.
Multi-Head Attention
Instead of one attention operation, run h parallel "heads," each with its own W_Q, W_K, W_V.
MultiHead = Concat(head_1, ..., head_h) W_O
Each head can focus on different relationships (syntactic, semantic, positional).
Typical h = 8 or 16.
Positional Encoding
Self-attention is permutation-invariant โ it doesn't know word order. Add positional information:
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Or learnable position embeddings (BERT, GPT).
The Transformer Block
Each layer:
- Multi-head self-attention
- Add & Norm (residual + layer norm)
- Feed-forward (2 linear layers with ReLU)
- Add & Norm
Stack N such blocks (6 in original, 12-96 in modern models).
Encoder-Decoder Architecture
Encoder: stack of self-attention blocks. Each position attends to all positions.
Decoder: masked self-attention (each position attends only to earlier positions) + cross-attention to encoder outputs.
Used in translation (T5, BART).
BERT (Bidirectional Encoder Representations)
Devlin et al. (2018). Encoder-only transformer.
Pre-trained with two tasks:
- Masked language modeling (MLM): predict randomly masked words
- Next sentence prediction (NSP): predict if two sentences are consecutive
Fine-tuned per task (classification, QA, NER).
BERT-Base: 12 layers, 110M params.
BERT-Large: 24 layers, 340M params.
GPT (Generative Pre-trained Transformer)
OpenAI. Decoder-only transformer. Trained to predict the next token on massive text.
GPT-1 (2018): 117M params.
GPT-2 (2019): 1.5B.
GPT-3 (2020): 175B.
GPT-4 (2023): ~1.7T (estimated, MoE).
Zero-shot and few-shot learning via prompting.
Scaling Laws
Kaplan et al. (2020): model performance follows a power law in compute, parameters, and data.
Chinchilla (2022): models were undertrained. Optimal: ~20 tokens per parameter.
Modern models train on trillions of tokens.
Beyond NLP
Transformers now dominate:
- Vision (ViT): patches as tokens
- Speech (Whisper): spectrograms as tokens
- Protein folding (AlphaFold): amino acids as tokens
- Reinforcement learning (Decision Transformer): states as tokens
A single architecture for everything.
Key Takeaways
- Transformers replaced RNNs entirely
- Self-attention: every position attends to every other
- Multi-head for diverse relationships
- Positional encoding for order
- BERT (encoder) for understanding
- GPT (decoder) for generation
- Scaling laws guide model size
- Transformers dominate NLP, vision, speech, biology
Pretraining, Fine-tuning, and Transfer Learning
The Pretrain-Finetune Paradigm Training large models from scratch is expensive. Instead: 1. Pretrain on massive unlabeled text (months of GPU time) 2. Fine-tune on task-specific data (hours-days) 3....
The Pretrain-Finetune Paradigm
Training large models from scratch is expensive. Instead:
- Pretrain on massive unlabeled text (months of GPU time)
- Fine-tune on task-specific data (hours-days)
- Deploy on the target task
Transfer learning is now standard in NLP.
Why Does Pretraining Work?
The model learns general language understanding:
- Syntax (grammatical structure)
- Semantics (word meanings)
- World knowledge (facts about the world)
- Reasoning patterns
This knowledge transfers to any downstream NLP task.
Pretraining Objectives
Causal LM (GPT): predict next token.
P(x_t | x_1, ..., x_{t-1})
Masked LM (BERT): predict masked tokens.
P(x_masked | context)
Span corruption (T5): replace spans with sentinel tokens, predict them.
Prefix LM (PaLM, UL2): hybrid.
Each has different strengths.
Fine-tuning
Add a task-specific head and train on labeled data:
[CLS] The movie was great [SEP]
โ
Encoder
โ
[CLS] representation โ classifier โ positive
Small dataset: 1k-100k examples. Training: hours.
Few-shot and Zero-shot Learning
Modern large LMs can perform tasks without fine-tuning:
Zero-shot: just a prompt.
"Translate to French: Hello"
Few-shot: include a few examples in the prompt.
"cat: chat. dog: chien. bird:"
GPT-3 popularized this. No gradient updates needed.
Instruction Tuning
Fine-tune on (instruction, response) pairs:
Instruction: "Summarize the following..."
Response: "..."
Models like InstructGPT, FLAN-T5, Alpaca are instruction-tuned.
Result: models follow instructions much better.
RLHF (Reinforcement Learning from Human Feedback)
Align model with human preferences:
- Supervised fine-tuning: train on demonstrations
- Reward model: learn to predict human preference
- RL (PPO): optimize model to maximize reward
Used for ChatGPT, Claude, Gemini.
The reward model captures what humans want (helpful, harmless, honest).
LoRA (Low-Rank Adaptation)
Fine-tuning large models is expensive. LoRA freezes the base model and adds small trainable matrices:
W' = W + B ร A
Where A and B are low-rank (rank 4-64). Only these are trained.
Benefits: 10,000x fewer parameters, similar performance, easy to swap.
QLoRA: quantized base + LoRA โ fine-tune 65B on a single 48GB GPU.
Prompt Engineering
Crafting prompts to get desired outputs:
- Zero-shot: "Classify sentiment: This product is amazing."
- Chain-of-thought: "Let's think step by step..."
- Role prompting: "You are a helpful assistant..."
- Few-shot: include examples
- ReAct: reason + act (for tool use)
Prompt engineering is now a skill, but its importance may fade as models improve.
Adapters
Add small trainable modules between frozen layers. Alternative to LoRA.
Multi-task adapters: one adapter per task, switch at inference.
Distillation
Train a small "student" model to mimic a large "teacher."
Student is faster, cheaper, sometimes nearly as good.
Examples:
- DistilBERT: 40% smaller, 60% faster, 97% of BERT performance
- TinyLlama, Phi-2: small but capable
In-Context Learning
The model learns from examples in the prompt without parameter updates.
Emergent behavior: only appears in large models (>1B params).
Why it works is still debated โ implicit gradient descent? Pattern matching?
Retrieval-Augmented Generation (RAG)
Combine language model with a retrieval system:
- Query a database for relevant documents
- Feed documents + question into the LLM
- LLM generates answer grounded in the documents
Benefits:
- Up-to-date knowledge (no retraining)
- Cite sources
- Reduce hallucination
Used in customer support, legal, medical.
Key Takeaways
- Pretrain-finetune is standard
- Pretraining learns general language understanding
- Fine-tuning adapts to tasks
- Few-shot and zero-shot via prompting
- Instruction tuning improves following
- RLHF aligns with human preferences
- LoRA makes fine-tuning affordable
- Prompt engineering is a new skill
- Distillation compresses large models
- RAG combines retrieval with generation
Modern NLP Applications
Machine Translation Translate from source to target language. Classical: rule-based โ statistical (phrase-based) โ neural (seq2seq). Neural MT: encoder-decoder transformer. State-of-the-art since...
Machine Translation
Translate from source to target language.
Classical: rule-based โ statistical (phrase-based) โ neural (seq2seq).
Neural MT: encoder-decoder transformer. State-of-the-art since 2016.
Challenges:
- Low-resource languages (limited parallel data)
- Long documents (coherence)
- Idioms, cultural references
- Morphologically rich languages
Modern systems: Google Translate, DeepL, Meta NLLB.
Question Answering
Given a question and context, produce an answer.
Extractive: select a span from context (BERT fine-tuned on SQuAD).
Generative: produce answer from scratch (GPT).
Retrieval-based: find answer in a knowledge base.
Open-domain: combine retrieval + reader (DPR, RAG).
Examples: ChatGPT, Google Search's "People also ask."
Text Summarization
Condense long text into a shorter version.
Extractive: select important sentences.
Abstractive: generate new text (transformer seq2seq).
Multi-document: summarize many documents.
Evaluation: ROUGE (n-gram overlap), BERTScore (semantic similarity), human eval.
Challenges: factual consistency (hallucination), length control, style.
Sentiment Analysis
Classify text as positive, negative, neutral. Or with fine-grained labels (1-5 stars).
Applications: product reviews, social media monitoring, customer feedback.
Simple with BERT fine-tuning. ~95% accuracy on standard benchmarks.
Beyond binary: aspect-based sentiment ("The food was great but the service was slow").
Named Entity Recognition (NER)
Identify people, organizations, locations, dates, etc.
Classical: CRF, HMM.
Neural: BiLSTM-CRF, BERT.
Modern: LLM with prompting.
Standard datasets: CoNLL-2003, OntoNotes.
Text Classification
Any task that assigns labels to text:
- Spam detection
- Topic classification
- Intent recognition (chatbots)
- Hate speech detection
- Fake news detection
BERT + softmax head is the standard approach.
Dialogue Systems
Task-oriented: book flights, order food. Slot filling + intent.
Open-domain: chat about anything. GPT-based.
Retrieval-based: select response from a database.
Generative: produce response from scratch.
Modern: LLMs are open-domain dialogue systems.
Information Extraction
Turn unstructured text into structured data:
- Relation extraction: (Obama, born_in, Hawaii)
- Event extraction: who, what, when, where
- Coreference resolution: "He" โ "Obama"
Foundation for knowledge graphs.
Text Generation
Generate coherent, fluent text:
- Story generation: given prompt, write story
- Code generation: given description, write code
- Dialogue: chatbot responses
- Data-to-text: describe a table in natural language
Modern: GPT-4, Claude, Gemini generate human-quality text.
Ethics and Bias
NLP models learn from human text, which contains bias.
Issues:
- Gender bias: "doctor" โ male, "nurse" โ female
- Racial bias: sentiment differs by dialect
- Political bias: training data skews
- Toxicity: models generate harmful content
Mitigation: data filtering, debiasing techniques, RLHF, red-teaming.
Evaluation
BLEU: n-gram overlap, machine translation.
ROUGE: n-gram overlap, summarization.
METEOR: synonyms and stemming.
BERTScore: semantic similarity via embeddings.
Perplexity: language model quality.
Human eval: gold standard, but expensive.
No metric is perfect. Multiple metrics + human eval is best.
Deployment
Large models are expensive to serve:
- Quantization: 8-bit, 4-bit reduce memory
- Distillation: train smaller student
- Pruning: remove weights
- Batching: process multiple requests
- Caching: store frequent responses
- Edge deployment: small models on device
Trade-offs between latency, cost, and quality.
Future Directions
- Multimodal: text + image + audio (GPT-4V, Gemini)
- Agents: LLMs that use tools, take actions
- Long context: million-token windows
- Efficiency: sparse models, MoE
- Reasoning: chain-of-thought, tree search
- Alignment: ensuring helpful, harmless, honest
Rapidly evolving field.
Key Takeaways
- Translation, QA, summarization are mature
- NER, classification, sentiment standard
- Dialogue systems revolutionized by LLMs
- Text generation approaching human quality
- Bias and ethics are serious concerns
- Multiple evaluation metrics needed
- Deployment requires efficiency techniques
- Future: multimodal, agents, long context
๐ Problem Set 1: Word Embeddings, Attention
Q1. Given embeddings king = [0.2, 0.5], man = [0.1, 0.3], woman = [0.4, 0.6], compute the nearest word to 'king - man + woman'.
Show Solution
king - man + woman = [0.2-0.1+0.4, 0.5-0.3+0.6] = [0.5, 0.8]. Compare to queen = [0.5, 0.7]. Close! Nearest word in vocab is likely queen.
Q2. Compute self-attention for tokens x1 = [1, 0], x2 = [0, 1], with W_Q = W_K = W_V = I (identity).
Show Solution
Q = X, K = X, V = X. Scores: QK^T = [[1, 0], [0, 1]]. Scaled by 1/โ2: [[0.707, 0], [0, 0.707]]. Softmax row 1: [0.67, 0.33]. Row 2: [0.33, 0.67]. Output row 1: 0.67ร[1,0] + 0.33ร[0,1] = [0.67, 0.33]. Row 2: [0.33, 0.67].
Q3. Explain why transformer attention is O(nยฒ) in sequence length and how recent models address this.
Show Solution
Attention matrix is nรn (every token attends to every other). n=10k โ 100M entries. Solutions: (1) Sparse attention (Longformer, BigBird), (2) Linear attention (Performer), (3) Recurrent memory (Transformer-XL), (4) State-space models (Mamba, S4), (5) Flash attention โ same complexity but IO-efficient.
๐ฏ Midterm 1: Embeddings, RNNs, Transformers
Q1. Compute the output of a 2-unit RNN at time t=2, given x1=[1,0], x2=[0,1], h0=[0,0], W_hh = [[0.5, 0], [0, 0.5]], W_xh = I, b_h = 0, tanh activation.
Show Solution
t=1: h1 = tanh(W_hh h0 + W_xh x1) = tanh([1, 0]) = [0.762, 0]. t=2: h2 = tanh(W_hh h1 + W_xh x2) = tanh([0.381+0, 0+1]) = tanh([0.381, 1]) = [0.363, 0.762].
Q2. Explain why BERT is bidirectional but GPT is not. How does this affect their use cases?
Show Solution
BERT: MLM objective โ some tokens masked, model sees context on both sides. Bidirectional. Good for understanding (classification, QA, NER). GPT: causal LM โ each token only sees earlier tokens. Unidirectional. Good for generation (text completion, dialogue). Bidirectional is stronger for encoding; unidirectional matches generation's autoregressive nature.
Q3. A language model assigns probability 0.01 to each of the 100 tokens in a sentence. What is the perplexity?
Show Solution
PPL = exp(-(1/N) ฮฃ log P(w_i)) = exp(-log(0.01)) = exp(4.605) = 100. Perplexity = 100 means the model is as uncertain as choosing uniformly among 100 options.