Reference topics: Berkeley EE16A · independent study notes, not affiliated

Designing Information Devices

Linear algebra, circuit analysis, and system design for engineers. Foundational for embedded systems.

5Lectures
1Assignments
1Exams
42hDuration
1

Linear Systems and Circuit Basics

Why Linear Systems? Linear systems are the foundation of all engineering analysis. If a system is linear, superposition applies: the response to a sum of inputs is the sum of the individual...

⏱ 50 min read

Why Linear Systems?

Linear systems are the foundation of all engineering analysis. If a system is linear, superposition applies: the response to a sum of inputs is the sum of the individual responses.

Most real systems are approximately linear over some range. Even nonlinear systems are often analyzed by linearizing around an operating point.

Linearity Definition

A system S is linear if for all inputs x₁, x₂ and scalars a, b:

S(a x₁ + b x₂) = a S(x₁) + b S(x₂)

This is the superposition principle.

Linearity implies:

  • Homogeneity: S(ax) = a S(x)
  • Additivity: S(x₁ + x₂) = S(x₁) + S(x₂)
  • Zero input → zero output (for linear systems with no offset)

Circuit Elements as Linear Systems

Resistor: V = I × R. Linear.

Capacitor: I = C × dV/dt. Linear (differential operator).

Inductor: V = L × dI/dt. Linear.

Op-amp (ideal): Vout = A × (V+ - V-). Linear.

Diode: I = Is × (e^(V/Vt) - 1). Nonlinear.

Nodal Analysis

The standard method for solving circuits:

  1. Choose a reference node (ground)
  2. Assign node voltages to all other nodes
  3. Write KCL at each node: sum of currents leaving = 0
  4. Solve the system of equations

For n-1 unknown nodes, we get n-1 equations. Express in matrix form: G × v = i.

Example: Two-Node Circuit

Vin ──R1──┬──R2── GND
│
R3
│
GND

KCL at node 1:

(V1 - Vin)/R1 + V1/R2 + V1/R3 = 0

V1 × (1/R1 + 1/R2 + 1/R3) = Vin/R1

V1 = Vin/R1 / (1/R1 + 1/R2 + 1/R3)

The circuit is a voltage divider (R2 || R3 in series with R1).

Matrix Form

For larger circuits, the conductance matrix G is symmetric, which allows efficient solution.

G = [g11 g12 ...; g21 g22 ...; ...]

where gii = sum of conductances at node i, and gij = -conductance between i and j (i ≠ j).

Modern circuit simulators (SPICE) solve circuits this way.

Thevenin and Norton Equivalents

Any linear two-terminal network can be replaced by:

Thevenin: a voltage source V_th in series with a resistor R_th.

Norton: a current source I_n in parallel with a resistor R_n.

Conversion: V_th = I_n × R_n, R_th = R_n.

Finding Thevenin:

  1. Compute open-circuit voltage V_oc = V_th
  2. Zero out sources (short V sources, open I sources)
  3. Compute resistance between terminals = R_th

This is fundamental to circuit design.

Superposition Example

Given a circuit with two sources, compute the response by:

  1. Turn off source 1, compute response to source 2
  2. Turn off source 2, compute response to source 1
  3. Add the two responses

This is often easier than solving the full system at once.

Key Takeaways

  • Linearity = superposition principle
  • Nodal analysis is the standard method
  • G × v = i matrix form for larger circuits
  • Thevenin and Norton equivalents simplify analysis
  • Linear approximations work for small signals
  • Superposition applies only to linear systems
2

Matrix Methods for Circuits

Why Matrices? Real circuits have many nodes and meshes. Hand-solving becomes impractical. Matrix methods let computers solve circuits with thousands of nodes efficiently. Node vs Mesh Analysis Node...

⏱ 50 min read

Why Matrices?

Real circuits have many nodes and meshes. Hand-solving becomes impractical. Matrix methods let computers solve circuits with thousands of nodes efficiently.

Node vs Mesh Analysis

Node analysis: variables are node voltages. KCL at each node.

Mesh analysis: variables are loop currents. KVL around each mesh.

Node analysis is more general (works with any circuit including non-planar). Mesh analysis is simpler for planar circuits.

For a circuit with n nodes (excluding ground) and m meshes:

  • Node: n equations
  • Mesh: m equations

The Node Conductance Matrix

For a circuit with n nodes, the conductance matrix G is n × n:

G = [G11 G12 ... G1n;
G21 G22 ... G2n;
... ;
Gn1 Gn2 ... Gnn]

Diagonal Gii = sum of conductances connected to node i.

Off-diagonal Gij = -conductance between nodes i and j.

G is symmetric: Gij = Gji.

The system:

G × v = i

where v is the vector of node voltages and i is the vector of current sources.

Example: 3-Node Circuit

Nodes: 1, 2, 3. Ground is 0.

R12 between nodes 1 and 2
R13 between 1 and 3
R23 between 2 and 3
Current source I1 into node 1
Current source I3 into node 3

G matrix:

G11 = 1/R12 + 1/R13
G12 = -1/R12
G13 = -1/R13
G21 = -1/R12
G22 = 1/R12 + 1/R23
G23 = -1/R23
G31 = -1/R13
G32 = -1/R23
G33 = 1/R13 + 1/R23

Current vector:

i1 = I1
i2 = 0
i3 = I3

Solve G × v = i for v. Then currents are computed from Ohm's law.

Gaussian Elimination

The standard algorithm for solving G × v = i:

  1. Forward elimination: eliminate variables to get upper triangular form
  2. Back substitution: solve from last equation upward

Time: O(n³) for dense systems. Much faster for sparse (most circuit matrices are sparse).

LU Decomposition

Factor G = L × U where L is lower triangular and U is upper triangular.

Then solve L × y = i (easy forward substitution), then U × v = y (easy back substitution).

The factorization is done once and reused if only i changes. Useful when the circuit topology is fixed but sources vary.

Sparse Matrix Methods

Real circuits have very sparse G (most entries are zero). Sparse solvers store only the nonzero entries:

  • Compressed sparse row (CSR)
  • Compressed sparse column (CSC)

They use graph-based reordering (e.g., Markowitz, minimum degree, nested dissection) to reduce fill-in during elimination.

SPICE uses these techniques to handle millions of nodes.

Iterative Methods

For very large systems, direct methods are too slow. Iterative methods (Jacobi, Gauss-Seidel, conjugate gradient) converge to the solution gradually.

Used in:

  • Power grid simulation (millions of nodes)
  • Thermal simulation
  • 3D circuit extraction

Conjugate gradient is the workhorse for symmetric positive definite systems.

Applications Beyond Circuits

The same matrix methods apply to:

  • Structural analysis (FEA)
  • Fluid dynamics (CFD)
  • Heat transfer
  • Electromagnetics
  • Any physical system described by linear PDEs

Learn matrix methods for circuits, and you can apply them anywhere.

Key Takeaways

  • G × v = i is the master equation for circuits
  • Node conductance matrix is symmetric
  • Gaussian elimination is O(n³) for dense
  • LU decomposition reuses factorization
  • Sparse methods essential for large circuits
  • Iterative methods for very large systems
  • Same methods apply across engineering
3

Capacitors, Inductors, and Transients

Energy Storage Elements Unlike resistors (which dissipate energy), capacitors and inductors store energy: Capacitor: stores energy in electric field. I = C × dV/dt E = ½ × C × V² Inductor:...

⏱ 50 min read

Energy Storage Elements

Unlike resistors (which dissipate energy), capacitors and inductors store energy:

Capacitor: stores energy in electric field.

I = C × dV/dt
E = ½ × C × V²

Inductor: stores energy in magnetic field.

V = L × dI/dt
E = ½ × L × I²

RC Circuit — Charging

Given a resistor and capacitor in series with a DC voltage source:

Vin ──R──┬── Vcap
│
C
│
GND

The differential equation:

Vin = I × R + Vcap = RC × dVcap/dt + Vcap

Solution (charging from 0):

Vcap(t) = Vin × (1 - e^(-t/τ))

where τ = RC is the time constant.

After 1τ: 63% of final value.
After 5τ: 99.3% of final value (considered fully charged).

RC Circuit — Discharging

Starting from Vcap(0) = Vin, with the source removed:

Vcap(t) = Vin × e^(-t/τ)

The capacitor discharges to 37% after 1τ.

RL Circuit

Same math with inductor:

I(t) = (Vin/R) × (1 - e^(-t/τ))

where τ = L/R.

Step Response

The general form for a first-order system:

y(t) = y_final + (y_initial - y_final) × e^(-t/τ)

This applies to RC, RL, and any first-order linear system.

Second-Order Systems — RLC

Add an inductor to an RC circuit:

Vin ──R──L──┬── Vcap
│
C
│
GND

The equation:

LC × d²Vcap/dt² + RC × dVcap/dt + Vcap = Vin

Standard form:

d²V/dτ² + 2ζ × dV/dτ + V = Vin

where ζ (zeta) is the damping ratio, and ω_n = 1/√(LC).

Three regimes:

  • ζ > 1: overdamped (two real poles, slow response)
  • ζ = 1: critically damped (fastest without overshoot)
  • ζ < 1: underdamped (oscillates, decays)

Damping Ratio

ζ = (R/2) × √(C/L)

For ζ < 1, the natural frequency of oscillation:

ω_d = ω_n × √(1 - ζ²)

The period of oscillation:

T = 2π / ω_d

Applications

RC low-pass filter: smooths out high-frequency noise.

RLC resonant circuit: tuned circuits in radios (selective frequency).

Snubber circuits: RC across switches to absorb voltage spikes.

Timing circuits: 555 timer uses RC to generate delays.

Speaker crossover: LC filters separate high and low frequencies to different drivers.

Key Insights

  • Capacitors resist voltage changes
  • Inductors resist current changes
  • Time constant τ = RC or L/R
  • After 5τ, transient is essentially over
  • Second-order response depends on ζ
  • RLC oscillations decay at rate determined by R

Key Takeaways

  • Capacitors and inductors store energy
  • RC/RL time constant τ
  • First-order step response: exponential approach
  • Second-order RLC: depends on damping ratio ζ
  • Underdamped → oscillations, critically damped → fastest without overshoot
  • Applications everywhere in electronics
4

Fourier Analysis and Frequency Response

Why Fourier? Any periodic signal can be decomposed into a sum of sinusoids at harmonic frequencies. This lets us: Analyze any input as a sum of sine waves Apply linear system theory to each sine wave...

⏱ 50 min read

Why Fourier?

Any periodic signal can be decomposed into a sum of sinusoids at harmonic frequencies. This lets us:

  • Analyze any input as a sum of sine waves
  • Apply linear system theory to each sine wave
  • Reconstruct the output by summing

The Fourier transform is the most powerful tool in signal processing.

Fourier Series

For a periodic signal with period T:

f(t) = a₀/2 + Σ [a_n × cos(2πnt/T) + b_n × sin(2πnt/T)]

Coefficients:

a_n = (2/T) × ∫ f(t) cos(2πnt/T) dt
b_n = (2/T) × ∫ f(t) sin(2πnt/T) dt

The signal is decomposed into harmonics at 1/T, 2/T, 3/T, ...

Square Wave Example

A square wave of amplitude A and period T decomposes as:

f(t) = (4A/π) × [sin(ωt) + sin(3ωt)/3 + sin(5ωt)/5 + ...]

Only odd harmonics. Amplitudes decrease as 1/n. This is why a square wave sounds "buzzy" — it has lots of high harmonics.

Fourier Transform

For aperiodic signals, extend the period to infinity:

F(ω) = ∫ f(t) × e^(-jωt) dt

The inverse:

f(t) = (1/2π) × ∫ F(ω) × e^(jωt) dω

F(ω) is the spectrum — how much of each frequency is present.

Common Transform Pairs

| Time signal | Fourier transform |
|-------------|-------------------|
| δ(t) | 1 |
| u(t) (step) | 1/(jω) + πδ(ω) |
| e^(-at) u(t) | 1/(a + jω) |
| sin(ω₀t) | π[δ(ω - ω₀) - δ(ω + ω₀)]/j |
| Rect pulse | Sinc function |

Frequency Response

For a linear time-invariant system with impulse response h(t), the frequency response is H(ω) = Fourier transform of h(t).

If the input is x(t) with spectrum X(ω), the output spectrum is:

Y(ω) = H(ω) × X(ω)

The system multiplies each frequency component by H(ω). Amplitude and phase change but not frequency.

Bode Plots

Plot |H(ω)| in dB vs ω on log scale, and ∠H(ω) in degrees vs ω.

Key features:

  • Poles → -20 dB/decade per pole, -90° phase
  • Zeros → +20 dB/decade, +90° phase
  • Crossover where |H| = 0 dB (unity gain)

RC Low-Pass Filter

H(ω) = 1 / (1 + jωRC)

Magnitude: |H| = 1 / √(1 + (ωRC)²)

At ω = 1/RC: |H| = 1/√2 = -3 dB. This is the cutoff frequency.

Above cutoff: -20 dB/decade.

Phase: 0° at DC, -45° at cutoff, -90° at high frequencies.

RC High-Pass Filter

Swap R and C. Magnitude increases with frequency up to 1, then flat. Used for AC coupling.

Bandpass and Notch

Cascade low-pass and high-pass for bandpass.

Parallel LC forms a notch (rejects one frequency).

Applications

Audio: equalizers split signal into frequency bands.

Communications: filters select the desired channel.

Power: filters remove 50/60 Hz noise.

Imaging: 2D Fourier transform for image processing.

Vibration analysis: FFT to find resonances.

FFT — the Algorithm

The Discrete Fourier Transform (DFT) computes N frequency samples from N time samples in O(N²).

The Fast Fourier Transform (FFT) computes it in O(N log N). For N = 1,000,000: 10¹² vs 10⁷ operations — a 100,000x speedup.

FFT is one of the most important algorithms ever invented. It enables:

  • Real-time audio processing
  • Radar
  • Medical imaging (MRI, CT)
  • 5G communications
  • Astronomy (signal detection)

Key Takeaways

  • Fourier series: decompose periodic signals into harmonics
  • Fourier transform: extend to aperiodic signals
  • Frequency response H(ω) describes LTI system behavior
  • Bode plots visualize magnitude and phase
  • RC filters are first-order low/high-pass
  • FFT makes frequency analysis practical
  • Fourier analysis is universal in engineering
5

Information Theory and Coding

What is Information? Claude Shannon (1948) defined information as the reduction of uncertainty. If a message tells you something you already knew, it contains no information. Information is measured...

⏱ 50 min read

What is Information?

Claude Shannon (1948) defined information as the reduction of uncertainty. If a message tells you something you already knew, it contains no information.

Information is measured in bits. One bit is the information content of a yes/no answer when both are equally likely.

Shannon Entropy

For a discrete random variable X with probabilities p₁, p₂, ..., pn:

H(X) = -Σ p_i × log₂(p_i)

Units: bits.

Examples:

  • Fair coin: H = -0.5 log 0.5 - 0.5 log 0.5 = 1 bit
  • Biased coin (p=0.9): H = -0.9 log 0.9 - 0.1 log 0.1 ≈ 0.47 bits
  • Certain event (p=1): H = 0 bits

Higher entropy = more information per symbol.

Source Coding Theorem

Shannon's theorem: the minimum average number of bits per symbol needed to encode a source is H(X).

You cannot compress below the entropy. You can get arbitrarily close with the right code.

This is the foundation of data compression.

Huffman Coding

A simple optimal prefix code:

  1. Build a binary tree from symbol frequencies
  2. Assign shorter codes to frequent symbols
  3. Longer codes to rare symbols

Example:

  • A: 0.4, B: 0.3, C: 0.2, D: 0.1
  • Huffman: A → 0, B → 10, C → 110, D → 111
  • Average length: 0.4 + 0.6 + 0.6 + 0.3 = 1.9 bits/symbol
  • Entropy: 1.846 bits/symbol (close!)

Used in JPEG, DEFLATE, MP3, and many other formats.

Channel Capacity

A channel has noise, which limits how much information can be reliably transmitted.

Shannon's channel capacity for a channel with bandwidth B and signal-to-noise ratio SNR:

C = B × log₂(1 + SNR)

Units: bits per second.

Example: a phone line with B = 3 kHz and SNR = 30 dB (1000 linear):

C = 3000 × log₂(1001) ≈ 30 kbps

This is why early modems topped out at 33.6 kbps.

Error-Correcting Codes

To combat channel noise, add redundancy so errors can be detected and corrected.

Repetition code: send each bit 3 times, majority vote. Simple, but only 1/3 efficiency.

Parity bit: add one bit to detect single-bit errors. No correction.

Hamming codes: detect and correct single-bit errors. Hamming(7,4) uses 7 bits to send 4 data bits.

Reed-Solomon: corrects bursts of errors. Used in CDs, DVDs, QR codes, deep space probes.

LDPC and Turbo codes: near-Shannon-limit performance. Used in 4G/5G, Wi-Fi, DVB.

Polar codes: the first codes to achieve Shannon capacity. Used in 5G control channels.

The Shannon Limit

For any noisy channel, there's a maximum rate C below which error-free communication is possible (with the right code). Above C, errors are unavoidable.

This is a profound result. Before Shannon, engineers thought noise imposed fundamental limits. Shannon showed that with clever coding, you can approach zero error rate.

Applications

Compression: ZIP, JPEG, MP3, MP4 — all use Huffman, LZ, or arithmetic coding.

Storage: CDs, DVDs, hard drives — Reed-Solomon corrects physical defects.

Communication: 4G/5G, Wi-Fi, Bluetooth — LDPC, Turbo, Polar codes.

Space: deep space probes use powerful Reed-Solomon and convolutional codes.

Cryptography: information-theoretic security (one-time pad) requires key as long as message.

Key Takeaways

  • Information = reduction of uncertainty
  • Shannon entropy H(X) = -Σ p log p
  • Source coding theorem: minimum bits = entropy
  • Huffman codes are optimal prefix codes
  • Channel capacity C = B log₂(1 + SNR)
  • Error-correcting codes approach Shannon limit
  • Information theory is the foundation of all communication

📝 Problem Set 1: Linear Systems, Nodal Analysis

Q1. Use superposition to find V_out in a circuit with a 5V source through R1=1kΩ to node A, a 2mA current source into node A, and R2=2kΩ from node A to ground.

Show Solution

Source 1 (5V, kill current): V_A1 = 5 × 2k/(1k+2k) = 3.33V. Source 2 (2mA, kill 5V): V_A2 = 2mA × (1k || 2k) = 2mA × 667Ω = 1.33V. V_A = 3.33 + 1.33 = 4.67V.

Q2. Find the Thevenin equivalent of a circuit with a 12V source, R1=4kΩ in series, R2=6kΩ to ground, and load R_L at the R1-R2 junction.

Show Solution

V_th = 12 × 6/(4+6) = 7.2V. R_th = 4k || 6k = 2.4kΩ. Thevenin = 7.2V source + 2.4kΩ series resistor.

Q3. Design a voltage divider to produce 1.8V from 3.3V with current draw under 100 µA.

Show Solution

R_total ≥ 3.3V/100µA = 33kΩ. V_out/V_in = R2/(R1+R2) = 1.8/3.3 = 0.545. Choose R1 = 15kΩ, R2 = 18kΩ. Total = 33kΩ. Check: 3.3 × 18/33 = 1.8V ✓. Current = 100 µA ✓.

🎯 Midterm 1: Linear Systems, Thevenin, Transients

Q1. Design an RC low-pass filter with cutoff at 1 kHz. Choose R and C. What is the attenuation at 10 kHz?

Show Solution

f_c = 1/(2πRC). Choose C = 100 nF. R = 1/(2π × 1000 × 100e-9) = 1.59 kΩ. At 10 kHz, |H| = 1/√(1+(10)²) = 0.0995 → -20 dB. 20 dB/decade roll-off.

Q2. Find the impulse response of an LTI system described by y[n] = 0.5y[n-1] + x[n].

Show Solution

h[0] = 1. h[1] = 0.5. h[2] = 0.25. General: h[n] = (0.5)^n u[n]. This is an IIR low-pass filter.

Q3. A signal x(t) = 3cos(2π1000t) + 2cos(2π3000t) is sampled at 8 kHz. What frequencies appear in the spectrum?

Show Solution

Sampling at 8 kHz → Nyquist = 4 kHz. Both frequencies are below 4 kHz → no aliasing. Spectrum: peaks at 1 kHz (amp 3) and 3 kHz (amp 2). If sampling at 5 kHz → 3 kHz would alias to 2 kHz.