Designing Information Devices
Linear algebra, circuit analysis, and system design for engineers. Foundational for embedded systems.
Linear Systems and Circuit Basics
Why Linear Systems? Linear systems are the foundation of all engineering analysis. If a system is linear, superposition applies: the response to a sum of inputs is the sum of the individual...
Why Linear Systems?
Linear systems are the foundation of all engineering analysis. If a system is linear, superposition applies: the response to a sum of inputs is the sum of the individual responses.
Most real systems are approximately linear over some range. Even nonlinear systems are often analyzed by linearizing around an operating point.
Linearity Definition
A system S is linear if for all inputs x₁, x₂ and scalars a, b:
S(a x₁ + b x₂) = a S(x₁) + b S(x₂)
This is the superposition principle.
Linearity implies:
- Homogeneity: S(ax) = a S(x)
- Additivity: S(x₁ + x₂) = S(x₁) + S(x₂)
- Zero input → zero output (for linear systems with no offset)
Circuit Elements as Linear Systems
Resistor: V = I × R. Linear.
Capacitor: I = C × dV/dt. Linear (differential operator).
Inductor: V = L × dI/dt. Linear.
Op-amp (ideal): Vout = A × (V+ - V-). Linear.
Diode: I = Is × (e^(V/Vt) - 1). Nonlinear.
Nodal Analysis
The standard method for solving circuits:
- Choose a reference node (ground)
- Assign node voltages to all other nodes
- Write KCL at each node: sum of currents leaving = 0
- Solve the system of equations
For n-1 unknown nodes, we get n-1 equations. Express in matrix form: G × v = i.
Example: Two-Node Circuit
Vin ──R1──┬──R2── GND
│
R3
│
GND
KCL at node 1:
(V1 - Vin)/R1 + V1/R2 + V1/R3 = 0
V1 × (1/R1 + 1/R2 + 1/R3) = Vin/R1
V1 = Vin/R1 / (1/R1 + 1/R2 + 1/R3)
The circuit is a voltage divider (R2 || R3 in series with R1).
Matrix Form
For larger circuits, the conductance matrix G is symmetric, which allows efficient solution.
G = [g11 g12 ...; g21 g22 ...; ...]
where gii = sum of conductances at node i, and gij = -conductance between i and j (i ≠ j).
Modern circuit simulators (SPICE) solve circuits this way.
Thevenin and Norton Equivalents
Any linear two-terminal network can be replaced by:
Thevenin: a voltage source V_th in series with a resistor R_th.
Norton: a current source I_n in parallel with a resistor R_n.
Conversion: V_th = I_n × R_n, R_th = R_n.
Finding Thevenin:
- Compute open-circuit voltage V_oc = V_th
- Zero out sources (short V sources, open I sources)
- Compute resistance between terminals = R_th
This is fundamental to circuit design.
Superposition Example
Given a circuit with two sources, compute the response by:
- Turn off source 1, compute response to source 2
- Turn off source 2, compute response to source 1
- Add the two responses
This is often easier than solving the full system at once.
Key Takeaways
- Linearity = superposition principle
- Nodal analysis is the standard method
- G × v = i matrix form for larger circuits
- Thevenin and Norton equivalents simplify analysis
- Linear approximations work for small signals
- Superposition applies only to linear systems
Matrix Methods for Circuits
Why Matrices? Real circuits have many nodes and meshes. Hand-solving becomes impractical. Matrix methods let computers solve circuits with thousands of nodes efficiently. Node vs Mesh Analysis Node...
Why Matrices?
Real circuits have many nodes and meshes. Hand-solving becomes impractical. Matrix methods let computers solve circuits with thousands of nodes efficiently.
Node vs Mesh Analysis
Node analysis: variables are node voltages. KCL at each node.
Mesh analysis: variables are loop currents. KVL around each mesh.
Node analysis is more general (works with any circuit including non-planar). Mesh analysis is simpler for planar circuits.
For a circuit with n nodes (excluding ground) and m meshes:
- Node: n equations
- Mesh: m equations
The Node Conductance Matrix
For a circuit with n nodes, the conductance matrix G is n × n:
G = [G11 G12 ... G1n;
G21 G22 ... G2n;
... ;
Gn1 Gn2 ... Gnn]
Diagonal Gii = sum of conductances connected to node i.
Off-diagonal Gij = -conductance between nodes i and j.
G is symmetric: Gij = Gji.
The system:
G × v = i
where v is the vector of node voltages and i is the vector of current sources.
Example: 3-Node Circuit
Nodes: 1, 2, 3. Ground is 0.
R12 between nodes 1 and 2
R13 between 1 and 3
R23 between 2 and 3
Current source I1 into node 1
Current source I3 into node 3
G matrix:
G11 = 1/R12 + 1/R13
G12 = -1/R12
G13 = -1/R13
G21 = -1/R12
G22 = 1/R12 + 1/R23
G23 = -1/R23
G31 = -1/R13
G32 = -1/R23
G33 = 1/R13 + 1/R23
Current vector:
i1 = I1
i2 = 0
i3 = I3
Solve G × v = i for v. Then currents are computed from Ohm's law.
Gaussian Elimination
The standard algorithm for solving G × v = i:
- Forward elimination: eliminate variables to get upper triangular form
- Back substitution: solve from last equation upward
Time: O(n³) for dense systems. Much faster for sparse (most circuit matrices are sparse).
LU Decomposition
Factor G = L × U where L is lower triangular and U is upper triangular.
Then solve L × y = i (easy forward substitution), then U × v = y (easy back substitution).
The factorization is done once and reused if only i changes. Useful when the circuit topology is fixed but sources vary.
Sparse Matrix Methods
Real circuits have very sparse G (most entries are zero). Sparse solvers store only the nonzero entries:
- Compressed sparse row (CSR)
- Compressed sparse column (CSC)
They use graph-based reordering (e.g., Markowitz, minimum degree, nested dissection) to reduce fill-in during elimination.
SPICE uses these techniques to handle millions of nodes.
Iterative Methods
For very large systems, direct methods are too slow. Iterative methods (Jacobi, Gauss-Seidel, conjugate gradient) converge to the solution gradually.
Used in:
- Power grid simulation (millions of nodes)
- Thermal simulation
- 3D circuit extraction
Conjugate gradient is the workhorse for symmetric positive definite systems.
Applications Beyond Circuits
The same matrix methods apply to:
- Structural analysis (FEA)
- Fluid dynamics (CFD)
- Heat transfer
- Electromagnetics
- Any physical system described by linear PDEs
Learn matrix methods for circuits, and you can apply them anywhere.
Key Takeaways
- G × v = i is the master equation for circuits
- Node conductance matrix is symmetric
- Gaussian elimination is O(n³) for dense
- LU decomposition reuses factorization
- Sparse methods essential for large circuits
- Iterative methods for very large systems
- Same methods apply across engineering
Capacitors, Inductors, and Transients
Energy Storage Elements Unlike resistors (which dissipate energy), capacitors and inductors store energy: Capacitor: stores energy in electric field. I = C × dV/dt E = ½ × C × V² Inductor:...
Energy Storage Elements
Unlike resistors (which dissipate energy), capacitors and inductors store energy:
Capacitor: stores energy in electric field.
I = C × dV/dt
E = ½ × C × V²
Inductor: stores energy in magnetic field.
V = L × dI/dt
E = ½ × L × I²
RC Circuit — Charging
Given a resistor and capacitor in series with a DC voltage source:
Vin ──R──┬── Vcap
│
C
│
GND
The differential equation:
Vin = I × R + Vcap = RC × dVcap/dt + Vcap
Solution (charging from 0):
Vcap(t) = Vin × (1 - e^(-t/τ))
where τ = RC is the time constant.
After 1τ: 63% of final value.
After 5τ: 99.3% of final value (considered fully charged).
RC Circuit — Discharging
Starting from Vcap(0) = Vin, with the source removed:
Vcap(t) = Vin × e^(-t/τ)
The capacitor discharges to 37% after 1τ.
RL Circuit
Same math with inductor:
I(t) = (Vin/R) × (1 - e^(-t/τ))
where τ = L/R.
Step Response
The general form for a first-order system:
y(t) = y_final + (y_initial - y_final) × e^(-t/τ)
This applies to RC, RL, and any first-order linear system.
Second-Order Systems — RLC
Add an inductor to an RC circuit:
Vin ──R──L──┬── Vcap
│
C
│
GND
The equation:
LC × d²Vcap/dt² + RC × dVcap/dt + Vcap = Vin
Standard form:
d²V/dτ² + 2ζ × dV/dτ + V = Vin
where ζ (zeta) is the damping ratio, and ω_n = 1/√(LC).
Three regimes:
- ζ > 1: overdamped (two real poles, slow response)
- ζ = 1: critically damped (fastest without overshoot)
- ζ < 1: underdamped (oscillates, decays)
Damping Ratio
ζ = (R/2) × √(C/L)
For ζ < 1, the natural frequency of oscillation:
ω_d = ω_n × √(1 - ζ²)
The period of oscillation:
T = 2π / ω_d
Applications
RC low-pass filter: smooths out high-frequency noise.
RLC resonant circuit: tuned circuits in radios (selective frequency).
Snubber circuits: RC across switches to absorb voltage spikes.
Timing circuits: 555 timer uses RC to generate delays.
Speaker crossover: LC filters separate high and low frequencies to different drivers.
Key Insights
- Capacitors resist voltage changes
- Inductors resist current changes
- Time constant τ = RC or L/R
- After 5τ, transient is essentially over
- Second-order response depends on ζ
- RLC oscillations decay at rate determined by R
Key Takeaways
- Capacitors and inductors store energy
- RC/RL time constant τ
- First-order step response: exponential approach
- Second-order RLC: depends on damping ratio ζ
- Underdamped → oscillations, critically damped → fastest without overshoot
- Applications everywhere in electronics
Fourier Analysis and Frequency Response
Why Fourier? Any periodic signal can be decomposed into a sum of sinusoids at harmonic frequencies. This lets us: Analyze any input as a sum of sine waves Apply linear system theory to each sine wave...
Why Fourier?
Any periodic signal can be decomposed into a sum of sinusoids at harmonic frequencies. This lets us:
- Analyze any input as a sum of sine waves
- Apply linear system theory to each sine wave
- Reconstruct the output by summing
The Fourier transform is the most powerful tool in signal processing.
Fourier Series
For a periodic signal with period T:
f(t) = a₀/2 + Σ [a_n × cos(2πnt/T) + b_n × sin(2πnt/T)]
Coefficients:
a_n = (2/T) × ∫ f(t) cos(2πnt/T) dt
b_n = (2/T) × ∫ f(t) sin(2πnt/T) dt
The signal is decomposed into harmonics at 1/T, 2/T, 3/T, ...
Square Wave Example
A square wave of amplitude A and period T decomposes as:
f(t) = (4A/π) × [sin(ωt) + sin(3ωt)/3 + sin(5ωt)/5 + ...]
Only odd harmonics. Amplitudes decrease as 1/n. This is why a square wave sounds "buzzy" — it has lots of high harmonics.
Fourier Transform
For aperiodic signals, extend the period to infinity:
F(ω) = ∫ f(t) × e^(-jωt) dt
The inverse:
f(t) = (1/2π) × ∫ F(ω) × e^(jωt) dω
F(ω) is the spectrum — how much of each frequency is present.
Common Transform Pairs
| Time signal | Fourier transform |
|-------------|-------------------|
| δ(t) | 1 |
| u(t) (step) | 1/(jω) + πδ(ω) |
| e^(-at) u(t) | 1/(a + jω) |
| sin(ω₀t) | π[δ(ω - ω₀) - δ(ω + ω₀)]/j |
| Rect pulse | Sinc function |
Frequency Response
For a linear time-invariant system with impulse response h(t), the frequency response is H(ω) = Fourier transform of h(t).
If the input is x(t) with spectrum X(ω), the output spectrum is:
Y(ω) = H(ω) × X(ω)
The system multiplies each frequency component by H(ω). Amplitude and phase change but not frequency.
Bode Plots
Plot |H(ω)| in dB vs ω on log scale, and ∠H(ω) in degrees vs ω.
Key features:
- Poles → -20 dB/decade per pole, -90° phase
- Zeros → +20 dB/decade, +90° phase
- Crossover where |H| = 0 dB (unity gain)
RC Low-Pass Filter
H(ω) = 1 / (1 + jωRC)
Magnitude: |H| = 1 / √(1 + (ωRC)²)
At ω = 1/RC: |H| = 1/√2 = -3 dB. This is the cutoff frequency.
Above cutoff: -20 dB/decade.
Phase: 0° at DC, -45° at cutoff, -90° at high frequencies.
RC High-Pass Filter
Swap R and C. Magnitude increases with frequency up to 1, then flat. Used for AC coupling.
Bandpass and Notch
Cascade low-pass and high-pass for bandpass.
Parallel LC forms a notch (rejects one frequency).
Applications
Audio: equalizers split signal into frequency bands.
Communications: filters select the desired channel.
Power: filters remove 50/60 Hz noise.
Imaging: 2D Fourier transform for image processing.
Vibration analysis: FFT to find resonances.
FFT — the Algorithm
The Discrete Fourier Transform (DFT) computes N frequency samples from N time samples in O(N²).
The Fast Fourier Transform (FFT) computes it in O(N log N). For N = 1,000,000: 10¹² vs 10⁷ operations — a 100,000x speedup.
FFT is one of the most important algorithms ever invented. It enables:
- Real-time audio processing
- Radar
- Medical imaging (MRI, CT)
- 5G communications
- Astronomy (signal detection)
Key Takeaways
- Fourier series: decompose periodic signals into harmonics
- Fourier transform: extend to aperiodic signals
- Frequency response H(ω) describes LTI system behavior
- Bode plots visualize magnitude and phase
- RC filters are first-order low/high-pass
- FFT makes frequency analysis practical
- Fourier analysis is universal in engineering
Information Theory and Coding
What is Information? Claude Shannon (1948) defined information as the reduction of uncertainty. If a message tells you something you already knew, it contains no information. Information is measured...
What is Information?
Claude Shannon (1948) defined information as the reduction of uncertainty. If a message tells you something you already knew, it contains no information.
Information is measured in bits. One bit is the information content of a yes/no answer when both are equally likely.
Shannon Entropy
For a discrete random variable X with probabilities p₁, p₂, ..., pn:
H(X) = -Σ p_i × log₂(p_i)
Units: bits.
Examples:
- Fair coin: H = -0.5 log 0.5 - 0.5 log 0.5 = 1 bit
- Biased coin (p=0.9): H = -0.9 log 0.9 - 0.1 log 0.1 ≈ 0.47 bits
- Certain event (p=1): H = 0 bits
Higher entropy = more information per symbol.
Source Coding Theorem
Shannon's theorem: the minimum average number of bits per symbol needed to encode a source is H(X).
You cannot compress below the entropy. You can get arbitrarily close with the right code.
This is the foundation of data compression.
Huffman Coding
A simple optimal prefix code:
- Build a binary tree from symbol frequencies
- Assign shorter codes to frequent symbols
- Longer codes to rare symbols
Example:
- A: 0.4, B: 0.3, C: 0.2, D: 0.1
- Huffman: A → 0, B → 10, C → 110, D → 111
- Average length: 0.4 + 0.6 + 0.6 + 0.3 = 1.9 bits/symbol
- Entropy: 1.846 bits/symbol (close!)
Used in JPEG, DEFLATE, MP3, and many other formats.
Channel Capacity
A channel has noise, which limits how much information can be reliably transmitted.
Shannon's channel capacity for a channel with bandwidth B and signal-to-noise ratio SNR:
C = B × log₂(1 + SNR)
Units: bits per second.
Example: a phone line with B = 3 kHz and SNR = 30 dB (1000 linear):
C = 3000 × log₂(1001) ≈ 30 kbps
This is why early modems topped out at 33.6 kbps.
Error-Correcting Codes
To combat channel noise, add redundancy so errors can be detected and corrected.
Repetition code: send each bit 3 times, majority vote. Simple, but only 1/3 efficiency.
Parity bit: add one bit to detect single-bit errors. No correction.
Hamming codes: detect and correct single-bit errors. Hamming(7,4) uses 7 bits to send 4 data bits.
Reed-Solomon: corrects bursts of errors. Used in CDs, DVDs, QR codes, deep space probes.
LDPC and Turbo codes: near-Shannon-limit performance. Used in 4G/5G, Wi-Fi, DVB.
Polar codes: the first codes to achieve Shannon capacity. Used in 5G control channels.
The Shannon Limit
For any noisy channel, there's a maximum rate C below which error-free communication is possible (with the right code). Above C, errors are unavoidable.
This is a profound result. Before Shannon, engineers thought noise imposed fundamental limits. Shannon showed that with clever coding, you can approach zero error rate.
Applications
Compression: ZIP, JPEG, MP3, MP4 — all use Huffman, LZ, or arithmetic coding.
Storage: CDs, DVDs, hard drives — Reed-Solomon corrects physical defects.
Communication: 4G/5G, Wi-Fi, Bluetooth — LDPC, Turbo, Polar codes.
Space: deep space probes use powerful Reed-Solomon and convolutional codes.
Cryptography: information-theoretic security (one-time pad) requires key as long as message.
Key Takeaways
- Information = reduction of uncertainty
- Shannon entropy H(X) = -Σ p log p
- Source coding theorem: minimum bits = entropy
- Huffman codes are optimal prefix codes
- Channel capacity C = B log₂(1 + SNR)
- Error-correcting codes approach Shannon limit
- Information theory is the foundation of all communication
📝 Problem Set 1: Linear Systems, Nodal Analysis
Q1. Use superposition to find V_out in a circuit with a 5V source through R1=1kΩ to node A, a 2mA current source into node A, and R2=2kΩ from node A to ground.
Show Solution
Source 1 (5V, kill current): V_A1 = 5 × 2k/(1k+2k) = 3.33V. Source 2 (2mA, kill 5V): V_A2 = 2mA × (1k || 2k) = 2mA × 667Ω = 1.33V. V_A = 3.33 + 1.33 = 4.67V.
Q2. Find the Thevenin equivalent of a circuit with a 12V source, R1=4kΩ in series, R2=6kΩ to ground, and load R_L at the R1-R2 junction.
Show Solution
V_th = 12 × 6/(4+6) = 7.2V. R_th = 4k || 6k = 2.4kΩ. Thevenin = 7.2V source + 2.4kΩ series resistor.
Q3. Design a voltage divider to produce 1.8V from 3.3V with current draw under 100 µA.
Show Solution
R_total ≥ 3.3V/100µA = 33kΩ. V_out/V_in = R2/(R1+R2) = 1.8/3.3 = 0.545. Choose R1 = 15kΩ, R2 = 18kΩ. Total = 33kΩ. Check: 3.3 × 18/33 = 1.8V ✓. Current = 100 µA ✓.
🎯 Midterm 1: Linear Systems, Thevenin, Transients
Q1. Design an RC low-pass filter with cutoff at 1 kHz. Choose R and C. What is the attenuation at 10 kHz?
Show Solution
f_c = 1/(2πRC). Choose C = 100 nF. R = 1/(2π × 1000 × 100e-9) = 1.59 kΩ. At 10 kHz, |H| = 1/√(1+(10)²) = 0.0995 → -20 dB. 20 dB/decade roll-off.
Q2. Find the impulse response of an LTI system described by y[n] = 0.5y[n-1] + x[n].
Show Solution
h[0] = 1. h[1] = 0.5. h[2] = 0.25. General: h[n] = (0.5)^n u[n]. This is an IIR low-pass filter.
Q3. A signal x(t) = 3cos(2π1000t) + 2cos(2π3000t) is sampled at 8 kHz. What frequencies appear in the spectrum?
Show Solution
Sampling at 8 kHz → Nyquist = 4 kHz. Both frequencies are below 4 kHz → no aliasing. Spectrum: peaks at 1 kHz (amp 3) and 3 kHz (amp 2). If sampling at 5 kHz → 3 kHz would alias to 2 kHz.