Transformers & Large Language Model Internals

Tokenization, attention mechanics, RoPE, KV cache, and decoding algorithms.

3 weeks · wk 5–7Core / deep pace6 topics18 Interview Questions

Why this pace:To design low-latency AI backends, configure context windows, and optimize GPU memory, you must master the exact mechanics of token generation and KV cache expansion.

SEC FOCUS

KV cache consumption grows linearly with batch size and sequence length; without paging or quantization, serving spikes cause sudden GPU OOM crashes.

Topics & Action Items (6)

Tokenization: BPE, Tiktoken, SentencePiece & Token Tax

4 hrs

Self-Attention & Scaled Dot-Product Math

6 hrs

Multi-Query Attention (MQA) & Grouped-Query Attention (GQA)

4 hrs

Rotary Position Embedding (RoPE) & Context Extension

5 hrs

The KV Cache & Autoregressive Decoding Dynamics

5 hrs

Decoding Algorithms: Greedy, Nucleus (Top-p), Temperature & Min-P

4 hrs

Previous Stage

Deep Learning & Neural Foundation Mechanics

Next Stage

Prompt Engineering, Structured Outputs & Tool Use