[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-1647":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":8,"languages":8,"totalLinesOfCode":8,"stars":9,"forks":10,"watchers":11,"openIssues":12,"contributorsCount":12,"subscribersCount":12,"size":12,"stars1d":13,"stars7d":13,"stars30d":14,"stars90d":12,"forks30d":12,"starsTrendScore":15,"compositeScore":16,"rankGlobal":8,"rankLanguage":8,"license":8,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":19,"hasPages":17,"topics":20,"createdAt":8,"pushedAt":8,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":12,"starSnapshotCount":12,"syncStatus":13,"lastSyncTime":24,"discoverSource":25},1647,"LLM-Math-Handbook","h9-tec\u002FLLM-Math-Handbook","h9-tec",null,221,28,3,0,2,25,6,4.39,false,"main",true,[],"2026-06-12 02:00:31","# Equation Bible for Production LLMs, RAG, and AI Agents\n\n**Author:** Hesham Haroon\n\n> This README is a GitHub-ready mathematical reference for Large Language Models, Retrieval-Augmented Generation, and AI Agents. It is designed as a production study guide and interview reference. It prioritizes breadth. It includes foundational, common, derivative, production, and less frequently cited equations. No finite document can literally contain every equation ever used in the literature, but this covers the major equation families required to reason about modern LLM, RAG, and agent systems.\n\n## Table of Contents\n- [Notation](#notation)\n- [Part I: Large Language Models](#part-i-large-language-models)\n  - [1. Probabilistic Foundations](#1-probabilistic-foundations)\n  - [2. Embeddings and Transformer Blocks](#2-embeddings-and-transformer-blocks)\n  - [3. Attention Variants](#3-attention-variants)\n  - [4. Positional Encoding and Long Context](#4-positional-encoding-and-long-context)\n  - [5. Normalization](#5-normalization)\n  - [6. Activations and Gated MLPs](#6-activations-and-gated-mlps)\n  - [7. Training Losses](#7-training-losses)\n  - [8. Optimization](#8-optimization)\n  - [9. Sampling and Decoding](#9-sampling-and-decoding)\n  - [10. Alignment and Preference Optimization](#10-alignment-and-preference-optimization)\n  - [11. PEFT and Adapter Methods](#11-peft-and-adapter-methods)\n  - [12. Quantization](#12-quantization)\n  - [13. Scaling Laws](#13-scaling-laws)\n  - [14. Mixture of Experts](#14-mixture-of-experts)\n  - [15. Tokenization and Embedding Pretraining](#15-tokenization-and-embedding-pretraining)\n  - [16. Evaluation Metrics](#16-evaluation-metrics)\n  - [17. Production Inference Math](#17-production-inference-math)\n  - [18. Non-Transformer Sequence Models](#18-non-transformer-sequence-models)\n  - [19. Calibration and Uncertainty](#19-calibration-and-uncertainty)\n- [Part II: Retrieval-Augmented Generation](#part-ii-retrieval-augmented-generation)\n  - [20. Similarity and Distance Metrics](#20-similarity-and-distance-metrics)\n  - [21. Sparse Retrieval](#21-sparse-retrieval)\n  - [22. Dense Retrieval](#22-dense-retrieval)\n  - [23. Hybrid Search and Fusion](#23-hybrid-search-and-fusion)\n  - [24. ANN and Vector Index Math](#24-ann-and-vector-index-math)\n  - [25. Reranking and Learning to Rank](#25-reranking-and-learning-to-rank)\n  - [26. Retrieval Metrics](#26-retrieval-metrics)\n  - [27. Chunking and Context Packing](#27-chunking-and-context-packing)\n  - [28. Diversity and Redundancy Control](#28-diversity-and-redundancy-control)\n  - [29. RAG Objectives and RAGAS-Style Metrics](#29-rag-objectives-and-ragas-style-metrics)\n  - [30. Query Expansion and Feedback](#30-query-expansion-and-feedback)\n  - [31. GraphRAG](#31-graphrag)\n- [Part III: AI Agents](#part-iii-ai-agents)\n  - [32. MDP and POMDP Foundations](#32-mdp-and-pomdp-foundations)\n  - [33. Temporal-Difference and Value-Based RL](#33-temporal-difference-and-value-based-rl)\n  - [34. Policy Gradient and Actor-Critic](#34-policy-gradient-and-actor-critic)\n  - [35. Advantage Estimation](#35-advantage-estimation)\n  - [36. Exploration and Bandits](#36-exploration-and-bandits)\n  - [37. Planning and Tree Search](#37-planning-and-tree-search)\n  - [38. Offline RL and Imitation](#38-offline-rl-and-imitation)\n  - [39. Multi-Agent RL](#39-multi-agent-rl)\n  - [40. LLM-Agent Specific Math](#40-llm-agent-specific-math)\n  - [41. Reward Modeling](#41-reward-modeling)\n  - [42. Memory and Context](#42-memory-and-context)\n  - [43. Risk, Safety, and Constrained Agents](#43-risk-safety-and-constrained-agents)\n- [Important Python\u002FPyTorch Snippets](#important-pythonpytorch-snippets)\n- [References](#references)\n\n## Notation\n\n- `x_{1:T}`: token sequence of length `T`.\n- `V`: vocabulary size.\n- `d`: model dimension.\n- `d_h`: attention head dimension.\n- `H`: number of attention heads.\n- `L`: number of Transformer layers.\n- `Q,K,V`: query, key, value matrices.\n- `\\theta`: model parameters.\n- `\\pi_\\theta`: policy induced by an LLM or RL agent.\n- `D`: document corpus.\n- `q`: query.\n- `d_i`: document or chunk.\n- `s_t,a_t,r_t`: state, action, reward at time `t`.\n- `\\gamma`: reward discount factor.\n- `\\epsilon`: small numerical constant.\n\n# Part I: Large Language Models\n\n## 1. Probabilistic Foundations\n\n#### Autoregressive factorization\n\n**Equation**\n\n$$\np_\\theta(x_{1:T}) = \\prod_{t=1}^{T} p_\\theta(x_t | x_{1:t-1})\n$$\n\n**Plain-English explanation.** A language model assigns probability to a full sequence by multiplying conditional next-token probabilities.\n\n**Symbols.** `x_t` is token `t`; `x_{1:t-1}` are earlier tokens; `\\theta` are model parameters.\n\n**Practical use case.** Defines causal language modeling, pretraining, scoring, perplexity, and decoding.\n\n**Source.** Bengio et al. (2003); Mikolov et al. (2010); Radford et al. (2018).\n\n#### Conditional generation factorization\n\n**Equation**\n\n$$\np_\\theta(y_{1:T}| x)=\\prod_{t=1}^{T}p_\\theta(y_t| x,y_{1:t-1})\n$$\n\n**Plain-English explanation.** The model generates an output sequence conditioned on an input prompt or source sequence.\n\n**Symbols.** `x` is prompt\u002Fcontext; `y_t` is generated token.\n\n**Practical use case.** Instruction following, translation, summarization, tool-call generation.\n\n**Source.** Sutskever et al. (2014); Vaswani et al. (2017).\n\n#### Softmax over logits\n\n**Equation**\n\n$$\np_i = softmax(z)_i = \\frac{e^{z_i}}{\\sum_{j=1}^{V}e^{z_j}}\n$$\n\n**Plain-English explanation.** Converts raw logits into a probability distribution over vocabulary tokens.\n\n**Symbols.** `z_i` is logit for token `i`; `V` is vocabulary size.\n\n**Practical use case.** The final step before sampling or loss computation.\n\n**Source.** Bridle (1990); Goodfellow, Bengio, Courville (2016).\n\n#### Log-softmax\n\n**Equation**\n\n$$\n\\log p_i = z_i - \\log\\sum_{j=1}^{V} e^{z_j}\n$$\n\n**Plain-English explanation.** Computes log-probabilities in a numerically stable way.\n\n**Symbols.** `z_i` is a logit; `p_i` is normalized probability.\n\n**Practical use case.** Stable cross-entropy and beam search scoring.\n\n**Source.** Goodfellow, Bengio, Courville (2016).\n\n#### Log-sum-exp trick\n\n**Equation**\n\n$$\n\\log\\sum_i e^{z_i}=m+\\log\\sum_i e^{z_i-m},\\; m=\\max_i z_i\n$$\n\n**Plain-English explanation.** Prevents overflow when exponentiating large logits.\n\n**Symbols.** `m` is the maximum logit.\n\n**Practical use case.** Stable inference and training kernels.\n\n**Source.** Numerical optimization folklore; Bishop (2006).\n\n#### Negative log-likelihood\n\n**Equation**\n\n$$\nL_{NLL}(\\theta)=-\\sum_{t=1}^{T}\\log p_\\theta(x_t| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Penalizes the model when it assigns low probability to the correct next token.\n\n**Symbols.** `T` is sequence length; `p_\\theta` is the predicted token probability.\n\n**Practical use case.** Standard causal LM training objective.\n\n**Source.** Bengio et al. (2003); Radford et al. (2018).\n\n#### Cross-entropy\n\n**Equation**\n\n$$\nH(y,p)=-\\sum_{i=1}^{V} y_i\\log p_i\n$$\n\n**Plain-English explanation.** Measures mismatch between the target distribution and predicted distribution.\n\n**Symbols.** `y_i` is target probability; `p_i` is predicted probability.\n\n**Practical use case.** Training classifiers and next-token predictors.\n\n**Source.** Shannon (1948); Goodfellow, Bengio, Courville (2016).\n\n#### Cross-entropy decomposition\n\n**Equation**\n\n$$\nH(P,Q)=H(P)+D_{KL}(P|| Q)\n$$\n\n**Plain-English explanation.** Cross-entropy equals irreducible target entropy plus KL divergence from target to model.\n\n**Symbols.** `P` is data distribution; `Q` is model distribution.\n\n**Practical use case.** Explains why minimizing cross-entropy minimizes KL to the data distribution.\n\n**Source.** Cover and Thomas (1991).\n\n#### KL divergence\n\n**Equation**\n\n$$\nD_{KL}(P|| Q)=\\sum_x P(x)\\log\\frac{P(x)}{Q(x)}\n$$\n\n**Plain-English explanation.** Measures how much one probability distribution differs from another.\n\n**Symbols.** `P,Q` are distributions over the same support.\n\n**Practical use case.** RLHF KL penalties, distillation, variational inference, calibration.\n\n**Source.** Kullback and Leibler (1951).\n\n#### Reverse KL divergence\n\n**Equation**\n\n$$\nD_{KL}(Q|| P)=\\sum_x Q(x)\\log\\frac{Q(x)}{P(x)}\n$$\n\n**Plain-English explanation.** A mode-seeking divergence often behaving differently from forward KL.\n\n**Symbols.** `P` is reference\u002Ftarget distribution; `Q` is approximating distribution.\n\n**Practical use case.** Preference optimization, variational inference, policy regularization analysis.\n\n**Source.** Kullback and Leibler (1951).\n\n#### Entropy\n\n**Equation**\n\n$$\nH(P)=-\\sum_x P(x)\\log P(x)\n$$\n\n**Plain-English explanation.** Measures uncertainty or randomness in a distribution.\n\n**Symbols.** `P(x)` is probability of event `x`.\n\n**Practical use case.** Sampling diversity, RL exploration, calibration, active learning.\n\n**Source.** Shannon (1948).\n\n#### Sequence log-likelihood\n\n**Equation**\n\n$$\n\\log p_\\theta(x_{1:T})=\\sum_{t=1}^{T}\\log p_\\theta(x_t| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Turns a product of probabilities into a sum of token log-probabilities.\n\n**Symbols.** `x_{1:T}` is a token sequence.\n\n**Practical use case.** Scoring completions, preference data, reranking generations.\n\n**Source.** Bengio et al. (2003).\n\n#### Length-normalized score\n\n**Equation**\n\n$$\nS(x_{1:T})=\\frac{1}{T^\\alpha}\\sum_{t=1}^{T}\\log p_\\theta(x_t| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Normalizes sequence likelihood to reduce short-sequence bias.\n\n**Symbols.** `\\alpha` controls strength of length penalty.\n\n**Practical use case.** Beam search, response reranking, translation decoding.\n\n**Source.** Wu et al. (2016).\n\n#### Teacher forcing objective\n\n**Equation**\n\n$$\nL_{TF}=-\\sum_t \\log p_\\theta(y_t| y_{1:t-1}^{\\star},x)\n$$\n\n**Plain-English explanation.** The model predicts each next target token using gold previous tokens during training.\n\n**Symbols.** `y_{1:t-1}^{star}` are ground-truth previous tokens.\n\n**Practical use case.** Seq2seq training and instruction tuning.\n\n**Source.** Williams and Zipser (1989); Sutskever et al. (2014).\n\n#### Perplexity\n\n**Equation**\n\n$$\nPPL=\\exp(-\\frac{1}{T}\\sum_{t=1}^{T}\\log p_\\theta(x_t| x_{1:t-1}))\n$$\n\n**Plain-English explanation.** Exponentiated average negative log-likelihood; lower is better.\n\n**Symbols.** `T` is token count.\n\n**Practical use case.** Language model evaluation.\n\n**Source.** Jelinek et al. (1977); Bengio et al. (2003).\n\n#### Bits per byte\n\n**Equation**\n\n$$\nBPB= -\\frac{1}{B\\log 2}\\sum_{t=1}^{T}\\log p_\\theta(x_t| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Measures compression quality normalized by byte count.\n\n**Symbols.** `B` is number of bytes in the text.\n\n**Practical use case.** Byte-level model comparison across tokenizers.\n\n**Source.** Shannon (1948); Graves (2013).\n\n## 2. Embeddings and Transformer Blocks\n\n#### Token embedding lookup\n\n**Equation**\n\n$$\nX = E[ids],\\; E\\in R^{V\\times d}\n$$\n\n**Plain-English explanation.** Maps token IDs to dense vectors.\n\n**Symbols.** `E` is embedding matrix; `V` vocabulary size; `d` hidden dimension.\n\n**Practical use case.** First layer of almost every language model.\n\n**Source.** Bengio et al. (2003).\n\n#### Token plus position embedding\n\n**Equation**\n\n$$\nh_i^{(0)} = E_{x_i} + P_i\n$$\n\n**Plain-English explanation.** Combines token identity with position information.\n\n**Symbols.** `E_{x_i}` token embedding; `P_i` positional embedding.\n\n**Practical use case.** Original Transformer and GPT-style models with learned or sinusoidal positions.\n\n**Source.** Vaswani et al. (2017); Radford et al. (2018).\n\n#### LM head\n\n**Equation**\n\n$$\nz_t = h_t W_U + b_U\n$$\n\n**Plain-English explanation.** Projects hidden state to vocabulary logits.\n\n**Symbols.** `h_t` is final hidden state; `W_U` is unembedding matrix.\n\n**Practical use case.** Next-token prediction.\n\n**Source.** Bengio et al. (2003); Radford et al. (2018).\n\n#### Tied input-output embeddings\n\n**Equation**\n\n$$\nW_U = E^T\n$$\n\n**Plain-English explanation.** Reuses the token embedding matrix as the output projection.\n\n**Symbols.** `E` is token embedding matrix; `W_U` unembedding matrix.\n\n**Practical use case.** Reduces parameters and can improve language modeling.\n\n**Source.** Press and Wolf (2017); Inan et al. (2017).\n\n#### Pre-norm Transformer block\n\n**Equation**\n\n$$\n\\tilde{h}^{(l)}=h^{(l)}+Attn(LN(h^{(l)})),\\; h^{(l+1)}=\\tilde{h}^{(l)}+MLP(LN(\\tilde{h}^{(l)}))\n$$\n\n**Plain-English explanation.** Applies normalization before each sublayer and adds residual connections.\n\n**Symbols.** `h^{(\\ell)}` is layer input; `\\ell` is layer index.\n\n**Practical use case.** Stable training for deep decoder-only LLMs.\n\n**Source.** Baevski and Auli (2019); Xiong et al. (2020).\n\n#### Post-norm Transformer block\n\n**Equation**\n\n$$\n\\tilde{h}^{(l)}=LN(h^{(l)}+Attn(h^{(l)})),\\; h^{(l+1)}=LN(\\tilde{h}^{(l)}+MLP(\\tilde{h}^{(l)}))\n$$\n\n**Plain-English explanation.** Applies normalization after the residual addition.\n\n**Symbols.** `h^{(\\ell)}` is layer input.\n\n**Practical use case.** Original Transformer architecture.\n\n**Source.** Vaswani et al. (2017).\n\n#### Transformer feed-forward network\n\n**Equation**\n\n$$\nFFN(x)=\\phi(xW_1+b_1)W_2+b_2\n$$\n\n**Plain-English explanation.** Applies a position-wise nonlinear transformation after attention.\n\n**Symbols.** `W_1,W_2` are learned matrices; `\\phi` is activation.\n\n**Practical use case.** The MLP part of a Transformer block.\n\n**Source.** Vaswani et al. (2017).\n\n#### Dropout\n\n**Equation**\n\n$$\nDropout(x)=\\frac{m\\odot x}{1-p},\\; m_i\\sim Bernoulli(1-p)\n$$\n\n**Plain-English explanation.** Randomly zeroes activations during training and rescales survivors.\n\n**Symbols.** `p` is dropout probability; `m` is binary mask.\n\n**Practical use case.** Regularization during pretraining\u002Ffine-tuning.\n\n**Source.** Srivastava et al. (2014).\n\n#### Stochastic depth\n\n**Equation**\n\n$$\nh^{(l+1)}=h^{(l)}+b_l f_l(h^{(l)}),\\; b_l\\sim Bernoulli(p_l)\n$$\n\n**Plain-English explanation.** Randomly skips residual branches during training.\n\n**Symbols.** `b_\\ell` controls whether layer branch is active.\n\n**Practical use case.** Regularizes very deep networks.\n\n**Source.** Huang et al. (2016).\n\n#### Xavier initialization\n\n**Equation**\n\n$$\nW_{ij}\\sim U[-\\sqrt{\\frac{6}{n_{in}+n_{out}}},\\sqrt{\\frac{6}{n_{in}+n_{out}}}]\n$$\n\n**Plain-English explanation.** Initializes weights so variance is controlled across layers.\n\n**Symbols.** `n_{\\text{in}},n_{\\text{out}}` are fan-in and fan-out.\n\n**Practical use case.** Stable model initialization.\n\n**Source.** Glorot and Bengio (2010).\n\n#### He initialization\n\n**Equation**\n\n$$\nW_{ij}\\sim N(0,\\frac{2}{n_{in}})\n$$\n\n**Plain-English explanation.** Initialization designed for ReLU-like activations.\n\n**Symbols.** `n_{\\text{in}}` is fan-in.\n\n**Practical use case.** Training deep networks with ReLU-family activations.\n\n**Source.** He et al. (2015).\n\n## 3. Attention Variants\n\n#### Scaled dot-product attention\n\n**Equation**\n\n$$\nAttention(Q,K,V)=softmax(\\frac{QK^T}{\\sqrt{d_k}})V\n$$\n\n**Plain-English explanation.** Computes a weighted average of values using query-key similarity.\n\n**Symbols.** `Q,K,V` are query\u002Fkey\u002Fvalue matrices; `d_k` is key dimension.\n\n**Practical use case.** Core Transformer attention.\n\n**Source.** Vaswani et al. (2017).\n\n#### Causal masked attention\n\n**Equation**\n\n$$\nA=softmax((QK^T)\u002F\\sqrt{d_k}+M),\\; M_{ij}=0 \\; for \\; j\\leq i,\\; M_{ij}=-\\infty \\; for \\; j>i\n$$\n\n**Plain-English explanation.** Prevents a token from attending to future tokens.\n\n**Symbols.** `M` is causal mask; `i,j` are token positions.\n\n**Practical use case.** Autoregressive decoder-only LLMs.\n\n**Source.** Vaswani et al. (2017).\n\n#### Multi-head attention\n\n**Equation**\n\n$$\nMHA(X)=Concat(head_1,\\dots,head_H)W^O,\\; head_h=Attention(XW_h^Q,XW_h^K,XW_h^V)\n$$\n\n**Plain-English explanation.** Runs attention in multiple learned subspaces then combines the results.\n\n**Symbols.** `H` heads; `W^Q,W^K,W^V,W^O` projections.\n\n**Practical use case.** Allows different heads to capture different patterns.\n\n**Source.** Vaswani et al. (2017).\n\n#### Cross-attention\n\n**Equation**\n\n$$\nCrossAttn(Q_x,K_y,V_y)=softmax(\\frac{Q_xK_y^T}{\\sqrt{d_k}})V_y\n$$\n\n**Plain-English explanation.** Queries from one sequence attend to keys\u002Fvalues from another sequence.\n\n**Symbols.** `x` is decoder\u002Fquery sequence; `y` is encoder\u002Fsource sequence.\n\n**Practical use case.** Encoder-decoder translation, multimodal models, RAG context fusion.\n\n**Source.** Bahdanau et al. (2015); Vaswani et al. (2017).\n\n#### Multi-query attention\n\n**Equation**\n\n$$\nhead_h=Attention(XW_h^Q,XW^K,XW^V)\n$$\n\n**Plain-English explanation.** Each query head has its own query projection but all heads share keys and values.\n\n**Symbols.** `W_h^Q` is per-head; `W^K,W^V` are shared.\n\n**Practical use case.** Reduces KV-cache memory and decode bandwidth.\n\n**Source.** Shazeer (2019).\n\n#### Grouped-query attention\n\n**Equation**\n\n$$\nhead_h=Attention(XW_h^Q,XW_{g(h)}^K,XW_{g(h)}^V)\n$$\n\n**Plain-English explanation.** Groups query heads so multiple query heads share a key\u002Fvalue head.\n\n**Symbols.** `g(h)` maps query head `h` to a KV group.\n\n**Practical use case.** Middle ground between MHA quality and MQA efficiency.\n\n**Source.** Ainslie et al. (2023).\n\n#### Sliding-window attention\n\n**Equation**\n\n$$\nA_{ij}=0\\;if\\; |i-j|> w\n$$\n\n**Plain-English explanation.** Restricts attention to a local window.\n\n**Symbols.** `w` is window radius.\n\n**Practical use case.** Long-context inference with lower memory and compute.\n\n**Source.** Beltagy et al. (2020); Child et al. (2019).\n\n#### Sparse attention\n\n**Equation**\n\n$$\nSparseAttn(Q,K,V)=softmax(\\frac{QK^T}{\\sqrt{d_k}}+M_S)V\n$$\n\n**Plain-English explanation.** Uses a sparsity mask so only selected query-key pairs interact.\n\n**Symbols.** `M_S` is zero for allowed positions and `-\\infty` otherwise.\n\n**Practical use case.** Efficient long-sequence modeling.\n\n**Source.** Child et al. (2019); Beltagy et al. (2020).\n\n#### Linear attention kernelization\n\n**Equation**\n\n$$\nAttn(Q,K,V)_i=\\frac{\\phi(q_i)^T\\sum_j\\phi(k_j)v_j^T}{\\phi(q_i)^T\\sum_j\\phi(k_j)}\n$$\n\n**Plain-English explanation.** Rewrites attention using kernel features to avoid explicit quadratic attention matrix.\n\n**Symbols.** `\\phi` is a positive feature map.\n\n**Practical use case.** Linear-time sequence models and long-context approximations.\n\n**Source.** Katharopoulos et al. (2020).\n\n#### Performer FAVOR+ attention\n\n**Equation**\n\n$$\n\\exp(q^T k)\\approx \\phi(q)^T\\phi(k)\n$$\n\n**Plain-English explanation.** Approximates softmax attention with random feature maps.\n\n**Symbols.** `\\phi` is a random feature mapping.\n\n**Practical use case.** Efficient attention approximation for long sequences.\n\n**Source.** Choromanski et al. (2021).\n\n#### Linformer projection\n\n**Equation**\n\n$$\nAttn(Q,K,V)=softmax(\\frac{Q(EK)^T}{\\sqrt{d_k}})FV\n$$\n\n**Plain-English explanation.** Projects sequence-length dimension of keys and values to lower rank.\n\n**Symbols.** `E,F\\in\\mathbb{R}^{k\\times n}` are learned projections.\n\n**Practical use case.** Low-rank approximation of attention.\n\n**Source.** Wang et al. (2020).\n\n#### Reformer LSH attention\n\n**Equation**\n\n$$\nh(q)=argmax([qR;-qR])\n$$\n\n**Plain-English explanation.** Hashes similar queries\u002Fkeys into buckets using random projections.\n\n**Symbols.** `R` is random rotation\u002Fprojection matrix.\n\n**Practical use case.** Approximate attention with locality-sensitive hashing.\n\n**Source.** Kitaev et al. (2020).\n\n#### FlashAttention online softmax\n\n**Equation**\n\n$$\nm_i=\\max(m_i^{old},m_i^{block}),\\; l_i=e^{m_i^{old}-m_i}l_i^{old}+e^{m_i^{block}-m_i}l_i^{block}\n$$\n\n**Plain-English explanation.** Computes exact attention in blocks while maintaining stable softmax statistics.\n\n**Symbols.** `m_i` is row max; `l_i` is row normalizer.\n\n**Practical use case.** Memory-efficient exact attention kernels on GPU.\n\n**Source.** Dao et al. (2022).\n\n#### Attention output with online normalization\n\n**Equation**\n\n$$\no_i=\\frac{e^{m_i^{old}-m_i}l_i^{old}o_i^{old}+e^{m_i^{block}-m_i}l_i^{block}o_i^{block}}{l_i}\n$$\n\n**Plain-English explanation.** Merges partial block outputs into the exact softmax attention output.\n\n**Symbols.** `o_i` output row; `l_i` softmax denominator.\n\n**Practical use case.** FlashAttention and tiled GPU attention.\n\n**Source.** Dao et al. (2022).\n\n## 4. Positional Encoding and Long Context\n\n#### Sinusoidal positional encoding\n\n**Equation**\n\n$$\nPE_{pos,2i}=\\sin(\\frac{pos}{10000^{2i\u002Fd}}),\\; PE_{pos,2i+1}=\\cos(\\frac{pos}{10000^{2i\u002Fd}})\n$$\n\n**Plain-English explanation.** Encodes absolute position using sinusoids at different frequencies.\n\n**Symbols.** `pos` is token position; `i` dimension index; `d` model dimension.\n\n**Practical use case.** Original Transformer positional representation.\n\n**Source.** Vaswani et al. (2017).\n\n#### Learned absolute position embedding\n\n**Equation**\n\n$$\nh_i^{(0)}=E_{x_i}+P_i,\\; P\\in R^{n_{max}\\times d}\n$$\n\n**Plain-English explanation.** Learns one position vector per position.\n\n**Symbols.** `P_i` is learned position embedding.\n\n**Practical use case.** GPT-style models with finite context length.\n\n**Source.** Radford et al. (2018); Devlin et al. (2019).\n\n#### RoPE rotation matrix\n\n**Equation**\n\n$$\nR_{\\theta,i}=((\\cos(i\\theta),-\\sin(i\\theta)),(\\sin(i\\theta),\\cos(i\\theta)))\n$$\n\n**Plain-English explanation.** Rotates query\u002Fkey vector pairs by position-dependent angles.\n\n**Symbols.** `i` is position; `\\theta` is frequency.\n\n**Practical use case.** Relative-position-aware attention without additive embeddings.\n\n**Source.** Su et al. (2021).\n\n#### RoPE query-key property\n\n**Equation**\n\n$$\n(R_m q)^T(R_n k)=q^T R_{n-m}k\n$$\n\n**Plain-English explanation.** Dot products depend on relative position difference.\n\n**Symbols.** `m,n` are positions.\n\n**Practical use case.** Long-context decoder attention and LLaMA-style models.\n\n**Source.** Su et al. (2021).\n\n#### RoPE complex form\n\n**Equation**\n\n$$\n\\tilde{x}_{2j}+i\\tilde{x}_{2j+1}=(x_{2j}+ix_{2j+1})e^{i m\\theta_j}\n$$\n\n**Plain-English explanation.** Applies RoPE as complex multiplication.\n\n**Symbols.** `m` is position; `\\theta_j` frequency for pair `j`.\n\n**Practical use case.** Efficient implementation and derivation of RoPE.\n\n**Source.** Su et al. (2021).\n\n#### ALiBi bias\n\n**Equation**\n\n$$\nscore_{ij}=\\frac{q_i^T k_j}{\\sqrt{d_k}}-m_h(i-j)\n$$\n\n**Plain-English explanation.** Adds head-specific linear distance penalty to attention scores.\n\n**Symbols.** `m_h` is slope for head `h`; `i-j` is distance.\n\n**Practical use case.** Length extrapolation without learned position embeddings.\n\n**Source.** Press et al. (2021).\n\n#### T5 relative position bias\n\n**Equation**\n\n$$\nscore_{ij}=\\frac{q_i^T k_j}{\\sqrt{d_k}}+b_{bucket(i-j)}\n$$\n\n**Plain-English explanation.** Adds a learned bias based on bucketed relative distance.\n\n**Symbols.** `b` is learned relative position bias.\n\n**Practical use case.** Encoder-decoder and text-to-text Transformer models.\n\n**Source.** Raffel et al. (2020).\n\n#### xPos scale\n\n**Equation**\n\n$$\nq_m'=R_m q_m\\odot s_m,\\; k_n'=R_n k_n\\odot s_n^{-1}\n$$\n\n**Plain-English explanation.** Adds a multiplicative scale to RoPE to improve length extrapolation.\n\n**Symbols.** `s_m` is position-dependent scale.\n\n**Practical use case.** Long-context extrapolation.\n\n**Source.** Sun et al. (2022).\n\n#### NoPE\n\n**Equation**\n\n$$\nh_i^{(0)}=E_{x_i}\n$$\n\n**Plain-English explanation.** Uses no explicit positional encoding.\n\n**Symbols.** `E_{x_i}` is token embedding only.\n\n**Practical use case.** Analyzing whether attention can infer position implicitly.\n\n**Source.** Haviv et al. (2022); Kazemnejad et al. (2023).\n\n#### Position interpolation\n\n**Equation**\n\n$$\nm' = m\\cdot\\frac{L_{train}}{L_{target}}\n$$\n\n**Plain-English explanation.** Compresses target positions into the training position range.\n\n**Symbols.** `L_{train}` original context; `L_{target}` extended context.\n\n**Practical use case.** Extending RoPE models to longer context.\n\n**Source.** Chen et al. (2023).\n\n#### NTK-aware RoPE scaling\n\n**Equation**\n\n$$\n\\theta_j' = \\theta_j\\cdot \\alpha^{-2j\u002F(d-2)}\n$$\n\n**Plain-English explanation.** Changes RoPE frequencies to support longer context.\n\n**Symbols.** `\\alpha` is scale factor; `j` frequency index.\n\n**Practical use case.** Long-context LLaMA-style inference and fine-tuning.\n\n**Source.** bloc97 community method; RoPE scaling literature (2023).\n\n#### YaRN ramp scaling\n\n**Equation**\n\n$$\n\\theta_j' = \\theta_j \u002F s_j,\\; s_j = (1-r_j)s_{low}+r_j s_{high}\n$$\n\n**Plain-English explanation.** Uses dimension-dependent scaling for RoPE frequencies.\n\n**Symbols.** `r_j` is ramp interpolation by dimension; `s_j` is scale.\n\n**Practical use case.** Efficient context-window extension with limited fine-tuning.\n\n**Source.** Peng et al. (2023).\n\n#### Attention sink score\n\n**Equation**\n\n$$\n\\alpha_{ij}=\\frac{\\exp(q_i^T k_j\u002F\\sqrt{d})}{\\sum_{t}\\exp(q_i^T k_t\u002F\\sqrt{d})}\n$$\n\n**Plain-English explanation.** Certain early tokens can receive persistent attention mass and act as sinks.\n\n**Symbols.** `\\alpha_{ij}` is attention probability.\n\n**Practical use case.** Streaming LLM and long-context stability analysis.\n\n**Source.** Xiao et al. (2023).\n\n## 5. Normalization\n\n#### LayerNorm\n\n**Equation**\n\n$$\nLN(x)=\\gamma\\odot\\frac{x-\\mu}{\\sqrt{\\sigma^2+\\epsilon}}+\\beta,\\; \\mu=\\frac{1}{d}\\sum_i x_i,\\; \\sigma^2=\\frac{1}{d}\\sum_i(x_i-\\mu)^2\n$$\n\n**Plain-English explanation.** Normalizes features within one token representation.\n\n**Symbols.** `\\gamma,\\beta` are learned scale and bias.\n\n**Practical use case.** Standard Transformer normalization.\n\n**Source.** Ba et al. (2016).\n\n#### RMSNorm\n\n**Equation**\n\n$$\nRMSNorm(x)=\\gamma\\odot\\frac{x}{\\sqrt{\\frac{1}{d}\\sum_i x_i^2+\\epsilon}}\n$$\n\n**Plain-English explanation.** Normalizes by root mean square without subtracting mean.\n\n**Symbols.** `d` feature dimension; `\\gamma` learned scale.\n\n**Practical use case.** Efficient normalization in LLaMA-style models.\n\n**Source.** Zhang and Sennrich (2019).\n\n#### BatchNorm\n\n**Equation**\n\n$$\nBN(x)=\\gamma\\frac{x-\\mu_B}{\\sqrt{\\sigma_B^2+\\epsilon}}+\\beta\n$$\n\n**Plain-English explanation.** Normalizes using mini-batch statistics.\n\n**Symbols.** `\\mu_B,\\sigma_B^2` are batch mean and variance.\n\n**Practical use case.** Mostly vision\u002FCNNs; less common in autoregressive LLMs.\n\n**Source.** Ioffe and Szegedy (2015).\n\n#### GroupNorm\n\n**Equation**\n\n$$\nGN(x)=\\gamma\\frac{x-\\mu_G}{\\sqrt{\\sigma_G^2+\\epsilon}}+\\beta\n$$\n\n**Plain-English explanation.** Normalizes groups of channels independently.\n\n**Symbols.** `G` denotes a feature group.\n\n**Practical use case.** Vision-language and convolutional components.\n\n**Source.** Wu and He (2018).\n\n#### DeepNorm residual scaling\n\n**Equation**\n\n$$\nx_{l+1}=LN(\\alpha x_l+G_l(x_l))\n$$\n\n**Plain-English explanation.** Scales residual paths to stabilize very deep Transformers.\n\n**Symbols.** `\\alpha` is residual scaling coefficient; `G_l` sublayer.\n\n**Practical use case.** Training extremely deep Transformers.\n\n**Source.** Wang et al. (2022).\n\n## 6. Activations and Gated MLPs\n\n#### ReLU\n\n**Equation**\n\n$$\nReLU(x)=\\max(0,x)\n$$\n\n**Plain-English explanation.** Keeps positive values and zeroes negative values.\n\n**Symbols.** `x` is scalar or tensor.\n\n**Practical use case.** Basic neural activation.\n\n**Source.** Nair and Hinton (2010).\n\n#### GELU exact\n\n**Equation**\n\n$$\nGELU(x)=x\\Phi(x)\n$$\n\n**Plain-English explanation.** Weights input by the Gaussian CDF.\n\n**Symbols.** `\\Phi` is standard normal CDF.\n\n**Practical use case.** BERT\u002FGPT-family activation.\n\n**Source.** Hendrycks and Gimpel (2016).\n\n#### GELU tanh approximation\n\n**Equation**\n\n$$\nGELU(x)\\approx 0.5x(1+\\tanh(\\sqrt{2\u002F\\pi}(x+0.044715x^3)))\n$$\n\n**Plain-English explanation.** Fast approximation of GELU.\n\n**Symbols.** `x` is activation.\n\n**Practical use case.** Efficient Transformer MLPs.\n\n**Source.** Hendrycks and Gimpel (2016).\n\n#### SiLU \u002F Swish\n\n**Equation**\n\n$$\nSiLU(x)=x\\sigma(x)=\\frac{x}{1+e^{-x}}\n$$\n\n**Plain-English explanation.** Smooth gated activation.\n\n**Symbols.** `\\sigma` is logistic sigmoid.\n\n**Practical use case.** Used in SwiGLU and modern LLM MLPs.\n\n**Source.** Elfwing et al. (2018); Ramachandran et al. (2017).\n\n#### Mish\n\n**Equation**\n\n$$\nMish(x)=x\\tanh(\\log(1+e^x))\n$$\n\n**Plain-English explanation.** Smooth non-monotonic activation.\n\n**Symbols.** `\\log(1+e^x)` is softplus.\n\n**Practical use case.** Alternative activation in deep networks.\n\n**Source.** Misra (2019).\n\n#### GLU\n\n**Equation**\n\n$$\nGLU(x)= (xW_a+b_a)\\odot\\sigma(xW_b+b_b)\n$$\n\n**Plain-English explanation.** One projection gates another projection.\n\n**Symbols.** `W_a,W_b` are learned matrices.\n\n**Practical use case.** Gated feed-forward layers.\n\n**Source.** Dauphin et al. (2017).\n\n#### ReGLU\n\n**Equation**\n\n$$\nReGLU(x)= (xW_a)\\odot ReLU(xW_b)\n$$\n\n**Plain-English explanation.** Uses a ReLU gate in a GLU-style block.\n\n**Symbols.** `\\odot` is elementwise product.\n\n**Practical use case.** Transformer feed-forward alternatives.\n\n**Source.** Shazeer (2020).\n\n#### GeGLU\n\n**Equation**\n\n$$\nGeGLU(x)= (xW_a)\\odot GELU(xW_b)\n$$\n\n**Plain-English explanation.** Uses GELU as the gate.\n\n**Symbols.** `W_a,W_b` are projections.\n\n**Practical use case.** T5 and modern Transformer FFN variants.\n\n**Source.** Shazeer (2020).\n\n#### SwiGLU\n\n**Equation**\n\n$$\nSwiGLU(x)= (xW_a)\\odot SiLU(xW_b)\n$$\n\n**Plain-English explanation.** Uses SiLU as the multiplicative gate.\n\n**Symbols.** `W_a,W_b` are learned matrices.\n\n**Practical use case.** LLaMA-family gated MLPs.\n\n**Source.** Shazeer (2020); Touvron et al. (2023).\n\n## 7. Training Losses\n\n#### Label smoothing\n\n**Equation**\n\n$$\ny_i^{LS}=(1-\\epsilon)y_i+\\frac{\\epsilon}{V}\n$$\n\n**Plain-English explanation.** Moves some probability mass from the correct class to other classes.\n\n**Symbols.** `\\epsilon` smoothing coefficient; `V` vocabulary size.\n\n**Practical use case.** Regularizes classification and seq2seq models.\n\n**Source.** Szegedy et al. (2016).\n\n#### Focal loss\n\n**Equation**\n\n$$\nL_{focal}=-(1-p_t)^\\gamma\\log p_t\n$$\n\n**Plain-English explanation.** Down-weights easy examples and focuses on hard examples.\n\n**Symbols.** `p_t` probability assigned to the true class; `\\gamma` focusing parameter.\n\n**Practical use case.** Imbalanced classification and safety classifiers.\n\n**Source.** Lin et al. (2017).\n\n#### InfoNCE\n\n**Equation**\n\n$$\nL_{InfoNCE}=-\\log\\frac{\\exp(s(q,k^+)\u002F\\tau)}{\\exp(s(q,k^+)\u002F\\tau)+\\sum_{k^-}\\exp(s(q,k^-)\u002F\\tau)}\n$$\n\n**Plain-English explanation.** Contrasts a positive pair against negatives.\n\n**Symbols.** `s` similarity; `\\tau` temperature; `k^+` positive; `k^-` negatives.\n\n**Practical use case.** Contrastive sentence embeddings, DPR, CLIP-style training.\n\n**Source.** Oord et al. (2018).\n\n#### NT-Xent\n\n**Equation**\n\n$$\nl_{i,j}=-\\log\\frac{\\exp(sim(z_i,z_j)\u002F\\tau)}{\\sum_{k\\ne i}\\exp(sim(z_i,z_k)\u002F\\tau)}\n$$\n\n**Plain-English explanation.** Normalized temperature-scaled contrastive loss.\n\n**Symbols.** `z_i,z_j` are paired representations.\n\n**Practical use case.** SimCLR-style contrastive learning.\n\n**Source.** Chen et al. (2020).\n\n#### Triplet loss\n\n**Equation**\n\n$$\nL=\\max(0,d(a,p)-d(a,n)+m)\n$$\n\n**Plain-English explanation.** Pushes anchor-positive closer than anchor-negative by a margin.\n\n**Symbols.** `a` anchor; `p` positive; `n` negative; `m` margin.\n\n**Practical use case.** Embedding training and retrieval.\n\n**Source.** Schroff et al. (2015).\n\n#### Masked language modeling loss\n\n**Equation**\n\n$$\nL_{MLM}=-\\sum_{i\\in M}\\log p_\\theta(x_i| x_{\\setminus M})\n$$\n\n**Plain-English explanation.** Predicts masked tokens using bidirectional context.\n\n**Symbols.** `M` masked positions.\n\n**Practical use case.** BERT-style pretraining.\n\n**Source.** Devlin et al. (2019).\n\n#### Causal language modeling loss\n\n**Equation**\n\n$$\nL_{CLM}=-\\sum_{t}\\log p_\\theta(x_t| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Predicts each token from previous tokens only.\n\n**Symbols.** `x_{1:t-1}` past context.\n\n**Practical use case.** GPT-style pretraining.\n\n**Source.** Radford et al. (2018).\n\n#### Span corruption objective\n\n**Equation**\n\n$$\nL_{span}=-\\sum_{t}\\log p_\\theta(y_t| y_{1:t-1},\\tilde{x})\n$$\n\n**Plain-English explanation.** Reconstructs masked spans from corrupted input.\n\n**Symbols.** `\\tilde{x}` corrupted input; `y` target spans.\n\n**Practical use case.** T5-style denoising pretraining.\n\n**Source.** Raffel et al. (2020).\n\n#### Knowledge distillation\n\n**Equation**\n\n$$\nL_{KD}=T^2D_{KL}(softmax(z_T\u002FT)||softmax(z_S\u002FT))\n$$\n\n**Plain-English explanation.** Trains a student to match teacher soft logits.\n\n**Symbols.** `T` temperature; `z_T,z_S` teacher\u002Fstudent logits.\n\n**Practical use case.** Compressing LLMs and transferring behavior.\n\n**Source.** Hinton et al. (2015).\n\n#### Hidden-state distillation\n\n**Equation**\n\n$$\nL_{hid}=\\sum_{l}\\|H_{S}^{(l)}W_l-H_T^{(m(l))}\\|_2^2\n$$\n\n**Plain-English explanation.** Matches student hidden states to teacher hidden states.\n\n**Symbols.** `H_S,H_T` student\u002Fteacher states; `m(\\ell)` layer mapping.\n\n**Practical use case.** DistilBERT-style model compression.\n\n**Source.** Sanh et al. (2019).\n\n## 8. Optimization\n\n#### SGD\n\n**Equation**\n\n$$\n\\theta_{t+1}=\\theta_t-\\eta\\nabla_\\theta L(\\theta_t)\n$$\n\n**Plain-English explanation.** Moves parameters opposite the gradient.\n\n**Symbols.** `\\eta` learning rate.\n\n**Practical use case.** Baseline neural optimization.\n\n**Source.** Robbins and Monro (1951).\n\n#### Momentum\n\n**Equation**\n\n$$\nv_t=\\mu v_{t-1}+g_t,\\; \\theta_{t+1}=\\theta_t-\\eta v_t\n$$\n\n**Plain-English explanation.** Accumulates gradient direction over time.\n\n**Symbols.** `v_t` velocity; `\\mu` momentum; `g_t` gradient.\n\n**Practical use case.** Faster, smoother optimization.\n\n**Source.** Polyak (1964).\n\n#### Adam moments\n\n**Equation**\n\n$$\nm_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\; v_t=\\beta_2v_{t-1}+(1-\\beta_2)g_t^2\n$$\n\n**Plain-English explanation.** Tracks first and second moments of gradients.\n\n**Symbols.** `m_t` mean; `v_t` uncentered variance.\n\n**Practical use case.** Default optimizer family for LLMs.\n\n**Source.** Kingma and Ba (2015).\n\n#### Adam update\n\n**Equation**\n\n$$\n\\hat{m}_t=\\frac{m_t}{1-\\beta_1^t},\\; \\hat{v}_t=\\frac{v_t}{1-\\beta_2^t},\\; \\theta_{t+1}=\\theta_t-\\eta\\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t}+\\epsilon}\n$$\n\n**Plain-English explanation.** Bias-corrects moments and applies adaptive step size.\n\n**Symbols.** `\\beta_1,\\beta_2` decay rates.\n\n**Practical use case.** Pretraining and fine-tuning.\n\n**Source.** Kingma and Ba (2015).\n\n#### AdamW\n\n**Equation**\n\n$$\n\\theta_{t+1}=\\theta_t-\\eta(\\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t}+\\epsilon}+\\lambda\\theta_t)\n$$\n\n**Plain-English explanation.** Decouples weight decay from adaptive gradient update.\n\n**Symbols.** `\\lambda` weight decay coefficient.\n\n**Practical use case.** Standard LLM training optimizer.\n\n**Source.** Loshchilov and Hutter (2019).\n\n#### Lion\n\n**Equation**\n\n$$\nm_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\; \\theta_{t+1}=\\theta_t-\\eta sign(m_t)\n$$\n\n**Plain-English explanation.** Uses sign of momentum-like update.\n\n**Symbols.** `m_t` momentum estimate.\n\n**Practical use case.** Memory-efficient optimization experiments.\n\n**Source.** Chen et al. (2023).\n\n#### Adafactor factored second moment\n\n**Equation**\n\n$$\n\\hat{V}_{ij}=\\frac{r_i c_j}{\\bar{r}},\\; r_i=\\frac{1}{n}\\sum_j V_{ij},\\; c_j=\\frac{1}{m}\\sum_i V_{ij}\n$$\n\n**Plain-English explanation.** Approximates second-moment matrix using row and column factors.\n\n**Symbols.** `r,c` are row\u002Fcolumn statistics.\n\n**Practical use case.** Memory-efficient training of large Transformers.\n\n**Source.** Shazeer and Stern (2018).\n\n#### Global gradient clipping\n\n**Equation**\n\n$$\ngarrow g\\cdot\\min(1,\\frac{c}{\\|g\\|_2})\n$$\n\n**Plain-English explanation.** Scales gradients down when norm exceeds threshold.\n\n**Symbols.** `c` clipping threshold.\n\n**Practical use case.** Prevents exploding gradients.\n\n**Source.** Pascanu et al. (2013).\n\n#### Cosine decay schedule\n\n**Equation**\n\n$$\n\\eta_t=\\eta_{min}+\\frac{1}{2}(\\eta_{max}-\\eta_{min})(1+\\cos\\frac{\\pi t}{T})\n$$\n\n**Plain-English explanation.** Smoothly decays learning rate following a cosine curve.\n\n**Symbols.** `T` total steps.\n\n**Practical use case.** LLM pretraining and fine-tuning schedules.\n\n**Source.** Loshchilov and Hutter (2017).\n\n#### Linear warmup\n\n**Equation**\n\n$$\n\\eta_t=\\eta_{max}\\frac{t}{T_{warm}},\\; t\\leq T_{warm}\n$$\n\n**Plain-English explanation.** Gradually increases learning rate at the beginning of training.\n\n**Symbols.** `T_{warm}` warmup steps.\n\n**Practical use case.** Stabilizes early large-scale training.\n\n**Source.** Vaswani et al. (2017).\n\n#### Inverse square-root schedule\n\n**Equation**\n\n$$\n\\eta_t=d_{model}^{-1\u002F2}\\min(t^{-1\u002F2},tT_{warm}^{-3\u002F2})\n$$\n\n**Plain-English explanation.** Warmup followed by inverse-square-root decay.\n\n**Symbols.** `d_{model}` hidden size.\n\n**Practical use case.** Original Transformer schedule.\n\n**Source.** Vaswani et al. (2017).\n\n## 9. Sampling and Decoding\n\n#### Greedy decoding\n\n**Equation**\n\n$$\nx_t=argmax_{i} p_\\theta(i| x_{1:t-1})\n$$\n\n**Plain-English explanation.** Selects the most likely next token.\n\n**Symbols.** `i` ranges over vocabulary.\n\n**Practical use case.** Deterministic generation.\n\n**Source.** Standard decoding method.\n\n#### Temperature sampling\n\n**Equation**\n\n$$\np_i^{(\\tau)}=\\frac{\\exp(z_i\u002F\\tau)}{\\sum_j\\exp(z_j\u002F\\tau)}\n$$\n\n**Plain-English explanation.** Controls sharpness of the output distribution.\n\n**Symbols.** `\\tau` temperature; lower is more deterministic.\n\n**Practical use case.** Creative vs precise generation control.\n\n**Source.** Ackley et al. (1985); neural LM practice.\n\n#### Top-k sampling\n\n**Equation**\n\n$$\nS_k=TopK(p,k),\\; p_i'=\\frac{p_i1[i\\in S_k]}{\\sum_{j\\in S_k}p_j}\n$$\n\n**Plain-English explanation.** Keeps only the k most likely tokens and renormalizes.\n\n**Symbols.** `S_k` top-k token set.\n\n**Practical use case.** Open-ended generation.\n\n**Source.** Fan et al. (2018).\n\n#### Nucleus top-p sampling\n\n**Equation**\n\n$$\nS_p=\\min\\{S:\\sum_{i\\in S}p_i\\geq p\\},\\; p_i'=\\frac{p_i1[i\\in S_p]}{\\sum_{j\\in S_p}p_j}\n$$\n\n**Plain-English explanation.** Keeps the smallest token set whose cumulative probability exceeds threshold p.\n\n**Symbols.** `p` is nucleus threshold.\n\n**Practical use case.** Adaptive sampling for open-ended text.\n\n**Source.** Holtzman et al. (2020).\n\n#### Typical sampling\n\n**Equation**\n\n$$\nS=\\{i:| -\\log p_i-H(p)|\\leq \\tau\\}\n$$\n\n**Plain-English explanation.** Keeps tokens whose surprise is close to distribution entropy.\n\n**Symbols.** `H(p)` entropy; `\\tau` typicality threshold.\n\n**Practical use case.** Avoids both dull and unlikely tokens.\n\n**Source.** Meister et al. (2023).\n\n#### Min-p sampling\n\n**Equation**\n\n$$\nS=\\{i:p_i\\geq \\alpha\\max_jp_j\\}\n$$\n\n**Plain-English explanation.** Keeps tokens whose probability is at least a fraction of the top token.\n\n**Symbols.** `\\alpha` min-p threshold.\n\n**Practical use case.** Sharper adaptive sampling for LLM inference.\n\n**Source.** Community decoding method, 2023.\n\n#### Beam search recursion\n\n**Equation**\n\n$$\nB_t=TopB_{y_{1:t}}[\\log p_\\theta(y_{1:t}| x)]\n$$\n\n**Plain-English explanation.** Maintains the top B partial sequences by log-probability.\n\n**Symbols.** `B_t` beam set at step `t`.\n\n**Practical use case.** Translation and structured decoding.\n\n**Source.** Lowerre (1976); Sutskever et al. (2014).\n\n#### Beam length penalty\n\n**Equation**\n\n$$\ns(y)=\\frac{\\log p(y| x)}{(\\frac{5+|y|}{6})^\\alpha}\n$$\n\n**Plain-English explanation.** Penalizes or normalizes beam scores by sequence length.\n\n**Symbols.** `|y|` output length; `\\alpha` penalty strength.\n\n**Practical use case.** Reduces short-output bias.\n\n**Source.** Wu et al. (2016).\n\n#### Contrastive decoding\n\n**Equation**\n\n$$\nx_t=argmax_{x}[\\log p_{expert}(x| c)-\\alpha\\log p_{amateur}(x| c)]\n$$\n\n**Plain-English explanation.** Selects tokens preferred by a strong model over a weaker model.\n\n**Symbols.** `\\alpha` controls amateur penalty.\n\n**Practical use case.** Reducing degeneration and improving factuality.\n\n**Source.** Li et al. (2023).\n\n#### Speculative decoding acceptance\n\n**Equation**\n\n$$\na_t=\\min(1,\\frac{p(x_t| x_{1:t-1})}{q(x_t| x_{1:t-1})})\n$$\n\n**Plain-English explanation.** Accepts draft-model tokens with probability based on target\u002Fdraft likelihood ratio.\n\n**Symbols.** `p` target model; `q` draft model.\n\n**Practical use case.** Low-latency LLM serving.\n\n**Source.** Leviathan et al. (2023); Chen et al. (2023).\n\n#### Mirostat target surprise update\n\n**Equation**\n\n$$\n\\mu_{t+1}=\\mu_t-\\eta(s_t-\\tau)\n$$\n\n**Plain-English explanation.** Adjusts sampling to maintain target surprise\u002Fperplexity.\n\n**Symbols.** `s_t` observed surprise; `\\tau` target; `\\eta` update rate.\n\n**Practical use case.** Diversity control in text generation.\n\n**Source.** Basu et al. (2021).\n\n## 10. Alignment and Preference Optimization\n\n#### Supervised fine-tuning\n\n**Equation**\n\n$$\nL_{SFT}(\\theta)=-E_{(x,y)\\sim D}\\sum_t\\log\\pi_\\theta(y_t| x,y_{1:t-1})\n$$\n\n**Plain-English explanation.** Trains the model on demonstration responses.\n\n**Symbols.** `D` supervised instruction dataset.\n\n**Practical use case.** Instruction tuning before preference optimization.\n\n**Source.** Ouyang et al. (2022).\n\n#### Bradley-Terry preference model\n\n**Equation**\n\n$$\nP(y_w\\succ y_l| x)=\\sigma(r_\\phi(x,y_w)-r_\\phi(x,y_l))\n$$\n\n**Plain-English explanation.** Models probability that a winning response is preferred to a losing response.\n\n**Symbols.** `r_\\phi` reward model; `y_w,y_l` preferred\u002Frejected responses.\n\n**Practical use case.** Reward modeling for RLHF.\n\n**Source.** Bradley and Terry (1952); Ouyang et al. (2022).\n\n#### Reward model loss\n\n**Equation**\n\n$$\nL_{RM}=-E[\\log\\sigma(r_\\phi(x,y_w)-r_\\phi(x,y_l))]\n$$\n\n**Plain-English explanation.** Trains rewards so preferred responses score higher.\n\n**Symbols.** `\\phi` reward model parameters.\n\n**Practical use case.** RLHF reward model training.\n\n**Source.** Christiano et al. (2017); Ouyang et al. (2022).\n\n#### RLHF KL-regularized objective\n\n**Equation**\n\n$$\n\\max_\\theta\\;E_{y\\sim\\pi_\\theta}[r_\\phi(x,y)-\\beta D_{KL}(\\pi_\\theta(\\cdot| x)||\\pi_{ref}(\\cdot| x))]\n$$\n\n**Plain-English explanation.** Optimizes reward while preventing policy from drifting too far from reference model.\n\n**Symbols.** `\\beta` KL penalty; `\\pi_{ref}` reference policy.\n\n**Practical use case.** RLHF policy training.\n\n**Source.** Ziegler et al. (2019); Ouyang et al. (2022).\n\n#### PPO clipped objective\n\n**Equation**\n\n$$\nL^{CLIP}(\\theta)=E_t[\\min(r_t(\\theta)A_t,clip(r_t(\\theta),1-\\epsilon,1+\\epsilon)A_t)]\n$$\n\n**Plain-English explanation.** Limits how much a policy update can change action probabilities.\n\n**Symbols.** `r_t=\\pi_\\theta(a_t\\mid s_t)\u002F\\pi_{old}(a_t\\mid s_t)`; `A_t` advantage.\n\n**Practical use case.** RLHF fine-tuning.\n\n**Source.** Schulman et al. (2017); Ouyang et al. (2022).\n\n#### Value function loss\n\n**Equation**\n\n$$\nL_V=E_t[(V_\\psi(s_t)-R_t)^2]\n$$\n\n**Plain-English explanation.** Trains a critic to predict returns.\n\n**Symbols.** `V_\\psi` value function; `R_t` return.\n\n**Practical use case.** PPO\u002FRLHF critic training.\n\n**Source.** Sutton and Barto (2018); Schulman et al. (2017).\n\n#### GAE\n\n**Equation**\n\n$$\n\\hat{A}_t^{GAE}=\\sum_{l=0}^{\\infty}(\\gamma\\lambda)^l\\delta_{t+l},\\; \\delta_t=r_t+\\gamma V(s_{t+1})-V(s_t)\n$$\n\n**Plain-English explanation.** Computes a low-variance advantage estimate from TD residuals.\n\n**Symbols.** `\\gamma` discount; `\\lambda` bias-variance parameter.\n\n**Practical use case.** PPO and actor-critic training.\n\n**Source.** Schulman et al. (2016).\n\n#### DPO implicit reward\n\n**Equation**\n\n$$\nr_\\theta(x,y)=\\beta\\log\\frac{\\pi_\\theta(y| x)}{\\pi_{ref}(y| x)}+C(x)\n$$\n\n**Plain-English explanation.** Defines an implicit reward from policy\u002Freference log-ratio.\n\n**Symbols.** `\\beta` temperature; `C(x)` prompt-only constant.\n\n**Practical use case.** Direct preference optimization.\n\n**Source.** Rafailov et al. (2023).\n\n#### DPO loss\n\n**Equation**\n\n$$\nL_{DPO}=-E\\log\\sigma(\\beta\\log\\frac{\\pi_\\theta(y_w| x)}{\\pi_{ref}(y_w| x)}-\\beta\\log\\frac{\\pi_\\theta(y_l| x)}{\\pi_{ref}(y_l| x)})\n$$\n\n**Plain-English explanation.** Optimizes preference pairs directly without an explicit reward model.\n\n**Symbols.** `y_w,y_l` winner\u002Floser responses.\n\n**Practical use case.** Alignment after SFT.\n\n**Source.** Rafailov et al. (2023).\n\n#### IPO loss\n\n**Equation**\n\n$$\nL_{IPO}=E[(\\log\\frac{\\pi_\\theta(y_w| x)\\pi_{ref}(y_l| x)}{\\pi_\\theta(y_l| x)\\pi_{ref}(y_w| x)}-\\frac{1}{2\\beta})^2]\n$$\n\n**Plain-English explanation.** Uses a squared loss on preference log-ratio margins.\n\n**Symbols.** `\\beta` controls target margin.\n\n**Practical use case.** Stable preference optimization alternative to DPO.\n\n**Source.** Azar et al. (2023).\n\n#### KTO utility\n\n**Equation**\n\n$$\nv(x,y)=\\beta\\log\\frac{\\pi_\\theta(y| x)}{\\pi_{ref}(y| x)}\n$$\n\n**Plain-English explanation.** Computes desirability from policy\u002Freference log-ratio.\n\n**Symbols.** `v` is human utility surrogate.\n\n**Practical use case.** Preference tuning with desirable\u002Fundesirable labels.\n\n**Source.** Ethayarajh et al. (2024).\n\n#### ORPO loss\n\n**Equation**\n\n$$\nL_{ORPO}=L_{SFT}-\\lambda\\log\\sigma(\\log\\frac{\\pi_\\theta(y_w| x)}{1-\\pi_\\theta(y_w| x)}-\\log\\frac{\\pi_\\theta(y_l| x)}{1-\\pi_\\theta(y_l| x)})\n$$\n\n**Plain-English explanation.** Combines supervised learning with an odds-ratio preference penalty.\n\n**Symbols.** `\\lambda` preference weight.\n\n**Practical use case.** Reference-free preference alignment.\n\n**Source.** Hong et al. (2024).\n\n#### SimPO loss\n\n**Equation**\n\n$$\nL_{SimPO}=-\\log\\sigma(\\frac{\\beta}{|y_w|}\\log\\pi_\\theta(y_w| x)-\\frac{\\beta}{|y_l|}\\log\\pi_\\theta(y_l| x)-\\gamma)\n$$\n\n**Plain-English explanation.** Uses length-normalized policy likelihood margin without reference model.\n\n**Symbols.** `\\gamma` target margin.\n\n**Practical use case.** Simple reference-free preference optimization.\n\n**Source.** Meng et al. (2024).\n\n#### GRPO group advantage\n\n**Equation**\n\n$$\n\\hat{A}_{i}=\\frac{r_i-mean(r_{1:G})}{std(r_{1:G})}\n$$\n\n**Plain-English explanation.** Computes advantage relative to other sampled responses for the same prompt.\n\n**Symbols.** `G` samples per prompt; `r_i` reward of sample `i`.\n\n**Practical use case.** RL for reasoning models without a value model.\n\n**Source.** Shao et al. (2024).\n\n#### GRPO objective\n\n**Equation**\n\n$$\nJ_{GRPO}=E[\\frac{1}{G}\\sum_{i=1}^{G}\\min(\\rho_i\\hat{A}_i,clip(\\rho_i,1-\\epsilon,1+\\epsilon)\\hat{A}_i)-\\beta D_{KL}(\\pi_\\theta||\\pi_{ref})]\n$$\n\n**Plain-English explanation.** PPO-style clipped objective with group-relative advantages.\n\n**Symbols.** `\\rho_i=\\pi_\\theta(o_i\\mid q)\u002F\\pi_{old}(o_i\\mid q)`.\n\n**Practical use case.** RLVR training for math\u002Freasoning models.\n\n**Source.** Shao et al. (2024).\n\n#### RLOO baseline\n\n**Equation**\n\n$$\n\\hat{A}_i=r_i-\\frac{1}{G-1}\\sum_{j\\ne i}r_j\n$$\n\n**Plain-English explanation.** Uses leave-one-out average reward as baseline.\n\n**Symbols.** `G` completions for the same prompt.\n\n**Practical use case.** Low-variance RL for LLMs.\n\n**Source.** Kool et al. (2019); Ahmadian et al. (2024).\n\n#### Rejection sampling fine-tuning\n\n**Equation**\n\n$$\nD' = \\{(x,y): r_\\phi(x,y)\\geq \\tau\\},\\; L=-E_{(x,y)\\in D'}\\log\\pi_\\theta(y| x)\n$$\n\n**Plain-English explanation.** Filters generated samples by reward then trains on accepted samples.\n\n**Symbols.** `\\tau` reward threshold.\n\n**Practical use case.** Improving instruction models without online RL.\n\n**Source.** Ouyang et al. (2022).\n\n## 11. PEFT and Adapter Methods\n\n#### LoRA\n\n**Equation**\n\n$$\nW'=W+\\Delta W,\\; \\Delta W=\\frac{\\alpha}{r}BA\n$$\n\n**Plain-English explanation.** Adds a trainable low-rank update to frozen weights.\n\n**Symbols.** `A\\in\\mathbb{R}^{r\\times d_{in}}`, `B\\in\\mathbb{R}^{d_{out}\\times r}`; `r` rank.\n\n**Practical use case.** Memory-efficient fine-tuning.\n\n**Source.** Hu et al. (2022).\n\n#### QLoRA dequantized LoRA layer\n\n**Equation**\n\n$$\ny=xdequant(W_q)+x\\frac{\\alpha}{r}BA\n$$\n\n**Plain-English explanation.** Combines quantized frozen weights with trainable LoRA adapters.\n\n**Symbols.** `W_q` quantized base weight.\n\n**Practical use case.** Fine-tuning large models on limited GPU memory.\n\n**Source.** Dettmers et al. (2023).\n\n#### DoRA\n\n**Equation**\n\n$$\nW'=m\\frac{W+\\Delta W}{\\|W+\\Delta W\\|_c}\n$$\n\n**Plain-English explanation.** Separates weight magnitude and direction adaptation.\n\n**Symbols.** `m` magnitude parameter; `\\|\\cdot\\|_c` column norm.\n\n**Practical use case.** Improves LoRA-style fine-tuning quality.\n\n**Source.** Liu et al. (2024).\n\n#### AdaLoRA rank allocation\n\n**Equation**\n\n$$\n\\Delta W=\\sum_{i=1}^{r}s_i u_i v_i^T\n$$\n\n**Plain-English explanation.** Allocates rank budget by learned importance scores.\n\n**Symbols.** `s_i` singular value\u002Fimportance; `u_i,v_i` factors.\n\n**Practical use case.** Adaptive parameter-efficient fine-tuning.\n\n**Source.** Zhang et al. (2023).\n\n#### Prefix tuning\n\n**Equation**\n\n$$\nK'=[K_p;K],\\; V'=[V_p;V]\n$$\n\n**Plain-English explanation.** Prepends trainable key\u002Fvalue vectors to attention.\n\n**Symbols.** `K_p,V_p` learned prefix states.\n\n**Practical use case.** Task adaptation without updating model weights.\n\n**Source.** Li and Liang (2021).\n\n#### Prompt tuning\n\n**Equation**\n\n$$\nh^{(0)}=[P_1,\\dots,P_m,E_{x_1},\\dots,E_{x_n}]\n$$\n\n**Plain-English explanation.** Prepends trainable soft prompt embeddings.\n\n**Symbols.** `P_i` learned prompt vectors.\n\n**Practical use case.** Parameter-efficient adaptation.\n\n**Source.** Lester et al. (2021).\n\n#### P-tuning v2 deep prompts\n\n**Equation**\n\n$$\nh_l'=[P_l;h_l]\n$$\n\n**Plain-English explanation.** Adds trainable prompts at multiple layers.\n\n**Symbols.** `P_\\ell` prompt vectors for layer `\\ell`.\n\n**Practical use case.** Deep prompt tuning for NLU\u002FNLG.\n\n**Source.** Liu et al. (2022).\n\n#### IA3\n\n**Equation**\n\n$$\ny=(l_W\\odot W)x,\\; K'=l_K\\odot K,\\; V'=l_V\\odot V\n$$\n\n**Plain-English explanation.** Learns multiplicative vectors that rescale activations or weights.\n\n**Symbols.** `l_W,l_K,l_V` learned scaling vectors.\n\n**Practical use case.** Very lightweight adaptation.\n\n**Source.** Liu et al. (2022).\n\n#### BitFit\n\n**Equation**\n\n$$\n\\theta_{train}=\\{b: b\\in\\theta\\}\n$$\n\n**Plain-English explanation.** Fine-tunes only bias parameters.\n\n**Symbols.** `b` bias vectors.\n\n**Practical use case.** Cheap task adaptation baseline.\n\n**Source.** Ben Zaken et al. (2022).\n\n## 12. Quantization\n\n#### Affine quantization\n\n**Equation**\n\n$$\nq=clip(round(\\frac{x}{s})+z,q_{min},q_{max}),\\; \\hat{x}=s(q-z)\n$$\n\n**Plain-English explanation.** Maps floating-point values to integers using scale and zero-point.\n\n**Symbols.** `s` scale; `z` zero-point.\n\n**Practical use case.** INT8\u002FINT4 deployment.\n\n**Source.** Jacob et al. (2018).\n\n#### Symmetric quantization scale\n\n**Equation**\n\n$$\ns=\\frac{\\max(|x|)}{2^{b-1}-1},\\; q=round(x\u002Fs)\n$$\n\n**Plain-English explanation.** Uses zero-centered integer range.\n\n**Symbols.** `b` number of bits.\n\n**Practical use case.** Weight quantization.\n\n**Source.** Jacob et al. (2018).\n\n#### Asymmetric quantization scale\n\n**Equation**\n\n$$\ns=\\frac{x_{max}-x_{min}}{q_{max}-q_{min}},\\; z=round(q_{min}-\\frac{x_{min}}{s})\n$$\n\n**Plain-English explanation.** Uses non-zero zero-point to cover arbitrary ranges.\n\n**Symbols.** `x_{min},x_{max}` floating range.\n\n**Practical use case.** Activation quantization.\n\n**Source.** Jacob et al. (2018).\n\n#### Per-channel quantization\n\n**Equation**\n\n$$\ns_c=\\frac{\\max_i |W_{c,i}|}{2^{b-1}-1}\n$$\n\n**Plain-English explanation.** Computes a separate scale for each channel.\n\n**Symbols.** `c` output channel.\n\n**Practical use case.** Improves quantized weight accuracy.\n\n**Source.** Jacob et al. (2018).\n\n#### GPTQ objective\n\n**Equation**\n\n$$\n\\min_{\\hat{W}}\\|WX-\\hat{W}X\\|_2^2\n$$\n\n**Plain-English explanation.** Quantizes weights while minimizing output reconstruction error.\n\n**Symbols.** `X` calibration activations; `\\hat{W}` quantized weights.\n\n**Practical use case.** Post-training LLM quantization.\n\n**Source.** Frantar et al. (2023).\n\n#### AWQ salient-channel scaling\n\n**Equation**\n\n$$\n\\min_s\\|Q(Wdiag(s))diag(s)^{-1}X-WX\\|_2^2\n$$\n\n**Plain-English explanation.** Scales channels before quantization to protect important weights.\n\n**Symbols.** `s` channel scale vector; `Q` quantizer.\n\n**Practical use case.** Accurate low-bit weight-only quantization.\n\n**Source.** Lin et al. (2023).\n\n#### SmoothQuant transform\n\n**Equation**\n\n$$\n\\hat{X}=Xdiag(s)^{-1},\\; \\hat{W}=diag(s)W\n$$\n\n**Plain-English explanation.** Migrates activation outlier difficulty into weights.\n\n**Symbols.** `s` smoothing vector.\n\n**Practical use case.** W8A8 LLM quantization.\n\n**Source.** Xiao et al. (2023).\n\n#### LLM.int8 outlier decomposition\n\n**Equation**\n\n$$\nXW=X_{normal}W_{int8}+X_{outlier}W_{fp16}\n$$\n\n**Plain-English explanation.** Computes most values in int8 while keeping outliers in higher precision.\n\n**Symbols.** `X_{outlier}` activation outlier subspace.\n\n**Practical use case.** 8-bit inference for large LMs.\n\n**Source.** Dettmers et al. (2022).\n\n#### NF4 quantization\n\n**Equation**\n\n$$\nq_i=argmin_{c_j\\in C_{NF4}}|x_i\u002Fs-c_j|\n$$\n\n**Plain-English explanation.** Maps normalized values to a learned\u002Fnon-uniform 4-bit codebook optimized for normal weights.\n\n**Symbols.** `C_{NF4}` NF4 codebook; `s` block scale.\n\n**Practical use case.** QLoRA 4-bit finetuning.\n\n**Source.** Dettmers et al. (2023).\n\n#### Double quantization\n\n**Equation**\n\n$$\nq_s=Q(s),\\; \\hat{x}=dequant(q_x,dequant(q_s))\n$$\n\n**Plain-English explanation.** Quantizes the quantization scales themselves.\n\n**Symbols.** `q_s` quantized scale.\n\n**Practical use case.** Memory savings in QLoRA.\n\n**Source.** Dettmers et al. (2023).\n\n#### KV-cache quantization\n\n**Equation**\n\n$$\n\\hat{K}=s_K(Q(K)-z_K),\\; \\hat{V}=s_V(Q(V)-z_V)\n$$\n\n**Plain-English explanation.** Stores keys and values in lower precision and dequantizes for attention.\n\n**Symbols.** `K,V` cached attention tensors.\n\n**Practical use case.** Long-context serving memory reduction.\n\n**Source.** LLM inference quantization literature, 2023-2025.\n\n#### Quantization-aware training loss\n\n**Equation**\n\n$$\n\\min_\\theta E_{(x,y)}L(f_{Q(\\theta)}(x),y)\n$$\n\n**Plain-English explanation.** Trains while simulating quantized weights\u002Factivations.\n\n**Symbols.** `Q(\\theta)` fake-quantized parameters.\n\n**Practical use case.** Maintaining accuracy after deployment quantization.\n\n**Source.** Jacob et al. (2018).\n\n## 13. Scaling Laws\n\n#### Kaplan scaling law\n\n**Equation**\n\n$$\nL(N,D)=L_\\infty + aN^{-\\alpha}+bD^{-\\beta}\n$$\n\n**Plain-English explanation.** Models loss as a power law of parameter count and data size.\n\n**Symbols.** `N` parameters; `D` training tokens.\n\n**Practical use case.** Predicting model performance and compute trade-offs.\n\n**Source.** Kaplan et al. (2020).\n\n#### Compute estimate\n\n**Equation**\n\n$$\nC\\approx 6ND\n$$\n\n**Plain-English explanation.** Approximates training compute for dense Transformer training.\n\n**Symbols.** `C` FLOPs; `N` parameters; `D` tokens.\n\n**Practical use case.** Budgeting pretraining runs.\n\n**Source.** Kaplan et al. (2020); Hoffmann et al. (2022).\n\n#### Chinchilla compute-optimal relation\n\n**Equation**\n\n$$\nN_{opt}\\propto C^a,\\; D_{opt}\\propto C^b,\\; a\\approx b\\approx 0.5\n$$\n\n**Plain-English explanation.** Compute-optimal training scales parameters and data roughly equally.\n\n**Symbols.** `C` compute budget.\n\n**Practical use case.** Choosing model\u002Fdata size under fixed compute.\n\n**Source.** Hoffmann et al. (2022).\n\n#### Loss versus compute\n\n**Equation**\n\n$$\nL(C)=L_\\infty + AC^{-\\alpha}\n$$\n\n**Plain-English explanation.** Represents loss decreasing as a power law with compute.\n\n**Symbols.** `A,\\alpha` fitted constants.\n\n**Practical use case.** Forecasting model improvements.\n\n**Source.** Kaplan et al. (2020).\n\n#### Emergence-style logistic fit\n\n**Equation**\n\n$$\nP(success| C)=\\frac{1}{1+\\exp[-a(\\log C-b)]}\n$$\n\n**Plain-English explanation.** A smooth proxy for abrupt benchmark transitions.\n\n**Symbols.** `a` slope; `b` midpoint.\n\n**Practical use case.** Analyzing threshold-like capabilities.\n\n**Source.** Wei et al. (2022); Schaeffer et al. (2023).\n\n## 14. Mixture of Experts\n\n#### MoE gating\n\n**Equation**\n\n$$\ng(x)=softmax(W_gx)\n$$\n\n**Plain-English explanation.** Computes expert-selection probabilities.\n\n**Symbols.** `W_g` router weights.\n\n**Practical use case.** Routing tokens to experts.\n\n**Source.** Shazeer et al. (2017).\n\n#### Top-k expert routing\n\n**Equation**\n\n$$\nS(x)=TopK(g(x),k)\n$$\n\n**Plain-English explanation.** Selects the highest-scoring experts for a token.\n\n**Symbols.** `k` number of selected experts.\n\n**Practical use case.** Sparse MoE activation.\n\n**Source.** Shazeer et al. (2017).\n\n#### MoE output\n\n**Equation**\n\n$$\ny=\\sum_{e\\in S(x)}g_e(x)E_e(x)\n$$\n\n**Plain-English explanation.** Combines selected expert outputs weighted by router probabilities.\n\n**Symbols.** `E_e` expert network `e`.\n\n**Practical use case.** Sparse scaling of Transformer MLPs.\n\n**Source.** Shazeer et al. (2017).\n\n#### Switch Transformer routing\n\n**Equation**\n\n$$\ne^*=argmax_{e} g_e(x),\\; y=g_{e^*}(x)E_{e^*}(x)\n$$\n\n**Plain-English explanation.** Routes each token to one expert only.\n\n**Symbols.** `e^*` selected expert.\n\n**Practical use case.** Efficient sparse MoE training.\n\n**Source.** Fedus et al. (2021).\n\n#### MoE load balancing loss\n\n**Equation**\n\n$$\nL_{aux}=\\alpha N\\sum_{e=1}^{N}f_ep_e\n$$\n\n**Plain-English explanation.** Encourages balanced expert usage.\n\n**Symbols.** `f_e` fraction of tokens routed to expert `e`; `p_e` mean router probability.\n\n**Practical use case.** Avoids expert collapse.\n\n**Source.** Fedus et al. (2021).\n\n#### Expert capacity\n\n**Equation**\n\n$$\nC_e=\\lceil\\frac{T}{N}\\cdot c\\rceil\n$$\n\n**Plain-English explanation.** Limits how many tokens each expert can process.\n\n**Symbols.** `T` tokens; `N` experts; `c` capacity factor.\n\n**Practical use case.** MoE training\u002Finference memory control.\n\n**Source.** Fedus et al. (2021).\n\n#### Router z-loss\n\n**Equation**\n\n$$\nL_z=\\frac{1}{B}\\sum_{i=1}^{B}(\\log\\sum_j e^{z_{ij}})^2\n$$\n\n**Plain-English explanation.** Penalizes large router logits for stability.\n\n**Symbols.** `z_{ij}` router logit.\n\n**Practical use case.** Stable MoE router training.\n\n**Source.** Zoph et al. (2022).\n\n## 15. Tokenization and Embedding Pretraining\n\n#### BPE pair frequency\n\n**Equation**\n\n$$\n(a^*,b^*)=argmax_{(a,b)}count(a,b)\n$$\n\n**Plain-English explanation.** Merges the most frequent adjacent symbol pair.\n\n**Symbols.** `a,b` are symbols\u002Fsubwords.\n\n**Practical use case.** Byte-pair tokenization.\n\n**Source.** Sennrich et al. (2016).\n\n#### WordPiece score\n\n**Equation**\n\n$$\nscore(a,b)=\\frac{count(ab)}{count(a)count(b)}\n$$\n\n**Plain-English explanation.** Chooses merges using likelihood-inspired association score.\n\n**Symbols.** `ab` candidate merged token.\n\n**Practical use case.** BERT-style tokenization.\n\n**Source.** Schuster and Nakajima (2012); Devlin et al. (2019).\n\n#### Unigram LM tokenization\n\n**Equation**\n\n$$\nP(x)=\\prod_{t\\in seg(x)}p(t)\n$$\n\n**Plain-English explanation.** Assigns probability to a segmentation as product of token probabilities.\n\n**Symbols.** `t` subword token.\n\n**Practical use case.** SentencePiece unigram tokenization.\n\n**Source.** Kudo (2018).\n\n#### Unigram best segmentation\n\n**Equation**\n\n$$\nseg^*(x)=argmax_{s\\in S(x)}\\sum_{t\\in s}\\log p(t)\n$$\n\n**Plain-English explanation.** Finds most likely subword segmentation.\n\n**Symbols.** `\\mathcal{S}(x)` all segmentations.\n\n**Practical use case.** Tokenizing text with unigram models.\n\n**Source.** Kudo (2018).\n\n#### Skip-gram negative sampling\n\n**Equation**\n\n$$\nL=-\\log\\sigma(v_c^T v_w)-\\sum_{i=1}^{k}E_{n_i\\sim P_n}\\log\\sigma(-v_{n_i}^T v_w)\n$$\n\n**Plain-English explanation.** Predicts context words from a center word while contrasting negatives.\n\n**Symbols.** `v_w` center embedding; `v_c` context embedding.\n\n**Practical use case.** word2vec embeddings.\n\n**Source.** Mikolov et al. (2013).\n\n#### CBOW\n\n**Equation**\n\n$$\np(w_t| context)=softmax(W\\frac{1}{2m}\\sum_{-m\\leq j\\leq m,j\\ne0}v_{w_{t+j}})\n$$\n\n**Plain-English explanation.** Predicts a word from averaged context embeddings.\n\n**Symbols.** `m` context window radius.\n\n**Practical use case.** word2vec CBOW training.\n\n**Source.** Mikolov et al. (2013).\n\n#### GloVe\n\n**Equation**\n\n$$\nJ=\\sum_{i,j}f(X_{ij})(w_i^T\\tilde{w}_j+b_i+\\tilde{b}_j-\\log X_{ij})^2\n$$\n\n**Plain-English explanation.** Fits word vectors to log co-occurrence counts.\n\n**Symbols.** `X_{ij}` co-occurrence count.\n\n**Practical use case.** Static word embeddings.\n\n**Source.** Pennington et al. (2014).\n\n#### fastText subword embedding\n\n**Equation**\n\n$$\nv_w=\\sum_{g\\in G_w}z_g\n$$\n\n**Plain-English explanation.** Represents a word as sum of character n-gram embeddings.\n\n**Symbols.** `G_w` n-grams of word `w`.\n\n**Practical use case.** Robust embeddings for rare\u002Fmorphological words.\n\n**Source.** Bojanowski et al. (2017).\n\n## 16. Evaluation Metrics\n\n#### BLEU modified precision\n\n**Equation**\n\n$$\nBLEU=BP\\cdot\\exp(\\sum_{n=1}^{N}w_n\\log p_n)\n$$\n\n**Plain-English explanation.** Measures n-gram overlap with brevity penalty.\n\n**Symbols.** `p_n` modified n-gram precision; `BP` brevity penalty.\n\n**Practical use case.** Machine translation evaluation.\n\n**Source.** Papineni et al. (2002).\n\n#### BLEU brevity penalty\n\n**Equation**\n\n$$\nBP=1 \\; for \\; c>r,\\; BP=e^{1-r\u002Fc} \\; for \\; c\\leq r\n$$\n\n**Plain-English explanation.** Penalizes translations shorter than reference.\n\n**Symbols.** `c` candidate length; `r` reference length.\n\n**Practical use case.** Translation evaluation.\n\n**Source.** Papineni et al. (2002).\n\n#### ROUGE-N\n\n**Equation**\n\n$$\nROUGE-N=\\frac{\\sum_{S\\in Ref}\\sum_{gram_n\\in S}\\min(count_{cand}(gram_n),count_S(gram_n))}{\\sum_{S\\in Ref}\\sum_{gram_n\\in S}count_S(gram_n)}\n$$\n\n**Plain-English explanation.** Measures recall of reference n-grams.\n\n**Symbols.** `gram_n` n-gram.\n\n**Practical use case.** Summarization evaluation.\n\n**Source.** Lin (2004).\n\n#### ROUGE-L\n\n**Equation**\n\n$$\nROUGE-L=\\frac{(1+\\beta^2)R_{LCS}P_{LCS}}{R_{LCS}+\\beta^2P_{LCS}}\n$$\n\n**Plain-English explanation.** Uses longest common subsequence precision and recall.\n\n**Symbols.** `P_{LCS},R_{LCS}` LCS precision\u002Frecall.\n\n**Practical use case.** Summarization evaluation.\n\n**Source.** Lin (2004).\n\n#### METEOR\n\n**Equation**\n\n$$\nMETEOR=F_{mean}(1-Penalty)\n$$\n\n**Plain-English explanation.** Combines unigram precision\u002Frecall with fragmentation penalty.\n\n**Symbols.** `F_{mean}` weighted harmonic mean.\n\n**Practical use case.** Translation evaluation.\n\n**Source.** Banerjee and Lavie (2005).\n\n#### BERTScore precision\n\n**Equation**\n\n$$\nP=\\frac{1}{|\\hat{x}|}\\sum_{\\hat{x}_i}\\max_{x_j} \\hat{x}_i^T x_j\n$$\n\n**Plain-English explanation.** Matches candidate tokens to reference tokens in embedding space.\n\n**Symbols.** `\\hat{x}_i,x_j` contextual token embeddings.\n\n**Practical use case.** Semantic text evaluation.\n\n**Source.** Zhang et al. (2020).\n\n#### Exact match\n\n**Equation**\n\n$$\nEM=1[normalize(\\hat{y})=normalize(y)]\n$$\n\n**Plain-English explanation.** Checks whether normalized prediction exactly equals answer.\n\n**Symbols.** `\\hat{y}` prediction; `y` reference.\n\n**Practical use case.** QA evaluation.\n\n**Source.** Rajpurkar et al. (2016).\n\n#### Token F1\n\n**Equation**\n\n$$\nF1=\\frac{2PR}{P+R}\n$$\n\n**Plain-English explanation.** Harmonic mean of token precision and recall.\n\n**Symbols.** `P` precision; `R` recall.\n\n**Practical use case.** Extractive QA and answer matching.\n\n**Source.** Rajpurkar et al. (2016).\n\n#### pass@k\n\n**Equation**\n\n$$\npass@k=1-\\frac{\\binom{n-c}{k}}{\\binom{n}{k}}\n$$\n\n**Plain-English explanation.** Estimates probability that at least one of k samples passes tests.\n\n**Symbols.** `n` samples; `c` correct samples.\n\n**Practical use case.** Code generation evaluation.\n\n**Source.** Chen et al. (2021).\n\n#### MAUVE divergence curve\n\n**Equation**\n\n$$\nMAUVE=Area(\\{(e^{-D_{KL}(P|| Q_\\lambda)},e^{-D_{KL}(Q|| P_\\lambda)})\\}_{\\lambda})\n$$\n\n**Plain-English explanation.** Compares generated and real text distributions using divergence frontier.\n\n**Symbols.** `P,Q` text distributions; `\\lambda` mixture parameter.\n\n**Practical use case.** Open-ended text generation evaluation.\n\n**Source.** Pillutla et al. (2021).\n\n## 17. Production Inference Math\n\n#### Self-attention complexity\n\n**Equation**\n\n$$\nFLOPs_{attn}=O(n^2d)\n$$\n\n**Plain-English explanation.** Attention cost grows quadratically with sequence length.\n\n**Symbols.** `n` sequence length; `d` hidden dimension.\n\n**Practical use case.** Capacity planning for long-context LLMs.\n\n**Source.** Vaswani et al. (2017).\n\n#### MLP complexity\n\n**Equation**\n\n$$\nFLOPs_{MLP}\\approx 2ndd_{ff}\n$$\n\n**Plain-English explanation.** Feed-forward layers cost proportional to sequence length and intermediate dimension.\n\n**Symbols.** `d_{ff}` MLP hidden dimension.\n\n**Practical use case.** Estimating dense layer cost.\n\n**Source.** Transformer engineering literature.\n\n#### KV-cache memory\n\n**Equation**\n\n$$\nM_{KV}=2\\cdot L\\cdot B\\cdot S\\cdot H_{kv}\\cdot d_h\\cdot bytes\n$$\n\n**Plain-English explanation.** Memory needed to store cached keys and values.\n\n**Symbols.** `L` layers; `B` batch; `S` sequence length; `H_{kv}` KV heads.\n\n**Practical use case.** Sizing GPU memory for serving.\n\n**Source.** Transformer inference engineering literature.\n\n#### MQA\u002FGQA KV memory ratio\n\n**Equation**\n\n$$\n\\frac{M_{GQA}}{M_{MHA}}=\\frac{H_{kv}}{H_q}\n$$\n\n**Plain-English explanation.** KV-cache memory scales with KV heads, not query heads.\n\n**Symbols.** `H_{kv}` KV heads; `H_q` query heads.\n\n**Practical use case.** Choosing MQA\u002FGQA for latency and memory.\n\n**Source.** Shazeer (2019); Ainslie et al. (2023).\n\n#### Prefill latency\n\n**Equation**\n\n$$\nT_{prefill}\\approx\\frac{FLOPs_{prefill}}{FLOPs\u002Fs_{GPU}}+T_{overhead}\n$$\n\n**Plain-English explanation.** Estimates time to process prompt tokens.\n\n**Symbols.** `T_{overhead}` launch\u002Fscheduling overhead.\n\n**Practical use case.** TTFT budgeting.\n\n**Source.** LLM serving practice.\n\n#### Decode latency per token\n\n**Equation**\n\n$$\nT_{decode}\\approx\\frac{FLOPs_{decode}}{FLOPs\u002Fs_{GPU}}+\\frac{M_{read}}{BW}+T_{overhead}\n$$\n\n**Plain-English explanation.** Decode is often limited by memory reads and bandwidth.\n\n**Symbols.** `BW` memory bandwidth; `M_{read}` bytes read.\n\n**Practical use case.** Optimizing TPOT.\n\n**Source.** LLM serving practice.\n\n#### ","该项目是一个面向大型语言模型（LLM）、检索增强生成（RAG）和AI代理的数学参考手册。它涵盖了从概率基础到优化技术等多个方面的关键方程，旨在为开发人员提供全面的知识支持，特别是在准备面试或深入研究相关领域时。项目内容广泛，包括但不限于嵌入与Transformer结构、注意力机制变体、归一化方法、激活函数及门控MLP等核心概念。此外，还探讨了生产环境中推理相关的数学问题以及非Transformer序列模型等内容。无论是对于希望加深对现代自然语言处理技术理解的研究者，还是需要系统性学习这些技术以应对实际工作挑战的工程师而言，都是一个宝贵的资源。","2026-06-11 02:45:14","CREATED_QUERY"]