[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93143":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":10,"languages":10,"totalLinesOfCode":10,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":10,"rankLanguage":10,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":29,"readmeContent":30,"aiSummary":31,"trendingCount":14,"starSnapshotCount":14,"syncStatus":32,"lastSyncTime":33,"discoverSource":34},93143,"transformers-explained","amitshekhariitbhu\u002Ftransformers-explained","amitshekhariitbhu","Transformer architecture explained step by step - the full architecture, every attention variant, positional embeddings, and every layer inside a Transformer.","https:\u002F\u002Foutcomeschool.com\u002Fprogram\u002Fai-and-machine-learning",null,150,8,102,0,39,46.76,"Apache License 2.0",false,"main",true,[22,23,24,25,26,27,28],"architecture","attention-is-all-you-need","attention-mechanism","large-language-models","llm","llms","transformers","2026-07-22 04:02:08","\u003Cp align=\"center\">\n    \u003Cimg alt=\"Transformers Explained\" src=\"https:\u002F\u002Fgithub.com\u002Famitshekhariitbhu\u002Ftransformers-explained\u002Fblob\u002Fmain\u002Fassets\u002Fbanner.png\">\n\u003C\u002Fp>\n\n# Transformers Explained\n\n**Transformer architecture explained step by step - the full architecture, every attention variant, positional embeddings, and every layer inside a Transformer.**\n\n---\n\nPrepared and maintained by the **Founder** of Outcome School: [Amit Shekhar](https:\u002F\u002Fx.com\u002Famitiitbhu)\n\n---\n\n**Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.**\n\n---\n\n# Part 1: The Architecture\n\nWe start with the complete picture - every component of the Transformer and how data flows through it. The rest of this guide then zooms into each piece.\n\n---\n\n## Decoding Transformer Architecture\n\nIn this blog, we will learn about the Transformer architecture by decoding it piece by piece - understanding what each component does, how they work together, and why this architecture powers every modern Large Language Model (LLM).\n\nWe will cover the following:\n\n* Why the Transformer was needed\n* The two halves of the architecture\n* Tokenization, Embedding, and Positional Encoding\n* The Attention Mechanism and Multi-Head Attention\n* Feed-Forward Networks, Residual Connections, and Layer Normalization\n* How the Encoder and Decoder work\n* How data flows through the entire architecture\n* The three variants of the Transformer\n* Why the Transformer is so powerful\n\nLet's get started: [Decoding Transformer Architecture](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fdecoding-transformer-architecture)\n\n---\n\n# Part 2: Attention\n\nAttention is the heart of the Transformer. In this part, we go from the intuition to the exact math, and then through every attention variant used in modern models.\n\n---\n\n## Self Attention in Transformers\n\nIn this blog, we will learn about Self Attention in Transformers. We will understand what it is, how it works step by step, and why it is the heart of modern Large Language Models like BERT and GPT.\n\nWe will cover the following:\n\n* What is Self Attention?\n* Why do we need Self Attention?\n* Query, Key, and Value vectors\n* Step-by-step working of Self Attention\n* A simple example walk-through\n* Why Self Attention works so well\n* Multi-Head Self Attention\n* Where Self Attention is used\n\nLet's get started: [Self Attention in Transformers](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fself-attention-in-transformers)\n\n---\n\n## Math behind Attention - Q, K, and V\n\nIn this blog, we will learn about the math behind Attention - Query(Q), Key(K), and Value(V) with a step-by-step numeric example.\n\nWe will cover the following:\n\n* The Attention Formula\n* Setting Up: From Words to Vectors\n* Creating Q, K, and V Matrices\n* Computing Attention Scores (Q x K^T)\n* Scaling the Scores\n* Applying Softmax\n* Computing the Final Output (Attention Weights x V)\n* Putting It All Together\n\nLet's get started: [Math behind Attention - Q, K, and V](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fmath-behind-attention-qkv)\n\n---\n\n## Math behind √dₖ Scaling Factor in Attention\n\nIn this blog, we will learn about why we scale the dot product attention by √dₖ in the Transformer architecture with a step-by-step numeric example.\n\nWe will cover the following:\n\n* The Attention Formula (Quick Recap)\n* What Happens Without Scaling?\n* Why Do Dot Products Grow with dₖ?\n* Understanding Variance of the Dot Product\n* Proving It Step by Step: Variance of the Dot Product is dₖ\n* What Large Dot Products Do to Softmax\n* Why √dₖ is the Right Scaling Factor\n* Seeing It with Real Numbers\n* Putting It All Together\n\nLet's get started: [Math behind √dₖ Scaling Factor in Attention](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fscaling-dot-product-attention)\n\n---\n\n## Multi-Head Attention in Transformers\n\nIn this blog, we will learn about Multi-Head Attention in Transformers. We will understand what it is, how it works step by step, and why it gives Transformers their power to understand language so well.\n\nWe will cover the following:\n\n* What is Multi-Head Attention?\n* A quick recap of Self Attention\n* Why do we need Multi-Head Attention?\n* Step-by-step working of Multi-Head Attention\n* A simple example walk-through\n* Where Multi-Head Attention is used\n* Advantages of Multi-Head Attention\n\nLet's get started: [Multi-Head Attention in Transformers](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fmulti-head-attention-in-transformers)\n\n---\n\n## Cross Attention in Transformers\n\nIn this blog, we will learn about Cross Attention in Transformers. We will understand what it is, how it works step by step, how it is different from Self Attention, and where it is used.\n\nWe will cover the following:\n\n* What is Cross Attention?\n* Why do we need Cross Attention?\n* Query, Key, and Value in Cross Attention\n* Self Attention vs Cross Attention\n* Step-by-step working of Cross Attention\n* A simple example walk-through\n* Where Cross Attention is used\n* Importance of Cross Attention\n\nLet's get started: [Cross Attention in Transformers](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fcross-attention-in-transformers)\n\n---\n\n## Causal Masking in Attention\n\nIn this blog, we will learn about causal masking in attention.\n\nWe will start with the introduction of causal masking, understand the problem of seeing future tokens through an example, and then walk through its implementation to see how masked attention prevents the model from accessing future tokens.\n\nWe will cover the following:\n\n* Without Causal Masking\n* With Causal Masking\n* Implementation of Causal Masking\n* The Causal Mask Matrix\n\nLet's get started: [Causal Masking in Attention](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fcausal-masking-in-attention)\n\n---\n\n## Grouped Query Attention\n\nIn this blog, we will learn about Grouped-Query Attention (GQA) and how it differs from Multi-Head Attention (MHA). We will also learn about Multi-Query Attention (MQA) along the way and see when to use which one.\n\nWe will cover the following:\n\n* The Big Picture\n* Quick Recap: Multi-Head Attention (MHA)\n* The Problem with Multi-Head Attention\n* What is Multi-Query Attention (MQA)?\n* What is Grouped-Query Attention (GQA)?\n* How Grouped-Query Attention Works\n* GQA is a Generalization of MHA and MQA\n* GQA vs MHA vs MQA\n* Real-World Use Cases\n* A Note on Terminology\n* Uptraining: Converting MHA to GQA\n* Quick Summary\n\nLet's get started: [Grouped Query Attention](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fgrouped-query-attention)\n\n---\n\n## Decoding Flash Attention in LLMs\n\nIn this blog, we will learn about Flash Attention by decoding it piece by piece - understanding why standard attention is slow, what makes Flash Attention fast, how it uses GPU memory cleverly, and why it is used in almost every modern Large Language Model (LLM).\n\nWe will cover the following:\n\n* A quick recap of standard attention\n* Why standard attention is slow\n* How GPU memory actually works (HBM vs SRAM)\n* The core idea behind Flash Attention\n* Tiling: breaking the work into small blocks\n* Online softmax: computing softmax without the full matrix\n* Recomputation in the backward pass\n* Flash Attention 2\n* Flash Attention 3\n* Advantages and impact of Flash Attention\n\nLet's get started: [Decoding Flash Attention in LLMs](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fdecoding-flash-attention)\n\n---\n\n# Part 3: Position\n\nAttention by itself has no sense of order - it treats a sentence like a bag of tokens. Positional embeddings are how the Transformer knows which token came first.\n\n---\n\n## Positional Embeddings in LLMs\n\nIn this article, we will learn about Positional Embeddings in LLMs - how the Transformer gets the position information of every token in the sequence.\n\nLet's get started: [Positional Embeddings in LLMs](https:\u002F\u002Foutcomeschool.substack.com\u002Fp\u002Fpositional-embeddings-in-llms)\n\n---\n\n## Math Behind RoPE (Rotary Position Embedding)\n\nIn this blog, we will learn about the math behind Rotary Position Embedding (RoPE) and why it is used in modern Large Language Models.\n\nWe will cover the following:\n\n* The Big Picture\n* Why a Transformer Needs Position Information\n* Older Approaches and Their Problems\n* The Core Idea Behind RoPE\n* The 2D Rotation Math\n* How RoPE Is Applied to Q and K\n* Why the Dot Product Captures Relative Position\n* A Small Numeric Example\n* Real-World Use Cases\n* Quick Summary\n\nLet's get started: [Math Behind RoPE (Rotary Position Embedding)](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fmath-behind-rope-rotary-position-embedding)\n\n---\n\n# Part 4: The Other Layers\n\nAttention gets the fame, but a Transformer block is more than attention. The feed-forward network holds most of the parameters, and normalization keeps the whole thing trainable.\n\n---\n\n## Feed-Forward Networks in LLMs\n\nIn this blog, we will learn about Feed-Forward Networks in LLMs - understanding what they are, how they work inside the Transformer architecture, why every Transformer layer needs one, and what role they play in making Large Language Models so powerful.\n\nWe will cover the following:\n\n* What is a Feed-Forward Network?\n* Understanding Feed-Forward Networks with a Real-World Analogy\n* Where Does the Feed-Forward Network Sit in a Transformer?\n* How Does a Feed-Forward Network Work - Step by Step\n* The Expand-then-Contract Pattern\n* Why Does the FFN Expand and Then Contract?\n* ReLU and Activation Functions\n* What Does the Feed-Forward Network Actually Learn?\n* How Much of the Model is the Feed-Forward Network?\n* Feed-Forward Networks in Mixture of Experts\n* Why Feed-Forward Networks Are So Important\n\nLet's get started: [Feed-Forward Networks in LLMs](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Ffeed-forward-networks-in-llms)\n\n---\n\n## Batch Normalization vs Layer Normalization\n\nIn this blog, we are going to learn about Batch Normalization vs Layer Normalization. We will also see how Batch Normalization and Layer Normalization differ from each other and when to use which one.\n\nWe will cover the following:\n\n* What is Normalization?\n* Why do we need Normalization?\n* What is Batch Normalization?\n* What is Layer Normalization?\n* Batch Normalization vs Layer Normalization\n* When to use which one?\n\nLet's get started: [Batch Normalization vs Layer Normalization](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fbatch-normalization-vs-layer-normalization)\n\n---\n\n## RMSNorm (Root Mean Square Layer Normalization)\n\nIn this blog, we will learn about RMSNorm, a faster and simpler alternative to Layer Normalization that powers most modern Large Language Models like Llama, Mistral, Gemma, Qwen, PaLM, and DeepSeek.\n\nWe will cover the following:\n\n* Why normalization is needed in deep networks\n* A quick recap of Layer Normalization (LayerNorm)\n* What RMSNorm is and how it works\n* The math behind RMSNorm with a concrete numeric example\n* LayerNorm vs RMSNorm - the key differences\n* Why modern LLMs prefer RMSNorm\n* A code example\n* Where RMSNorm fits in a Transformer\n* Quick Summary\n\nLet's get started: [RMSNorm (Root Mean Square Layer Normalization)](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Frmsnorm-root-mean-square-layer-normalization)\n\n---\n\n# Part 5: Variants\n\nThe same core architecture stretches far beyond plain text models. Here are two variants you will meet everywhere: sparse expert models and Transformers for images.\n\n---\n\n## Mixture of Experts Explained\n\nIn this blog, we will learn about the Mixture of Experts (MoE) architecture - understanding what experts are, how the router picks them, why MoE makes large models faster and cheaper, and why it powers many of today's most powerful Large Language Models (LLMs).\n\nWe will cover the following:\n\n* Why Mixture of Experts was needed\n* What an \"expert\" really means\n* The router and how it picks experts\n* Where MoE sits inside a Transformer\n* Sparse activation and why it saves compute\n* Load balancing across experts\n* Advantages and challenges of MoE\n* Why MoE powers many modern LLMs\n\nLet's get started: [Mixture of Experts Explained](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fmixture-of-experts)\n\n---\n\n## Decoding Vision Transformer (ViT)\n\nIn this blog, we will learn about the Vision Transformer (ViT) by decoding how it splits an image into patches, turns those patches into tokens, and processes them with a transformer to classify the image.\n\nWe will cover the following:\n\n* The Big Picture\n* Decoding Step 1: Splitting the Image into Patches\n* Decoding Step 2: Patch Embedding\n* Decoding Step 3: The CLS Token\n* Decoding Step 4: Position Embeddings\n* Decoding Step 5: The Transformer Encoder\n* Decoding Step 6: The Classification Head\n* Putting It All Together\n* ViT vs CNN\n* Quick Summary\n\nLet's get started: [Decoding Vision Transformer (ViT)](https:\u002F\u002Foutcomeschool.com\u002Fblog\u002Fdecoding-vision-transformer-vit)\n\n---\n\n## More blogs and videos coming soon!\n\n### License\n```\n   Copyright (C) 2026 Outcome School\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n```\n","这是一个面向初学者和从业者的Transformer架构深度解析项目，系统讲解从整体结构到各组件（如自注意力、多头注意力、位置编码、层归一化等）的原理与实现细节。项目采用分步拆解方式，覆盖原始Transformer论文核心内容及主流变体，并辅以数学推导、图示说明和代码无关的概念阐释。适合机器学习工程师、NLP研究人员及AI学习者用于理解大语言模型底层机制、辅助教学或夯实基础理论知识。",2,"2026-07-12 02:30:02","CREATED_QUERY"]