[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-80019":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":16,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":13,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":15,"starSnapshotCount":15,"syncStatus":14,"lastSyncTime":29,"discoverSource":30},80019,"TIDE","PKU-YuanGroup\u002FTIDE","PKU-YuanGroup","Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models","https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.26951",null,"Python",69,3,2,0,1,1.81,"Other",false,"main",[22,23,24,25],"discrete-diffusion","distillation","llm","nlp","2026-06-12 02:03:57","\u003Cdiv align=\"center\">\n\u003Cimg src=\"assets\u002Flogo.gif\" alt=\"TIDE logo\" width=\"300px\">\n\u003C\u002Fdiv>\n\n\u003Ch1 align=\"center\">Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models\u003C\u002Fh1>\n\n\u003Ch5 align=\"center\">🌊 The first cross-architecture distillation framework for diffusion LLMs — 8B dense and 16B MoE teachers into a 0.6B student 🌊\u003C\u002Fh5>\n\n\u003Ch5 align=\"center\">\n\nGongbo Zhang\u003Csup>1\u003C\u002Fsup>  ·  Wen Wang\u003Csup>2\u003C\u002Fsup>  ·  Ye Tian\u003Csup>1\u003C\u002Fsup>  ·  Li Yuan\u003Csup>1,*\u003C\u002Fsup>\n\n\u003Csup>1\u003C\u002Fsup> Peking University  ·  \u003Csup>2\u003C\u002Fsup> Zhejiang University   (\u003Csup>*\u003C\u002Fsup> corresponding author)\n\n\u003C\u002Fh5>\n\n\u003Ch5 align=\"center\">\n\n[![arXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2604.26951-b31b1b.svg?logo=arxiv)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.26951)\n[![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-2ea44f)](https:\u002F\u002Fpku-yuangroup.github.io\u002FTIDE-Page\u002F)\n[![HF Paper](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗-Paper-orange)](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2604.26951)\n[![HF Models](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗-Models-blue)](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fmodels)\n[![HF Datasets](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗-Datasets-yellow)](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdatasets)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-blue.svg)](LICENSE)\n[![GitHub](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002FPKU-YuanGroup\u002FTIDE?style=social)](https:\u002F\u002Fgithub.com\u002FPKU-YuanGroup\u002FTIDE)\n\n\u003C\u002Fh5>\n\n\u003Cdiv align=\"center\">\nThis repository is the official implementation of \u003Cstrong>TIDE\u003C\u002Fstrong>, the first framework for \u003Cem>cross-architecture\u003C\u002Fem> dLLM distillation. While prior work focuses on step compression within a single architecture, TIDE bridges teachers and students that differ in \u003Cstrong>architecture\u003C\u002Fstrong>, \u003Cstrong>attention mechanism\u003C\u002Fstrong>, and \u003Cstrong>tokenizer\u003C\u002Fstrong>, via three modular components — \u003Cstrong>TIDAL\u003C\u002Fstrong>, \u003Cstrong>CompDemo\u003C\u002Fstrong>, and \u003Cstrong>Reverse CALM\u003C\u002Fstrong>.\n\u003C\u002Fdiv>\n\n\u003Cbr>\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002FFigure_teaser.png\" alt=\"TIDE: cross-architecture distillation overview\" width=\"100%\">\n\u003C\u002Fp>\n\n## ✨ Highlights\n\n1. **+1.53 average gain** over the non-distilled BD3LM baseline across 8 benchmarks (34.20 vs. 32.67).\n2. **+16.48 on HumanEval** over the equivalent-size AR baseline (48.78 vs. 32.30) — distilled dLLMs especially excel at code generation.\n3. **22× peak-memory reduction** vs. the 16B MoE LLaDA2 teacher (1.4 GB vs. 31.3 GB) and **5.2× faster inference** (6.25 s vs. 32.55 s for 256 tokens on H100), enabling commodity-hardware deployment.\n\n> All numbers reported in the paper — see [arxiv.org\u002Fabs\u002F2604.26951](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.26951) for full setup and ablations.\n\n## 🌊 The TIDE Framework\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002FFigure_main.png\" alt=\"TIDE framework: TIDAL + CompDemo + Reverse CALM\" width=\"100%\">\n\u003C\u002Fp>\n\n| Component | Paper | Role | One-line description |\n|---|:---:|---|---|\n| **TIDAL** | §2.1 | Scheduling — *when* to learn | Dual-axis interpolation along training-progress AND diffusion-timestep axes; deweights the teacher at high masking ratios where it is unreliable. Generalizes prior single-axis interpolation to the diffusion setting. |\n| **CompDemo** | §2.2 | Contextual — *what* to enrich | Two-pass teacher inference with complementary mask splits; every masked position sees ~50% revealed context, raising teacher signal quality at high noise. |\n| **Reverse CALM** | §2.3 | Output — *how* to project | Reverse-direction chunk-level binary cross-entropy for cross-tokenizer matching. Bounded gradient coefficient (depends only on the fixed teacher) and dual-end noise filtering; equivalent to a Bernoulli-KL mode-seeking objective. |\n\n## 🔄 Two Pipelines × Two Strategies\n\n> **Headline finding (§3.2): each pipeline favors its native strategy.**\n>\n> - **Cross-Tokenizer (LLaDA2 → BD3LM): native = TIDE-Cross** = Reverse CALM. Bounded-gradient mode-seeking tolerates the alignment noise from chunk-level cross-tokenizer matching. Beats the swapped TIDE-Shared by avg **+0.37**.\n> - **Shared-Tokenizer (WeDLM → BD3LM): native = TIDE-Shared** = TIDAL + CompDemo (over forward KL). Progressive scheduling and enriched signals work best when token-level alignment is exact. Beats the swapped TIDE-Cross by avg **+2.76**.\n\n| Pipeline | Teacher | Student | Tokenizer | Native strategy | Paper avg |\n|---|---|---|---|---|:---:|\n| **A — Cross-Tokenizer** | LLaDA2.0-mini (16B MoE) | Qwen3-0.6B-BD3LM | Cross (chunk align via tokenkit) | **TIDE-Cross** = Reverse CALM | **34.20** |\n| **B — Shared-Tokenizer** | WeDLM-8B-Instruct (8B dense) | Qwen3-0.6B-BD3LM | Shared (vocab 151646) | **TIDE-Shared** = TIDAL + CompDemo | 33.55 |\n\n## 📊 Main Results\n\nMain results across eight benchmarks. All distillation methods include a cross-entropy loss term. **Bold**: best among dLLM models; *italic*: second best.\n\n\u003Ctable>\n  \u003Cthead>\n    \u003Ctr>\n      \u003Cth align=\"left\" rowspan=\"2\">Benchmark\u003C\u002Fth>\n      \u003Cth align=\"center\" colspan=\"2\">\u003Cem>Qwen3-0.6B\u003C\u002Fem>\u003C\u002Fth>\n      \u003Cth align=\"center\" colspan=\"3\">\u003Cem>Shared-Tokenizer\u003C\u002Fem>\u003C\u002Fth>\n      \u003Cth align=\"center\" colspan=\"3\">\u003Cem>Cross-Tokenizer\u003C\u002Fem>\u003C\u002Fth>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Cth align=\"center\">AR\u003C\u002Fth>\n      \u003Cth align=\"center\">BD3LM\u003C\u002Fth>\n      \u003Cth align=\"center\">KL\u003C\u002Fth>\n      \u003Cth align=\"center\">TIDE-Cross\u003C\u002Fth>\n      \u003Cth align=\"center\">TIDE-Shared\u003C\u002Fth>\n      \u003Cth align=\"center\">CALM\u003C\u002Fth>\n      \u003Cth align=\"center\">TIDE-Shared\u003C\u002Fth>\n      \u003Cth align=\"center\">TIDE-Cross\u003C\u002Fth>\n    \u003C\u002Ftr>\n  \u003C\u002Fthead>\n  \u003Ctbody>\n    \u003Ctr>\n      \u003Ctd>GSM8K\u003C\u002Ftd>\n      \u003Ctd align=\"center\">59.60\u003C\u002Ftd>\n      \u003Ctd align=\"center\">45.56\u003C\u002Ftd>\n      \u003Ctd align=\"center\">43.97\u003C\u002Ftd>\n      \u003Ctd align=\"center\">45.03\u003C\u002Ftd>\n      \u003Ctd align=\"center\">48.98\u003C\u002Ftd>\n      \u003Ctd align=\"center\">48.60\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>49.89\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>52.24\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>MATH\u003C\u002Ftd>\n      \u003Ctd align=\"center\">32.40\u003C\u002Ftd>\n      \u003Ctd align=\"center\">13.08\u003C\u002Ftd>\n      \u003Ctd align=\"center\">9.40\u003C\u002Ftd>\n      \u003Ctd align=\"center\">9.76\u003C\u002Ftd>\n      \u003Ctd align=\"center\">11.16\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>13.14\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">12.98\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>13.20\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>BBH\u003C\u002Ftd>\n      \u003Ctd align=\"center\">41.50\u003C\u002Ftd>\n      \u003Ctd align=\"center\">26.32\u003C\u002Ftd>\n      \u003Ctd align=\"center\">25.79\u003C\u002Ftd>\n      \u003Ctd align=\"center\">26.00\u003C\u002Ftd>\n      \u003Ctd align=\"center\">26.79\u003C\u002Ftd>\n      \u003Ctd align=\"center\">24.21\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>26.85\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>27.37\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>MMLU-Pro\u003C\u002Ftd>\n      \u003Ctd align=\"center\">24.70\u003C\u002Ftd>\n      \u003Ctd align=\"center\">13.80\u003C\u002Ftd>\n      \u003Ctd align=\"center\">13.19\u003C\u002Ftd>\n      \u003Ctd align=\"center\">12.88\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>14.48\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">13.47\u003C\u002Ftd>\n      \u003Ctd align=\"center\">14.02\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>14.52\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>HellaSwag\u003C\u002Ftd>\n      \u003Ctd align=\"center\">47.40\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.28\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.78\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.50\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>40.50\u003C\u002Fstrong>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>40.42\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.57\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.88\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>MMLU\u003C\u002Ftd>\n      \u003Ctd align=\"center\">52.80\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.15\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.57\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.09\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>39.92\u003C\u002Fstrong>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.42\u003C\u002Ftd>\n      \u003Ctd align=\"center\">39.54\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>39.59\u003C\u002Fem>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>HumanEval\u003C\u002Ftd>\n      \u003Ctd align=\"center\">32.30\u003C\u002Ftd>\n      \u003Ctd align=\"center\">46.34\u003C\u002Ftd>\n      \u003Ctd align=\"center\">41.46\u003C\u002Ftd>\n      \u003Ctd align=\"center\">42.68\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>48.78\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">43.90\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>49.39\u003C\u002Fstrong>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">48.17\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>MBPP\u003C\u002Ftd>\n      \u003Ctd align=\"center\">36.60\u003C\u002Ftd>\n      \u003Ctd align=\"center\">37.80\u003C\u002Ftd>\n      \u003Ctd align=\"center\">31.20\u003C\u002Ftd>\n      \u003Ctd align=\"center\">31.40\u003C\u002Ftd>\n      \u003Ctd align=\"center\">37.80\u003C\u002Ftd>\n      \u003Ctd align=\"center\">34.80\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>38.40\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>38.60\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n    \u003Ctr>\n      \u003Ctd>\u003Cstrong>Avg\u003C\u002Fstrong>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">40.91\u003C\u002Ftd>\n      \u003Ctd align=\"center\">32.67\u003C\u002Ftd>\n      \u003Ctd align=\"center\">30.55\u003C\u002Ftd>\n      \u003Ctd align=\"center\">30.79\u003C\u002Ftd>\n      \u003Ctd align=\"center\">33.55\u003C\u002Ftd>\n      \u003Ctd align=\"center\">32.25\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cem>33.83\u003C\u002Fem>\u003C\u002Ftd>\n      \u003Ctd align=\"center\">\u003Cstrong>34.20\u003C\u002Fstrong>\u003C\u002Ftd>\n    \u003C\u002Ftr>\n  \u003C\u002Ftbody>\n\u003C\u002Ftable>\n\nSee the paper (§3.2) at [arxiv.org\u002Fabs\u002F2604.26951](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.26951) for the full discussion.\n\n## 🧭 Paper Variants ↔ Code Modes\n\nThis is the only place in the README where the legacy CLI strings `alm` \u002F `taid` appear, because the `--distill_mode` flag values include them.\n\n| Paper variant | Pipeline | Command | Notes |\n|---|:---:|---|---|\n| CALM (baseline, Cross-Tok) | A | `distill_llada2.sh --distill_mode alm` | — |\n| **TIDE-Cross (native, Cross-Tok)** | A | `distill_llada2.sh --distill_mode reverse_alm` | — |\n| TIDE-Shared (in Cross-Tok pipeline) | A | `distill_llada2.sh --distill_mode alm_taid --use_comp_demo True` | TIDAL + CompDemo |\n| KL (baseline, Shared-Tok) | B | `distill_wedlm.sh --distill_mode kl_aligned` | — |\n| **TIDE-Shared (native, Shared-Tok)** | B | `distill_wedlm.sh --distill_mode taid_aligned --use_comp_demo True` | TIDAL + CompDemo |\n| TIDE-Cross (in Shared-Tok pipeline) | B | `distill_wedlm.sh --distill_mode reverse_kl_aligned` | — |\n\n> 💡 **Note on combinations.** TIDAL is applied only to forward objectives. As discussed in the paper's gradient-analysis appendix, combining TIDAL with reverse objectives is counterproductive — the late-training $(1-\\lambda_t)$ factor suppresses the self-selection mechanism of Reverse CALM.\n\n## ⚙️ Setup\n\n```bash\n# Create environment\nconda create -n dllm python=3.10 -y && conda activate dllm\n\n# Install PyTorch (CUDA 12.4)\nconda install cuda=12.4 -c nvidia\npip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \\\n    --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu124\n\n# Install dllm\npip install -e .\n\n# Initialize submodules (lm-evaluation-harness + tokenkit)\ngit submodule update --init --recursive\n\n# Install eval harness\npip install -e \"lm-evaluation-harness[ifeval,math]\"\n\n# Install tokenkit (required for Pipeline A cross-tokenizer distillation)\npip install -e \"tokenkit[full]\"\n```\n\n## 📦 Released Models & Data\n\nSix distilled student checkpoints (3 per pipeline) are released under [🤗 TIDE-dllm Models](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fmodels), and two preprocessed SFT datasets are released under [🤗 TIDE-dllm Datasets](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdatasets).\n\n### Distilled student checkpoints\n\n| Pipeline | Variant | 🤗 Repo |\n|---|---|---|\n| A — Cross-Tokenizer (LLaDA2 teacher) | **TIDE-Cross** (native) | [`distill-LLaDA2-TIDE_Cross`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-LLaDA2-TIDE_Cross) |\n| A — Cross-Tokenizer (LLaDA2 teacher) | TIDE-Shared variant | [`distill-LLaDA2-TIDE_Shared`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-LLaDA2-TIDE_Shared) |\n| A — Cross-Tokenizer (LLaDA2 teacher) | CALM baseline | [`distill-LLaDA2-CALM`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-LLaDA2-CALM) |\n| B — Shared-Tokenizer (WeDLM teacher) | **TIDE-Shared** (native) | [`distill-WeDLM-TIDE_Shared`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-WeDLM-TIDE_Shared) |\n| B — Shared-Tokenizer (WeDLM teacher) | TIDE-Cross variant | [`distill-WeDLM-TIDE_Cross`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-WeDLM-TIDE_Cross) |\n| B — Shared-Tokenizer (WeDLM teacher) | KL baseline | [`distill-WeDLM-KL`](https:\u002F\u002Fhuggingface.co\u002FTIDE-dllm\u002Fdistill-WeDLM-KL) |\n\n### Preprocessed SFT datasets\n\nBoth datasets share the same composition as [`dllm-hub\u002FQwen3-0.6B-diffusion-bd3lm-v0.1`](https:\u002F\u002Fhuggingface.co\u002Fdllm-hub\u002FQwen3-0.6B-diffusion-bd3lm-v0.1) — `tulu-3-sft-mixture` + `smoltalk` + `opc-sft-stage1` + `opc-sft-stage2` — but tokenized for each teacher in advance to avoid NCCL timeouts during distillation.\n\n| Pipeline | 🤗 Repo |\n|---|---|\n| A — for the LLaDA2 teacher | [`distill_llada2_sft`](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FTIDE-dllm\u002Fdistill_llada2_sft) |\n| B — for the WeDLM teacher | [`distill_wedlm_sft`](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FTIDE-dllm\u002Fdistill_wedlm_sft) |\n\n### Download\n\n```bash\npip install \"huggingface_hub[cli]\"\n\n# Distilled checkpoint (example: native TIDE-Cross from Pipeline A)\nhuggingface-cli download TIDE-dllm\u002Fdistill-LLaDA2-TIDE_Cross \\\n    --local-dir ckpts\u002Fdistill-LLaDA2-TIDE_Cross\n\n# Preprocessed datasets\nhuggingface-cli download TIDE-dllm\u002Fdistill_llada2_sft \\\n    --repo-type dataset --local-dir data\u002Fdistill_llada2_sft\nhuggingface-cli download TIDE-dllm\u002Fdistill_wedlm_sft \\\n    --repo-type dataset --local-dir data\u002Fdistill_wedlm_sft\n```\n\nProject page: [pku-yuangroup.github.io\u002FTIDE-Page](https:\u002F\u002Fpku-yuangroup.github.io\u002FTIDE-Page\u002F).\n\n## 🚀 Quick Start\n\n### 1. Data Preprocessing\n\nDistillation requires offline-preprocessed data to avoid NCCL timeout during tokenization. **The fastest path is to download our preprocessed datasets** from `TIDE-dllm` (see [📦 Released Models & Data](#-released-models--data) above):\n\n```bash\nhuggingface-cli download TIDE-dllm\u002Fdistill_llada2_sft \\\n    --repo-type dataset --local-dir data\u002Fdistill_llada2_preprocessed\nhuggingface-cli download TIDE-dllm\u002Fdistill_wedlm_sft \\\n    --repo-type dataset --local-dir data\u002Fdistill_wedlm_preprocessed\n```\n\nIf you'd rather preprocess from scratch, the examples below use `tatsu-lab\u002Falpaca` for a quick smoke test. To reproduce the paper, replace the `--dataset` value with:\n\n```\nallenai\u002Ftulu-3-sft-mixture+HuggingFaceTB\u002Fsmoltalk+OpenCoder-LLM\u002Fopc-sft-stage1[lang:python]+OpenCoder-LLM\u002Fopc-sft-stage2[lang:python]\n```\n\n**Pipeline A (LLaDA2, cross-tokenizer):**\n```bash\nbash scripts\u002Fpreprocess_llada2_data.sh \\\n    --dataset tatsu-lab\u002Falpaca \\\n    --output_dir data\u002Fdistill_llada2_preprocessed\n```\n\n**Pipeline B (WeDLM, same-tokenizer):**\n```bash\nbash scripts\u002Fpreprocess_wedlm_data.sh \\\n    --dataset tatsu-lab\u002Falpaca \\\n    --output_dir data\u002Fdistill_wedlm_preprocessed\n```\n\n### 2. Distillation Training\n\nThe recommended command for each pipeline runs the **native strategy** (paper-best per §3.2).\n\n**Pipeline A — LLaDA2 teacher, TIDE-Cross (Reverse CALM):**\n```bash\nbash scripts\u002Fdistill_llada2.sh \\\n    --data_path data\u002Fdistill_llada2_preprocessed \\\n    --distill_mode reverse_alm \\\n    --num_gpus 8\n```\n\n**Pipeline B — WeDLM teacher, TIDE-Shared (TIDAL + CompDemo):**\n```bash\nbash scripts\u002Fdistill_wedlm.sh \\\n    --data_path data\u002Fdistill_wedlm_preprocessed \\\n    --distill_mode taid_aligned \\\n    --use_comp_demo True \\\n    --num_gpus 8\n```\n\n\u003Cdetails>\n\u003Csummary>📋 All training script parameters\u003C\u002Fsummary>\n\nBoth `distill_llada2.sh` and `distill_wedlm.sh` support:\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `--data_path` | *required* | Preprocessed data directory or HF dataset name |\n| `--output_dir` | `output\u002Fdistill_*` | Checkpoint output directory |\n| `--num_gpus` | `8` | Number of GPUs |\n| `--distill_mode` | `alm` \u002F `taid_aligned` | Distillation mode (see Paper Variants ↔ Code Modes table above) |\n| `--use_comp_demo` | `False` | Enable CompDemo (complementary demonstration) |\n| `--epochs` | `2` \u002F `3` | Number of training epochs |\n| `--lr` | `5e-5` | Learning rate |\n| `--batch_size` | `8` \u002F `10` | Per-device batch size |\n| `--student_model` | `dllm-collection\u002FQwen3-0.6B-diffusion-bd3lm-v0.1` | Student model |\n| `--teacher_model` | `inclusionAI\u002FLLaDA2.0-mini` \u002F `tencent\u002FWeDLM-8B-Instruct` | Teacher model |\n\nWeDLM-specific (TIDAL controls):\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `--taid_axis_mode` | `both` | TIDAL axis: `both`, `training_only`, `timestep_only` |\n| `--taid_timestep_weight` | `midrange` | Timestep weighting: `uniform`, `midrange` |\n| `--shared_vocab_size` | `151646` | Shared vocabulary size |\n| `--teacher_mask_token_id` | `151665` | Teacher mask token ID |\n\n\u003C\u002Fdetails>\n\n### 3. Evaluation\n\nRun all 8 benchmarks on a trained checkpoint:\n\n```bash\nbash scripts\u002Feval_all.sh --model_path \u002Fpath\u002Fto\u002Fcheckpoint --num_gpus 8\n```\n\nBenchmarks: `mmlu_generative_dream`, `mmlu_pro`, `hellaswag_gen`, `gsm8k_cot`, `bbh`, `minerva_math`, `humaneval_instruct`, `mbpp_instruct`.\n\n> Evaluation protocol: block size 32, CFG scale 0.0, sampling steps from 3 (HellaSwag\u002FMMLU) up to 256 (everything else). Results are saved to `eval_results\u002F` by default (override with `--output_dir`).\n\n## 📋 Training Hyperparameters\n\nTraining settings used for the paper experiments.\n\n| Parameter | Cross-Tokenizer (Pipeline A) | Shared-Tokenizer (Pipeline B) |\n|---|---|---|\n| Teacher | LLaDA2.0-mini (16B MoE) | WeDLM-8B-Instruct (8B) |\n| Student init | Qwen3-0.6B-BD3LM SFT v0.1 | Qwen3-0.6B-BD3LM SFT v0.1 |\n| Native method | Reverse CALM | TIDAL + CompDemo |\n| Learning rate | 5e-5 | 5e-5 |\n| Epochs | 10 | 10 |\n| Student \u002F teacher seq length | 512 \u002F 1024 | 512 \u002F 768 |\n| Block size | 32 | 32 |\n| Precision | bfloat16 | bfloat16 |\n| TIDAL $\\lambda_{\\text{init}} \\to \\lambda_{\\max}$ | — | $0.1 \\to 0.9$, cosine, midrange weighting |\n| CompDemo demo_ratio | — | 0.5 |\n| Temperature $T$ | — | 2.0 |\n| Dataset | Tulu-3 SFT + SmolTalk + OpenCoder-SFT-1\u002F2 (Python) | (same) |\n\n## 🛠️ Troubleshooting\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>\u003Ccode>ValueError: Sequence length N exceeds pad_to_length M\u003C\u002Fcode> during training\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nFor `*_aligned` modes (Pipeline B) the preprocessing script does **not** truncate samples to `--max_length` — it only filters samples whose prompt alone exceeds it. The training `--max_length` (and `--teacher_max_length`) must therefore be **at least as large as** the value used during preprocessing. The simplest rule: pass the same `--max_length` to both `preprocess_wedlm_data.sh` and `distill_wedlm.sh`.\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Pipeline B \u003Ccode>taid_aligned\u003C\u002Fcode> requires aligned preprocessed data\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nThe default `--align_mode` of `preprocess_wedlm_data.sh` is `kl_aligned`, which produces the dual-tokenizer fields (`teacher_input_ids`, `align_student`, `align_teacher`) needed by `*_aligned` training modes. If you preprocessed with `--align_mode none`, training in any `*_aligned` mode will crash with `KeyError: 'teacher_input_ids'`. Re-run preprocessing without overriding `--align_mode`.\n\u003C\u002Fdetails>\n\n## 📁 File Structure\n\n```\ndllm\u002Fcore\u002Ftrainers\u002F\n├── distill_bd3lm.py        # DistillBD3LMTrainer — all distillation modes (TIDAL, CompDemo, CALM, Reverse CALM, plus baselines)\n├── distill_collator.py     # DistillCollator — chunk-level CALM alignment via tokenkit (paper §2.3)\n├── bd3lm.py                # BD3LMTrainer (base block diffusion trainer)\n├── mdlm.py                 # MDLMTrainer (base masked diffusion trainer)\n└── losses\u002F\n    └── taid.py             # TIDAL loss implementation (paper §2.1)\n\nexamples\u002Fa2d\u002Fbd3lm\u002F\n├── distill.py              # Pipeline A entry: LLaDA2 cross-tokenizer distillation\n├── distill_wedlm.py        # Pipeline B entry: WeDLM same-tokenizer distillation\n├── distill_utils.py        # Shared utilities (alignment, tokenization)\n├── preprocess_distill_data.py       # Data preprocessing for Pipeline A\n└── preprocess_distill_wedlm_data.py # Data preprocessing for Pipeline B\n\nscripts\u002F\n├── distill_llada2.sh       # One-click training: Pipeline A\n├── distill_wedlm.sh        # One-click training: Pipeline B\n├── eval_all.sh             # One-click evaluation (8 benchmarks)\n├── preprocess_llada2_data.sh   # One-click preprocessing: Pipeline A\n└── preprocess_wedlm_data.sh    # One-click preprocessing: Pipeline B\n```\n\n## 📝 Citation\n\nIf you find TIDE useful for your research, please consider citing:\n\n```bibtex\n@misc{zhang2026turningtidecrossarchitecturedistillation,\n      title={Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models},\n      author={Gongbo Zhang and Wen Wang and Ye Tian and Li Yuan},\n      year={2026},\n      eprint={2604.26951},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.26951},\n}\n```\n\n## 🙏 Acknowledgements\n\nBuilt on the [dLLM](https:\u002F\u002Fgithub.com\u002FZHZisZZ\u002Fdllm) library; cross-tokenizer alignment via [tokenkit](https:\u002F\u002Fgithub.com\u002FNobody-Zhang\u002Ftokenkit); evaluation through [lm-evaluation-harness](https:\u002F\u002Fgithub.com\u002FEleutherAI\u002Flm-evaluation-harness).\n","TIDE 是一个用于扩散大语言模型的跨架构蒸馏框架，能够将80亿参数密集型和160亿参数混合专家系统教师模型的知识提炼到一个0.6亿参数的学生模型中。该项目通过TIDAL、CompDemo和Reverse CALM三个模块化组件，在不同架构、注意力机制及分词器之间搭建知识传递桥梁。TIDE在多个基准测试上相比非蒸馏基线平均提升了1.53个点，并且在代码生成任务上表现尤为出色，同时实现了22倍的峰值内存减少和5.2倍的推理加速，使得大模型能够在普通硬件上部署运行。适用于需要优化大型语言模型资源消耗或提升特定任务性能的场景。","2026-06-11 03:58:55","CREATED_QUERY"]