[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-96090":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":32,"readmeContent":33,"aiSummary":34,"trendingCount":14,"starSnapshotCount":14,"syncStatus":35,"lastSyncTime":36,"discoverSource":37},96090,"pd-bridge","chadhurley25075-png\u002Fpd-bridge","chadhurley25075-png","Heterogeneous prefill\u002Fdecode for DeepSeek-V4-Flash: CUDA prefill (DGX Spark, vLLM) -> Metal decode (Mac Studio, oMLX) over plain 10GbE",null,"Python",109,8,102,0,7,43.56,"Apache License 2.0",false,"main",true,[22,23,24,25,26,27,28,29,30,31],"apple-silicon","deepseek","dgx-spark","heterogeneous-inference","kv-cache","llm-inference","mlx","omlx","prefill-decode","vllm","2026-09-20 04:01:32","# pd-bridge — heterogeneous prefill\u002Fdecode for DeepSeek-V4-Flash\n\n[![License: Apache-2.0](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Flicense-Apache--2.0-blue.svg)](LICENSE)\n[![status: reference implementation](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fstatus-reference%20implementation-orange.svg)](#status-honestly)\n[![model: DeepSeek-V4-Flash](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fmodel-DeepSeek--V4--Flash-8A2BE2.svg)](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FDeepSeek-V4-Flash)\n[![prefill: vLLM \u002F CUDA](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fprefill-vLLM%20%2F%20CUDA-76b900.svg)](#running-it)\n[![decode: oMLX \u002F Metal](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fdecode-oMLX%20%2F%20Metal-lightgrey.svg)](#running-it)\n\n**Prefill a 284B model on NVIDIA. Decode it on Apple Silicon. Over plain 10GbE.**\n\n![How the bridge works](docs\u002Farchitecture.svg)\n\nTwo production inference engines that share no cache format, no framework, no vendor and no\nquantization, serving one request together. DeepSeek-V4-Flash is 284B total \u002F 13B active,\n256 routed experts, MLA + sparse attention, 149 GB resident on the prefill side and 156 GB on\nthe decode side:\n\n- **Prefill:** 2× NVIDIA DGX Spark (GB10), vLLM TP2, official `deepseek-ai\u002FDeepSeek-V4-Flash` **FP8**\n- **Decode:** 1× Mac Studio M3 Ultra, oMLX, `DV4-Flash-MXFP4-MLX` (**MXFP4**)\n- **Link:** ordinary 10 gigabit Ethernet. No RDMA, no Thunderbolt.\n\n```\ncold prompt        Mac Studio alone    Sparks prefill -> Mac decode\n   ~25K tokens          42.6 s               28.2 s     1.5x\n   ~82K tokens         205.8 s               72.9 s     2.8x\n  ~105K tokens         245.6 s               75.5 s     3.3x\n  ~241K tokens         732.3 s              200.3 s     3.7x\ndecode rate unchanged (23-25 tok\u002Fs both ways); warm turns bypass the bridge (4.9 s \u002F 19.3 s at 241K)\n```\n\nEvery row verdict-checked, 2026-09-06. The bridged leg scores **5\u002F5 on the judged quality eval**,\nsame as native.\n\n**Since then the served window went 262,144 -> 2,097,152 and the ceiling moved with it.**\nA **700,630-token cold prompt now bridges end to end in 11 minutes 26 seconds** — `verdict complete`,\n342 blocks, 1,117 tok\u002Fs of prefill. That is **2.9x past the largest prompt in the table above**, and\nthe pair has accepted over a million.\n\n```\ncold prompt     engine   tok\u002Fs    end-to-end   blocks   verdict\n   192,099      119.2 s   1,611      143.1 s      93    complete\n   385,838      288.7 s   1,337      466.6 s     188    complete\n   677,069      596.8 s   1,134      655.9 s     330    complete\n   700,630      627.0 s   1,117      685.5 s     342    complete\n   709,055      635.4 s   1,116      694.6 s     345    partial 706,560\u002F709,055 (99.6%)\n   988,487    1,037.8 s     952    1,142.5 s     376    partial 770,048\u002F988,487 (78%)\n 1,006,172    1,070.5 s     940    1,171.7 s     346    partial 708,608\u002F1,006,172 (70%)\n```\n\nAll 2026-09-08, one prefill pair, same 10GbE. `partial` is not a failure: past roughly 772K tokens the\ncapture crosses the prefill box's free-memory floor and **seals a valid contiguous prefix** instead of\ndying; the decoder mounts what arrived and natively prefills the remainder. See *Known limits*.\n\n**We have not run a Mac-alone control at these sizes**, so those rows carry no ratio — they are what\nthe bridge does, not a claimed speedup. The measured ratios stop at 241K and are shown above.\n\nFull numbers and methodology: [RESULTS.md](RESULTS.md) · [bench\u002FBENCHMARK-PROTOCOL.md](bench\u002FBENCHMARK-PROTOCOL.md)\n\n\n---\n\n## The idea\n\nPrefill\u002Fdecode disaggregation is well established, and so is the hardware argument for it: prefill is\ncompute-bound, decode is memory-bandwidth-bound, so run each phase where it is cheapest. Existing\nsystems do this by **transferring the KV cache** from the prefill worker to the decode worker.\n\nThat requires both ends to agree on a cache format. Ours never can. One side is CUDA\u002FvLLM with an\nFP8 paged cache; the other is Metal\u002FMLX with its own block layout. Worse, DeepSeek-V4-Flash does not\nhave \"a KV cache\" — each layer carries a rotating 128-token window, a compressor pool (ratio 4 with\noverlap carry, and ratio 128), and an indexer pool, all with layer-dependent RoPE.\n\n**So we don't transfer a cache. We compute the decoder's finished cache on the prefill machine,\nusing the decoder's own weights, and write it straight into the decoder's prefix-cache store.**\n\nThe prefill engine already computes the exact tensor those pools are a pure function of — the\nattention input. A hook takes it there, applies the *Mac's* projection and pooling math on the GPU,\nand emits the finished pools. The decode side assembles them into MLX cache objects and hands them\nto oMLX's own block writer. oMLX then sees a normal prefix-cache hit and only decodes. Neither\nengine is modified in its hot path; the decoder does not know a bridge exists.\n\nPayload: **~10 KB per token** — 0.80 GB for an 81K-token prompt, pulled in 1.08 s. The network\nstopped being the bottleneck; the prefill engine is now 65% of wall time, which is where you want it.\n\n### Why the reconstruction is trustworthy\n\nThe whole design rests on the pooled tensors being *the same tensors* the decoder would have\ncomputed. That is tested, not assumed:\n\n| check | result |\n|---|---|\n| Cache arrays rebuilt on the Mac vs. a full native forward | **313\u002F313 bit-exact** |\n| Blocks written by the bridge vs. blocks oMLX writes itself | **11\u002F11 identical** (only the `created_at` stamp differs) |\n| Torch pooling port vs. MLX ground truth (T=23,217) | projections, window, carries **bit-exact**; pooled tensors 99.95–99.96% identical, worst delta **one bf16 ulp** |\n| In-container hook selftest (chunked == one-shot) | **52\u002F52** |\n| Needle retrieval through a fully reconstructed 81K cache | correct on every benchmark run |\n\n`studio\u002Fverify_blocks.py`, `studio\u002Fpd_diff_state.py`, `spark\u002Fpd_pool_validate.py` and\n`spark\u002Fpd_pool_selftest.py` reproduce these. Compare tensors, never file hashes — `created_at` means\na bridge-written block can never be byte-identical as a *file*.\n\n---\n\n## Known limits — read this before you run it\n\n### The memory floor, and why big prompts come back `partial` (2026-09-08)\n\nThe capture lives in the prefill box's memory for the whole request and costs about **11.8 KB of\nunified memory per token** on rank 0 only (rank 1 stays flat — it does its half of the attention and\nholds no capture). A GB10 has ONE 121 GB pool shared by weights, the vLLM KV arena and everything\nelse, so steady-state free memory with the model up is ~22 GB.\n\nMeasured: the capture crosses a 5.0 GB free-memory floor at roughly **772,000 tokens**. Past that the\nhook **seals a valid contiguous prefix `[0,T)` and reports it** rather than dying — the decoder mounts\nwhat arrived and natively prefills the tail. That is the `partial` verdict in the table above. A\n700,630-token wake cleared the floor with **100 MB to spare**; a 1,006,172-token wake sealed at 70%.\n\nThis replaced a real failure. Before the fix, `_finish()` copied every layer to the host **without\nfreeing the layers it had already written**, so the full device capture and the growing host copies\nwere alive at once. At 1,021,199 tokens it died after 11 of 43 layers and drove the box into swap\nthrash — no sshd even over a 200G fabric. It needed a physical power button. The hook now releases\neach layer as its file lands and checks `MemAvailable` every 64 layer-chunks.\n\n**If you are memory-tight, this is your limit, not the window.** The window is a config number; the\nfloor is physics on your box. Measure `MemAvailable` during a long prefill before trusting either.\n\n\n**The ~82K-token ceiling was a real bug. It is fixed.** (History kept because the failure mode is\ninstructive and the arithmetic still matters on smaller machines.)\n\n`omlx_block_writer` used to hold one *materialised cumulative* cache snapshot per 2048-token\nboundary until `finalize()`: peak memory grew **quadratically** with prompt length (~`N(N+1)\u002F2`\nblocks' worth of arrays — ~16 GB at 39 boundaries, ~23 GB at 47, which exhausted a 256 GB M3 Ultra\nholding a 156 GB model and wrote zero blocks at 97,848 tokens).\n\nThe writer now **streams**: each boundary is stored through oMLX's own pipeline and released the\nmoment it is snapshotted (`begin_stream`\u002F`store_boundary`; `finalize` drains and verifies). Peak\nmemory is ONE boundary snapshot (~20 MB × boundary index \u002F N — tens of MB, not tens of GB). It is\nvalidated against a real oMLX reference block (synthetic layout match) and live at 52 boundaries \u002F\n109,085 tokens — see RESULTS.md. `PD_STREAM_BOUNDARIES=0` restores the batched path for comparison.\n\n`PD_MAX_BRIDGE_TOKENS` remains as a configurable envelope guard, not a bug workaround: raise it to\nyour machine's measured headroom.\n\nTwo related behaviours worth knowing:\n\n- **The fallback works, and it can no longer lie.** When a bridge fails the reply still comes back\n  correct — the decoder serves natively. Since the bench4 autopsy (docs\u002FFINDING-bench4-cold-fallback.md),\n  every response carries an `X-PD-Bridge` verdict (`complete` \u002F `partial B\u002FT` \u002F declined with reason),\n  and `bench_cold.py` records it — a silent native fallback can never again enter a results table\n  as a bridged number.\n- **The \"cold-start variance\" at 20K was not variance.** The 25.5 s vs 55.4 s spread was the capture\n  hook flushing *mid-request* during chunked prefill (see the FINDING): the 55 s runs were native\n  fallbacks wearing a bridge label. The hook now guards its idle flush with a CUDA-event query and a\n  chunk-alignment check, and the front validates every capture manifest before trusting it.\n\n## Status, honestly\n\nThis is a **reference implementation, not a library.** It is pinned hard and it is young.\n\n- **One model.** DeepSeek-V4-Flash. The pooling math is specific to its sparse attention.\n- **Pinned stacks.** oMLX 0.6.4; vLLM 0.21.1rc1 with the DeepSeek-V4 plugin (sparkrun image).\n- **It monkey-patches private internals of both engines** — a `sitecustomize` hook onto\n  `DeepseekV4MultiHeadLatentAttentionWrapper.attention_impl` on the vLLM side, and a\n  filesystem-fallback patch to oMLX's `PagedSSDCacheIndex` on the MLX side (oMLX indexes SSD blocks\n  at model load only, so externally written blocks are otherwise invisible). **Expect this to break\n  when either project moves.**\n- **The judged quality eval is five questions on one document.** The bridged leg scores 5\u002F5 on it,\n  twice (once from a fresh cold v3 bridge), same as native. Prefill runs FP8 weights and decode runs\n  MXFP4, so bridged output is *not* token-identical to native; it is factually faithful on what we\n  checked, which is a smaller claim than \"equivalent\".\n- **The flush signal was unreliable until 2026-09-06 17:52 — fixed.** The hook's watcher ran in three\n  processes and two of them deleted the signal before the capturing worker saw it (~1 in 3 hit rate). Hook v5\n  fixes it; captures now close 0.6–0.9 s after the engine returns (4.8 s at 236K, which is the block write).\n  Autopsy: `docs\u002FFINDING-flush-signal-three-watchers.md`. Unit test: `spark\u002Ftest_flush_decision.py`.\n- **The front door is threaded, with one caveat.** HTTP handlers run in threads (health, model list and\n  oMLX passthrough answer immediately, and concurrent decodes overlap because the decoder batches them),\n  but every `bridge()` call — the MLX cache assembly — is marshalled to the main thread and runs one at a\n  time, because MLX streams are thread-local and the model lives there. Measured 2026-09-06: a short\n  request completed in 47 s while an 86K-token cold bridge was in flight, instead of waiting it out.\n  Two clients do slow each other down; they no longer block each other.\n- Only cold, long prompts benefit. Warm turns bypass the bridge by design and are served natively.\n\n**The transferable idea is bigger than this code:** when two engines cannot share a cache format,\ncompute the *consumer's* finished cache on the *producer*, using the consumer's weights. That\ngeneralizes past this model and this hardware, and it is the part worth stealing.\n\n---\n\n## Layout\n\n```\nspark\u002F    prefill side (NVIDIA \u002F vLLM)\n  capture_sitecustomize_v3.py   the hook: projections + pooling on the GPU, per-layer safetensors\n  pd_pool_torch.py              torch port of the decoder's pooling math (RoPE, compress, rmsnorm)\n  pd_pool_selftest.py           in-container selftest (chunked == one-shot)\n  pd_pool_validate.py           validate the port against MLX ground truth\n  pd-launch-v3.sh               launch vLLM with the hook (PD_HOOK=off for a control run)\n  pd_capture_http.py            Range-capable server so the decoder can stream captures\n  pd_share.py                   the threaded share the pooled path actually runs (SimpleHTTP drops connections under poll+fetch)\n  pd-hf-layout.sh               lay the checkpoint out as an HF hub dir inside the container mount\n  POOL-VALIDATION.md            what the validation numbers mean\n\nstudio\u002F   decode side (Apple Silicon \u002F oMLX)\n  pd_front.py                   OpenAI-compatible front door; orchestrates a request end to end\n  omlx_block_writer.py          drive oMLX's own store pipeline to emit prefix-cache blocks\n  pd_assemble_blocks.py         build MLX cache objects from a pooled capture, snapshot per boundary\n  pd_export_proj_weights.py     export the MLX projection weights the prefill hook needs\n  pd_export_pool_truth.py       MLX-computed ground truth for validating the torch port\n  pd_make_v3_from_mlx.py        build a v3 capture entirely in MLX (acceptance harness)\n  pd_capture_mlx.py             capture attention inputs natively (test fixture)\n  pd_rebuild_mlx.py             attention-only replay (the v1 path, kept for comparison)\n  verify_blocks.py              directory-vs-directory block comparison\n  pd_diff_state.py              cache-array diff against a full forward\n  test_block_writer_synthetic.py\n\nbench\u002F    bench_cold.py (records the X-PD-Bridge verdict), hetero (the one-command demo client),\n          BENCHMARK-PROTOCOL.md\ndocs\u002F     DESIGN-v3-pooled.md — the pooling math and the hook points, derived from oMLX's own code\n          FINDING-bench4-cold-fallback.md — the mid-request-flush autopsy; what broke and what it taught\n          FINDING-flush-signal-three-watchers.md — why the flush signal was consumed by the wrong worker\n          FINDING-stale-limits-after-a-window-change.md — READ THIS BEFORE RAISING YOUR WINDOW.\n              Five numbers sized against the old window that break silently after you raise it,\n              including the one that clamps every long-context answer to a single token.\n```\n\n## Don't have this hardware? Start on the rung you can reach\n\nEverything in this repo is written against the exact machines we ran, on purpose: if you have the same\ngear you get an exact replica and the numbers in RESULTS.md. If you don't, the idea is the same and the\nrecipe scales down. Three rungs, honestly labeled:\n\n| rung | prefill side | decode side | model | status |\n|---|---|---|---|---|\n| **A · exact replica** | 2× DGX Spark, TP2 over their direct 200G cable | Mac Studio M3 Ultra 256 GB | DeepSeek-V4-Flash (284B \u002F 13B active) | **measured** — everything in RESULTS.md |\n| **B · one Spark + any Apple Silicon Mac** | 1× DGX Spark (128 GB) | Mac Studio \u002F Mac mini \u002F **iMac** with ≥32 GB unified memory | a model that fits *both* boxes: the FP8 V4-Flash does **not** fit one Spark, so pick an MLA-latent model that does — DeepSeek-V2-Lite (16B) is the obvious first | recipe only, **unmeasured** |\n| **C · the kid's stack** | one used CUDA gaming card (8–24 GB) in a beat PC, vLLM or sglang | an M-series iMac \u002F MacBook with 16 GB | the smallest MLA-latent model that fits both | recipe only, **unmeasured** |\n\nWhat is identical across all three rungs: the front door, the verdict header, the cold\u002Fwarm decision, the\nblock writer path, the benchmark protocol, the wire (ordinary Ethernet — ~10 KB\u002Ftoken means even 1 GbE\nmoves a 30K-token prompt in ~0.3 s). What changes when you move down: the model, and therefore the pooling\nmath in the capture hook (DeepSeek-V4-Flash's hook is specific to its sparse attention; a plain-MLA model\nlike V2-Lite is *simpler* — its per-layer cache is the K\u002FV rows themselves). `docs\u002FPORTING.md` names the\nfour seams you touch and has a two-question feasibility test that takes ten minutes.\n\nWhat to expect at the bottom rung, honestly: the win is the ratio of prefill speeds. A used 3090 prefills a\n16B MLA model far faster than a 16 GB Mac does, so the shape of the result should hold; the absolute\nnumbers will be smaller because the prompts and models are smaller. **We have not run rungs B or C\nourselves.** They are the first ports we want to see, a negative result is a result, and we will feature\nwhoever lands one. Open an issue.\n\nThe point of this repo is that the privilege travels down. Take it apart.\n\n## Running it\n\n### Prerequisites\n\n| side | hardware | software | model |\n|---|---|---|---|\n| prefill | 2× NVIDIA DGX Spark (GB10, 128 GB each) on a 200G RoCE link (TP2) | Docker + the sparkrun vLLM image with the DeepSeek-V4 plugin (`aidendle94\u002Fsparkrun-vllm-ds4-gb10:production-ready`, vLLM 0.21.1rc1) | `deepseek-ai\u002FDeepSeek-V4-Flash` (official FP8 checkpoint, ~149 GB) |\n| decode | 1× Mac Studio M3 Ultra, 256 GB | oMLX 0.6.4 in a venv + the one-file patch in `studio\u002F` | an MLX MXFP4-experts \u002F MXFP8-attention conversion of `deepseek-ai\u002FDeepSeek-V4-Flash-0731` (~156 GB; any bit-exact conversion works — ours keeps the DSpark MTP heads) |\n| link | any Ethernet ≥10 GbE between the two | SSH key from the Mac to the prefill head; Python 3.10+ on both | — |\n\nThe official FP8 checkpoint is ~149 GB, so it does **not** fit one 128 GB Spark: prefill is tensor-parallel\nacross two Sparks over their direct ConnectX-7 link (the standard two-Spark cable — box to box, no switch\ninvolved). The only traffic that crosses to the Mac is HTTP over ordinary Ethernet, through whatever switch\nyou have. The numbers in RESULTS.md are TP2 over a 10 GbE LAN.\n\n**The Mac-side model.** We run a local, bit-exact MLX conversion of `deepseek-ai\u002FDeepSeek-V4-Flash-0731`\n(MXFP4 experts, MXFP8 attention, DSpark MTP heads kept). There is no single published id to point at, so\nproduce your own; the closest one-liner is\n\n```bash\nmlx_lm.convert --hf-path deepseek-ai\u002FDeepSeek-V4-Flash-0731 --mlx-path ~\u002Fmodels\u002FDV4-Flash-MXFP4-MLX \\\n  -q --q-mode mxfp4 --q-bits 4 --q-group-size 32\n```\n\n(unverified by us end to end — our build was a mixed conversion). What matters for the bridge is\n**self-consistency**, not which conversion: `make weights` exports the attention-projection weights from\n*your* MLX model, and the prefill hook uses exactly those, so the pooled tensors match whatever the decoder\nactually runs.\n\n```bash\ncp config.example.env config.env && $EDITOR config.env   # nothing has a working default\nsource config.env\n```\n\n**1. Export the decoder's projection weights** (on the Mac, in the oMLX venv). These are what the\nprefill hook uses, so that the pooled tensors match the decoder's arithmetic rather than the\nprefill engine's:\n\n```bash\n$OMLX_PYTHON studio\u002Fpd_export_proj_weights.py --model \"$PD_MODEL\" --out \"$PD_V3\"\n```\n\nCopy `$PD_V3` (`pd_pool_torch.py`, `dv4_proj_weights.*`, `capture_sitecustomize_v3.py`) to **both**\nprefill nodes.\n\n**2. Start the prefill pair** (rank 0 = TP head, rank 1 = worker):\n\n```bash\n.\u002Fspark\u002Fpd-launch-v3.sh 1     # worker first\n.\u002Fspark\u002Fpd-launch-v3.sh 0     # then head\npython3 spark\u002Fpd_share.py \"$PD_CAPTURE_DIR\" 8010        # pooled mode (the one the numbers use)\n# or: python3 spark\u002Fpd_capture_http.py --root \"$PD_CAPTURE_DIR\" --port 8010   # Range-capable; needed by the older 'hidden' pipelined mode\n```\n\n**3. Patch and start oMLX**, then the front door (on the Mac):\n\n```bash\n# oMLX must notice blocks written after model load — one small patch, applied once, in the oMLX venv:\nOMLX_PKG=$($OMLX_PYTHON -c 'import omlx,os;print(os.path.dirname(omlx.__file__))')   # ...\u002Fsite-packages\u002Fomlx\npatch -p0 -d \"$OMLX_PKG\u002Fcache\" \u003C \"$OLDPWD\u002Fstudio\u002Fomlx-0.6.4-paged_ssd_cache-disk-index-fallback.patch\"\n# then (re)start oMLX serving $PD_MODEL on :8011, and start the front door:\n$OMLX_PYTHON studio\u002Fpd_front.py          # listens on $PD_PORT (8012), OpenAI-compatible\ncurl -s localhost:8012\u002Fhealth             # {\"ok\": true, \"front\": \"pd\", ...}\n```\n\nPoint any OpenAI-compatible client at `:8012`. Prompts under `PD_MIN_TOKENS` or with fewer than\n`PD_MIN_TAIL` uncached tokens go straight to oMLX; longer cold prompts are prefilled on the Sparks. The\n`X-PD-Bridge` response header says which happened. `make doctor` checks every link in the chain.\n\n**4. Benchmark:**\n\n```bash\npython3 bench\u002Fbench_cold.py --chars 330000 --seed 301 --url http:\u002F\u002F\u003Cdecoder>:8012   # bridged\npython3 bench\u002Fbench_cold.py --chars 330000 --seed 302 --url http:\u002F\u002F\u003Cdecoder>:8011   # native\n```\n\n## Gotchas that cost us hours\n\n### The bridge was fluent and WRONG above the native window for days (2026-09-08)\n\n**This is the most important thing in this file if you are raising the context window.**\n\nThe capture hook builds the decoder's rope table from its own projection sidecar\n(`pd_v3\u002Fdv4_proj_weights.json`). When we grew the served window 1M -> 2M we updated the engine flags\nand the per-node model `config.json`, but not the sidecar. It kept **YaRN factor 16 \u002F\nmax_position 1,048,576** while the decoder ran **factor 32 \u002F 2,097,152**.\n\nYaRN is numerically **identical below `original_max_position_embeddings` (65,536)** and diverges\nabove it. So every test at or under ~65K passed, and everything above it was quietly corrupt —\n`verdict complete`, `position_gaps False`, manifest `T` matching the request, and the model unable to\nretrieve a midpoint needle. Measured before the fix: **MISS at 98,779 \u002F 148,134 \u002F 189,031 \u002F 287,842 \u002F\n385,838 — five for five.** After correcting the factor: **PASS at 189K, at 200K on a cold cache, and\nat 386K.**\n\n**The window lives in FOUR places. All of them must agree:**\n\n1. the launch parameters (`PD_MAX_MODEL_LEN`, `PD_ROPE_FACTOR`)\n2. each prefill node's model `config.json` `rope_scaling`\n3. **`pd_v3\u002Fdv4_proj_weights.json` `rope_scaling` + `max_position_embeddings`** — the one we missed\n4. the decoder's `config.json`\n\n**LAW: a window change is not done until a needle is retrieved THROUGH THE BRIDGE above the native\nwindow.** Our previously published \"verified ceiling\" had been measured on the *native* fallback path\n(the capture share happened to be down that day), so the bridged path's correctness at depth had\nnever actually been tested.\n\n**Two traps that produce false verdicts:**\n\n- **A reused seed reintroduces the bug you just fixed.** Our first post-fix retest MISSED and meant\n  nothing: the document shared its first ~100K tokens with an earlier run, so the decoder served\n  88,064 tokens of cached blocks built under the old rope. **After any change to rope, projection\n  weights or capture format, verify with a FRESH SEED.**\n- **The decoder's block cache has no notion of rope config.** It is keyed by token prefix only.\n  Fixing the config does not repair what is already cached — move the cache aside (rename, do not\n  delete) or you will keep serving the old geometry.\n\nAnd one in the harness itself: `needle.py --max-tokens` defaulted to **48**. When the model spends\nthat budget reasoning, `content` comes back empty and the harness scores a **MISS**. A 120K run\n\"failed\" at 48 and passed at 300 on the same document. Default raised to 300.\n\n\n- **Prefix caching must be OFF on the prefill engine.** With it on, vLLM skips a repeated document\n  prefix, the hook sees `43 layers x 0 tokens`, and the decoder waits forever for rows that will\n  never arrive. The decoder owns the caches; the prefill engine must compute every token it pools.\n- **`--enforce-eager`.** Prefill-only engine: CUDA graphs buy nothing and cost ~13 min of boot. The\n  first async version of the hook also invalidated vLLM's graph capture at startup\n  (`cudaErrorStreamCaptureInvalidated`); guard any hook with `torch.cuda.is_current_stream_capturing()`.\n- **Keep exactly one model build resident on the decoder.** Two builds in oMLX's pool exceeded the\n  admission target and cost a ~33 s evict-and-reload on every request that targeted the other one.\n  It looks exactly like a bridge regression and is not.\n- **Restart order matters.** Stop the old oMLX server and *wait for its shutdown line* before\n  starting a new one, or the new server's first load hits a memory settle barrier and aborts.\n- **`NCCL_IB_GID_INDEX` is fabric-specific.** Ours is 5; the common recipe says 3. Check `show_gids`.\n- **Same math is not the same bits.** Computing the 128-row window on a 128-row slice differs by one\n  bf16 ulp from taking those rows out of the 2048-row chunk matmul — MXFP4 kernel tiling. Slice from\n  the chunk computation. Relatedly, MLX's bf16 `sum` is serial for 8 rows and 32 strided bf16\n  partials combined in f32 for 128 rows; plain f32 accumulation matched only ~50% of elements.\n- **An idle-flush capture hook and chunked prefill are a dangerous pair.** With\n  `--max-num-batched-tokens 8192` the engine grinds ~4 s per chunk; a 2 s idle watcher fires\n  *between* chunks (or inside a chunk's enqueue burst) and ships a DONE manifest for a request that\n  is still running. Guard the flush with a CUDA-event query (GPU busy = mid-request) and a\n  chunk-alignment check (`calls % n_layers == 0`), and make the consumer validate `manifest T ==\n  request T` before trusting any DONE. This cost us a whole benchmark round: docs\u002FFINDING-bench4-cold-fallback.md.\n- **Benchmark with a real token budget.** A 64-token cap truncated answers mid-reasoning and read as\n  a retrieval failure on *both* paths.\n\n## Contributing\n\nThe most useful things anyone could add, roughly in order:\n\n1. ~~Kill the quadratic snapshot memory~~ — **done 2026-09-06** (streaming boundary store; see\n   *Known limits* and RESULTS.md). The bridge is validated to 109K tokens on a 256 GB machine.\n2. **A second model.** The bridge shape should generalize to any MLA\u002Fsparse-attention model whose\n   caches are a pure function of the attention input. Porting the pooling math is the work.\n3. **Stream the capture over a socket** instead of staging it on the prefill node's NVMe.\n4. **Ship the decoder's carry state to the prefill engine** so warm-but-extended prompts can prefill\n   only the tail instead of the whole thing.\n5. **Finish the judged quality eval** on the bridged leg (`bench\u002FBENCHMARK-PROTOCOL.md`).\n6. **Make the engine patches survive upstream.** Both would be better as small upstream hooks than\n   as monkey patches — a cache-rescan API on the oMLX side especially.\n\nBenchmark numbers in a PR must follow `bench\u002FBENCHMARK-PROTOCOL.md`, including a native baseline on\nthe same hardware and a warm engine on both legs.\n\n## Credits\n\noMLX for the decoder and its cache format; the vLLM DeepSeek-V4 plugin and the sparkrun GB10 image;\nEXO Labs, whose DGX Spark + Mac Studio prefill\u002Fdecode result set the reference point this builds on;\nand the Spark↔Mac USB4\u002FRDMA work that made joining the two silicon families look worth trying.\n\nApache-2.0.\n","pd-bridge 是一个支持异构预填充与解码的 LLM 推理桥接系统，实现 CUDA 加速的 vLLM 预填充（运行于 NVIDIA DGX Spark）与 Metal 加速的 oMLX 解码（运行于 Apple M3 Ultra Mac Studio）在普通 10GbE 网络上的协同推理。核心特点是跨框架、跨硬件、跨量化格式（FP8 vs MXFP4）的 KV 缓存序列化与传输，无需 RDMA 或专用互联，保持解码吞吐稳定（23–25 tok\u002Fs），显著缩短超长提示（最高 100 万+ token）端到端延迟。适用于需兼顾大规模预填充算力与低功耗\u002F本地化解码的混合部署场景，如科研级超长上下文服务、边缘协同推理原型验证。",2,"2026-09-09 02:30:12","CREATED_QUERY"]