[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92455":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":13,"stars30d":14,"stars90d":13,"forks30d":13,"starsTrendScore":13,"compositeScore":15,"rankGlobal":8,"rankLanguage":8,"license":16,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":19,"hasPages":17,"topics":20,"createdAt":8,"pushedAt":8,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":13,"starSnapshotCount":13,"syncStatus":24,"lastSyncTime":25,"discoverSource":26},92455,"antidoom","Liquid4All\u002Fantidoom","Liquid4All",null,"Python",288,24,69,0,157,51.19,"Apache License 2.0",false,"main",true,[],"2026-07-22 04:02:06","# Antidoom\n\nAntidoom generates and trains targeted preference data for reducing model\nrepetition loops (doom loops).\n\nIt is a narrow tool for a narrow failure mode: sample model completions, detect\nwhere a repeated span begins, mark the first loop-starting token as rejected,\nchoose coherent alternative next tokens, then train a LoRA adapter with Final\nToken Preference Optimization (FTPO).\n\nThe method adapts the single-token preference training idea from\n[Antislop](https:\u002F\u002Farxiv.org\u002Fabs\u002F2510.15061) and its\n[auto-antislop implementation](https:\u002F\u002Fgithub.com\u002Fsam-paech\u002Fauto-antislop) to\nthe specific problem of runaway repetition during reasoning.\n\n## Prerequisites\n\nAntidoom needs a supported GPU runtime for PyTorch and vLLM. NVIDIA\u002FCUDA is the\ndefault path; AMD\u002FROCm requires a separate environment.\n\n- **NVIDIA \u002F CUDA:** use the pinned `uv.lock` and\n  [`configs\u002Fdefault.yaml`](configs\u002Fdefault.yaml).\n\n- **AMD \u002F ROCm (Instinct MI-series):** do not use the default `uv.lock`; it is\n  CUDA-only. See [AMD \u002F ROCm](#amd--rocm) to create the ROCm venv and run\n  with [`configs\u002Fdefault_amd.yaml`](configs\u002Fdefault_amd.yaml), which adds the required\n  `attention_backend: TRITON_ATTN` and `optim: adamw_torch` overrides.\n\n## Quick Start\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FLiquid4All\u002Fantidoom\ncd antidoom\nuv sync\n```\n\nEdit [`configs\u002Fdefault.yaml`](configs\u002Fdefault.yaml) and set `model_name` to the\ncheckpoint you want to antidoom. The default config reads prompts from\n[LiquidAI\u002Fantidoom-mix-v1.0](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FLiquidAI\u002Fantidoom-mix-v1.0), generates FTPO pairs, trains a LoRA adapter, and\nmerges it.\n\nRun the full generate-and-train flow:\n\n```bash\nuv run antidoom -c configs\u002Fdefault.yaml -r runs\u002Fantidoom1 \\\n  --temp 0.01 \\\n  --model-name LiquidAI\u002FLFM2.5-1.2B-Base\n```\n\nThis writes generated completions and FTPO pairs to `runs\u002Fantidoom1\u002F`, trains\non the pair file, and writes the adapter plus merged model under the same run\ndirectory.\n\n## Why Doom Loops Happen\n\nDoom loops tend to appear when three forces line up:\n\n- **Overtrained tokens.** Common reasoning tokens such as `Wait`, `So`,\n  `But`, `Alternatively`, can become unusually attractive after heavy\n  synthetic reasoning training. When the model is uncertain, those tokens can\n  dominate the next-token distribution without moving the reasoning forward.\n- **Self-reinforcing context.** Once a short sequence appears, the prior context\n  makes that sequence more likely to appear again. Across repeated turns of the\n  same loop, the probability of each token can climb toward certainty.\n- **Low-temperature sampling.** At temperature 0 or near 0, the model keeps\n  selecting the highest-probability continuation, so a locally reinforced loop\n  has no natural escape route.\n\nAntidoom attacks the failure at the token where the loop begins. Instead of\ntraining on full gold answers, it trains only on the local preference:\ndo not choose the token that starts the repetition; choose one of the plausible\nalternatives available at that same position.\n\n## How It Works\n\nFor each prompt, Antidoom generates a completion and scans it for inner\nrepetition. When a loop is found, it refines the boundary in token space so the\nrejected token is the first readable token of the repeated segment.\n\nEach FTPO row contains:\n\n- a prompt\u002Fcontext prefix ending immediately before the rejected token\n- one rejected token, the token that begins the loop\n- one or more chosen tokens sampled from filtered alternatives at that position\n- metadata about the source prompt, generated completion, and detected loop\n\nThe training step regularises overrepresented rejected and chosen tokens so the\nadapter learns a broad anti-loop preference rather than simply suppressing one\nword.\n\n## Dataset\n\nThe default config uses\n[LiquidAI\u002Fantidoom-mix-v1.0](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FLiquidAI\u002Fantidoom-mix-v1.0),\na prompt-only ShareGPT mixture built for this pipeline.\n\nThe dataset intentionally excludes gold answers, rationales, hidden tests,\nverifier targets, and answer labels.\n\nDataset build scripts and license notes live in `datasets\u002F`.\n\n## Configure\n\nStart from [`configs\u002Fdefault.yaml`](configs\u002Fdefault.yaml). The most common\nfields to change are:\n\n- `model_name`: base checkpoint or Hugging Face model id\n- `generation.hf_dataset`: prompt dataset, or set `generation.input_jsonl`\n- `generation.prompt_field`: column to read from local\u002FHF rows\n- `generation.target_pairs`: number of FTPO pairs to collect\n- `generation.temperature` or `generation.temperatures`: sampling temperature\n- `train.max_train_examples`: maximum FTPO rows used for training\n- `train.output_dir` and `train.merged_output_dir`: adapter and merged model dirs\n\n`prompt_field` is authoritative: generation reads that column and parses the\nvalue by shape. A string is treated as a plain user prompt, OpenAI-style\n`[{\"role\": ..., \"content\": ...}]` lists are used as chat messages, and\nShareGPT-style `[{\"from\": ..., \"value\": ...}]` lists are converted to chat\nmessages. The parser does not infer prompt format from the column name.\n\n### Important Hyperparameters\n\n`max_train_examples`\n\nIt is important to generate enough preference samples before training. The\nnumber of samples produced depends on the number of prompts available to the\ngeneration step, the number of temperature passes you run, and how doom-loopy\nthe checkpoint is. As a rule of thumb, start with at least 15k prompts and aim\nto produce roughly 15k-20k preference rows.\n\nSet `max_train_examples` below the number of generated preference rows, for\nexample 12k rows from a 15k-20k generated set. This gives\n`rejected_regularisation_strength` room to shave off overrepresented rejected\ntokens. Regularisation matters: without it, the generated set can be badly\nunbalanced, which can create poor training outcomes.\n\nUsage hint: set `max_train_examples` to at most 70% of the actual number of\npreference rows in the generated dataset.\n\n`learning_rate`\n\nThe Antidoom trainer can undertrain or overtrain. If overtrained, the model can\ndegrade and produce more doom loops, so the right learning rate can take a\nlittle trial and error.\n\nUsage hint: `0.00001`-`0.00002` is a good starting range when training on about\n12k samples. Early stopping is useful to avoid overtraining.\n\n`early_stopping_chosen_win`\n\nThis stops training when `chosen_win` exceeds the configured value.\n`chosen_win` is the share of samples where the chosen token is winning against\nthe rejected token. Around `0.15`-`0.3`, you should usually see a strong\nreduction in doom looping. Training past `0.5` may be overtraining, though more\nablations are needed.\n\nUsage hint: try `0.4` and adjust from there.\n\n### Other Hyperparameters\n\n`generation.*`\n\nThese control the generation stage, where we try to elicit doom loops to build\nthe preference training set. The defaults are usually a reasonable starting\npoint.\n\n`rejected_regularisation_strength`\n\nThis flattens the distribution of rejected-token frequency by culling samples.\nSome rejected tokens occur much more frequently than others, because they are\nthe tokens that begin doom loops. If the training set is too unbalanced, those\ntokens can be suppressed too aggressively.\n\nUsage hint: the default `0.3` is a good starting point.\n\n`chosen_regularisation_strength`\n\nThis flattens the distribution of chosen-token frequency. If a chosen token is\ntoo overrepresented, its probability can be increased too much and it may start\ncausing loops of its own. This step usually prunes chosen alternatives rather\nthan whole samples, because each sample can contain multiple chosen tokens.\n\nUsage hint: the default `0.5` is a good starting point.\n\n`filter_rejected_stop_words`\n\nFilters samples where the rejected token is a stop word like `the` or `a`.\nLeave this set to `false`; common words can genuinely be loop-starting tokens\nand the regularisation pass is the safer way to handle frequency.\n\n`source_balance_mode`\n\nOptionally balances the training set by source after rejected-token\nregularisation. The default `off` is fine. Older configs may call this\n`balance_by_source_task`.\n\n`min_chosen_tokens`\n\nThe minimum number of chosen tokens required for a sample to be included. It\ncan sometimes be useful to require more than one chosen token to avoid mode\ncollapse on a single preferred token, but chosen-token regularisation already\nmitigates this and `1` is usually fine for Antidoom training.\n\n`lora_r`\n\nA higher LoRA rank than usual tends to improve learnability with less\ndegradation for this trainer. If in doubt, use `128` or `256`.\n\n`lora_alpha`\n\nSet this to the same value as `lora_r`, or half of `lora_r`.\n\n`lora_dropout`\n\nThis has not been heavily ablated, but `0` works well in current runs.\n\n`target_modules`\n\nPrevious Antislop ablations found that training all layers learned more easily\nbut could also degrade more easily; restricting to `up_proj`, `down_proj`, and\n`lm_head` helped there. Antidoom training appears to prefer training all layers.\nThe safe starting point is:\n\n```text\n[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\", \"lm_head\"]\n```\n\n`freeze_early_layers`\n\nThis can reduce model impact and degradation at the expense of learnability. In\npractice, Antidoom training has worked well with freezing turned off.\n\n`lambda_mse_target`\n\nStrength of the MSE tether to the reference model for the chosen and rejected\nlogits.\n\n`tau_mse_target`\n\nPenalty-free logit delta from the reference before the target-logit MSE loss\nkicks in.\n\n`lambda_mse`\n\nStrength of the MSE tether to the reference model for the remaining vocabulary,\nexcluding chosen and rejected logits. This is typically set higher than\n`lambda_mse_target`, because chosen\u002Frejected logits should move more freely\nwhile the rest of the vocabulary stays close to the reference.\n\n`clip_epsilon_logits`\n\nOnce the chosen token is beating the rejected token by this logit margin, the\npreference loss turns off.\n\n## Generate and Train\n\nBy default, `antidoom` is a one-shot pipeline. It generates preference rows\nuntil `generation.target_pairs` is satisfied, then trains. If the target pair\nJSONL already exists and has enough rows, generation returns immediately and\ntraining starts.\n\n```bash\nuv run antidoom -c configs\u002Fdefault.yaml -r runs\u002Fantidoom1 \\\n  --temp 0.01 \\\n  --model-name LiquidAI\u002FLFM2.5-1.2B-Base\n```\n\nWith `-r runs\u002Fantidoom1`, generation writes:\n\n```text\nruns\u002Fantidoom1\u002Fiter_0_generations.jsonl\nruns\u002Fantidoom1\u002Fiter_0_ftpo_pairs.jsonl\n```\n\nand training writes:\n\n```text\nruns\u002Fantidoom1\u002Flora\nruns\u002Fantidoom1\u002F\u003Cauto-named-merged-model-dir>\n```\n\n## Stage Commands\n\nRun generation only:\n\n```bash\nuv run antidoom -c configs\u002Fdefault.yaml -r runs\u002Fantidoom1 generate \\\n  --model-name LiquidAI\u002FLFM2.5-1.2B-Base\n```\n\nTrain from an existing pair file:\n\n```bash\nuv run antidoom -c configs\u002Fdefault.yaml -r runs\u002Fantidoom1 train \\\n  --dataset-jsonl runs\u002Fantidoom1\u002Fiter_0_ftpo_pairs.jsonl \\\n  --model-name LiquidAI\u002FLFM2.5-1.2B-Base\n```\n\nMerge an adapter:\n\n```bash\nuv run antidoom -c configs\u002Fdefault.yaml -r runs\u002Fantidoom1 merge \\\n  --model-name LiquidAI\u002FLFM2.5-1.2B-Base\n```\n\n## Filtering\n\nGeneration skips prompts that exceed `generation.max_prompt_tokens`. The generated preference dataset is regularized so that any given token does not occur too frequently in the *chosen* or *rejected* sets.\n\n## Outputs\n\nEach FTPO row contains fields like:\n\n```json\n{\n  \"source_task\": \"openai\u002Fgsm8k\",\n  \"source_sample_id\": \"1234\",\n  \"context_with_chat_template\": \"... text before rejected token ...\",\n  \"rejected_decoded\": \" Wait\",\n  \"multi_chosen_decoded\": [\" Therefore\", \" The\"],\n  \"repetition\": {\n    \"start_char\": 1024,\n    \"repeat_start_char\": 1526,\n    \"rejected_token_index\": 342\n  }\n}\n```\n\n`iter_0_generations.jsonl` stores the full generated completion and detector\nmetadata. `iter_0_ftpo_pairs.jsonl` stores the preference rows used for\ntraining.\n\n## AMD \u002F ROCm\n\nThe Quick Start above targets NVIDIA\u002FCUDA, whose pinned `uv.lock` is CUDA-only and can't drive AMD\nGPUs. On AMD you build a **separate ROCm venv** from vLLM's prebuilt ROCm wheels and run with\n[`configs\u002Fdefault_amd.yaml`](configs\u002Fdefault_amd.yaml) — whose key setting is\n`attention_backend: TRITON_ATTN`, since vLLM's default `ROCM_ATTN` kernel **memory-faults** on LFM2.\nRun it with `uv run --no-sync antidoom …` — the `--no-sync` is required: a bare `uv run` (as in the\nQuick Start) would re-sync the venv to the CUDA-only `uv.lock` and clobber the ROCm install.\n\nValidated on **Instinct MI325 (gfx942)**. vLLM's ROCm wheels support MI200\u002FMI300\u002FMI350 and Radeon\nRX 7900\u002F9000; see vLLM's\n[GPU installation guide](https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fgetting_started\u002Finstallation\u002Fgpu.html) for the\nsupport matrix and ROCm requirements (ROCm 6.3+, or 7.0+ for MI350).\n\n### Environment setup (ROCm venv)\n\nRequires ROCm 6.3+ on the host (`rocminfo` lists your GPU) and Python 3.12 (already pinned in\n`.python-version`). From the repo root, create the venv and install antidoom + ROCm torch\u002FvLLM in a\nsingle command — `--extra-index-url` gives the ROCm wheel index priority, so `torch`\u002F`vllm` resolve to\ntheir ROCm builds instead of the CUDA PyPI wheels\n([ref](https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fgetting_started\u002Fquickstart\u002F#installation)):\n\n```bash\nuv venv && uv pip install -e . --extra-index-url https:\u002F\u002Fwheels.vllm.ai\u002Frocm\u002F\n```\n\nVerify:\n\n```bash\nuv run --no-sync python -c \"import torch, vllm; print(torch.__version__, torch.version.hip, vllm.__version__)\"\n# torch should print a '+rocm...' build and a non-None torch.version.hip\n```\n\n### ROCm-specific settings (already in `configs\u002Fdefault_amd.yaml`)\n\n[`configs\u002Fdefault_amd.yaml`](configs\u002Fdefault_amd.yaml) is `default.yaml` with exactly these overrides:\n\n- **`generation.vllm_kwargs`**\n  ```yaml\n  vllm_kwargs:\n    enforce_eager: true            # eager is stable and fast on ROCm\n    attention_backend: TRITON_ATTN # the fix: ROCM_ATTN memory-faults on LFM2\n  ```\n  The `attention_backend` value is forwarded to the vLLM engine. The `VLLM_ATTENTION_BACKEND`\n  env var is a **no-op** in recent vLLM — it must be the engine arg, which `vllm_kwargs` forwards.\n- **`train.optim: adamw_torch`** — the NVIDIA default `paged_adamw_32bit` needs `bitsandbytes`\n  (painful on ROCm). `adamw_torch` is the same 32-bit AdamW math; its \"paged\" CPU-offload is\n  pointless on a 256 GB MI325.\n\nThe multi-GPU generate path additionally sets, per worker (in `generate.py`, no config needed):\n`HIP_VISIBLE_DEVICES` (ROCm doesn't reliably pin GPUs via `CUDA_VISIBLE_DEVICES` alone), per-worker\nnode-local `VLLM_CACHE_ROOT`\u002F`TORCHINDUCTOR_CACHE_DIR`\u002F`TRITON_CACHE_DIR` (shared JIT caches race →\n\"Memory access fault\"), and a probed free port for `VLLM_PORT` (avoids `EADDRINUSE`). These are\nno-ops on NVIDIA.\n\n### Run on AMD\n\nSame commands as the Quick Start, but with `configs\u002Fdefault_amd.yaml` and `uv run --no-sync` (the\n`--no-sync` stops `uv run` from re-syncing the venv back to the CUDA lock):\n\n```bash\nuv run --no-sync antidoom -c configs\u002Fdefault_amd.yaml -r runs\u002Fantidoom1 \\\n  --temp 0.01 \\\n  --model-name \u003Cyour-lfm2-checkpoint-or-hf-id>\n```\n\nGeneration uses one single-GPU vLLM engine per visible GPU; set `CUDA_VISIBLE_DEVICES` \u002F\n`HIP_VISIBLE_DEVICES` to choose GPUs. Training is single-GPU LoRA + merge.\n\nPrefer a container? vLLM also ships official ROCm images — `vllm\u002Fvllm-openai-rocm:latest` (stable) or\n`:nightly` — listed on the\n[GPU installation guide](https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Fstable\u002Fgetting_started\u002Finstallation\u002Fgpu.html). Run\nantidoom inside one with the **same `configs\u002Fdefault_amd.yaml`**; the `TRITON_ATTN` setting is still\nrequired.\n\n### AMD troubleshooting\n\n| symptom | cause | fix |\n|---|---|---|\n| `Memory access fault by GPU` mid-generation | vLLM `ROCM_ATTN` kernel bug on LFM2 | use `configs\u002Fdefault_amd.yaml` (`attention_backend: TRITON_ATTN`) — mandatory, affects all cudagraph modes |\n| `ImportError: You need to install bitsandbytes` during train | `optim: paged_adamw_32bit` | use `configs\u002Fdefault_amd.yaml` (`optim: adamw_torch`) |\n| `RuntimeError: Device string must not be empty` at vLLM init | `amdsmi` missing (ROCm platform not detected) | `cp -r \"${ROCM_PATH:-\u002Fopt\u002Frocm}\u002Fshare\u002Famd_smi\" \u002Ftmp\u002Famd_smi_build && uv pip install \u002Ftmp\u002Famd_smi_build` (copy first — the ROCm-shipped tree is root-owned) |\n\n## Citation\n\n```yaml\n@article{liquidAI2026Antidoom,\n    author = {Liquid AI},\n    title = {Reducing Doom Loops with Final Token Preference Optimization},\n    journal = {Liquid AI Blog},\n    year = {2026},\n    note = {www.liquid.ai\u002Fblog\u002Fantidoom}\n}\n```","Antidoom 是一个专用于缓解大语言模型推理过程中重复性循环（即“doom loops”）的轻量级训练工具。它通过采样模型输出、精准定位重复起始token、构建单token级偏好对（拒绝循环起始token，偏好合理替代token），并基于Final Token Preference Optimization（FTPO）训练LoRA适配器来实现干预。项目支持CUDA与ROCm双后端，依赖vLLM高效推理和PyTorch训练，强调局部、低开销、任务聚焦的偏好微调。适用于需长期推理、易陷入自我重复的开源语言模型（如LFM系列）的稳定性增强场景，尤其适合合成数据驱动的推理优化流程。",2,"2026-07-09 02:30:03","CREATED_QUERY"]