[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93016":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":23,"readmeContent":24,"aiSummary":25,"trendingCount":16,"starSnapshotCount":16,"syncStatus":17,"lastSyncTime":26,"discoverSource":27},93016,"HiLS-Attention","Tencent-Hunyuan\u002FHiLS-Attention","Tencent-Hunyuan","Official code for HiLS-Attention","",null,"Python",56,4,51,1,0,2,39.3,false,"main",true,[],"2026-07-22 04:02:07","# HiLS-Attention: Hierarchical Sparse Attention Done Right\n\nOfficial code for the paper [Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.02980).\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.02980\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2607.02980-b31b1b?logo=arxiv\" alt=\"arXiv\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FHiLS-Attention\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FGitHub-HiLS--Attention-181717?logo=github&logoColor=white\" alt=\"GitHub\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Ftencent\u002FHiLS-Attention-7B\u002Ftree\u002Fmain\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Hugging%20Face-HiLS--Attention--7B-ffcc4d\" alt=\"Hugging Face\">\u003C\u002Fa>\n\u003C\u002Fp>\n\nHiLS-Attention is a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling loss, enabling native sparse training for efficient long-context modeling.\n\n\n\n![HiLS-Attention Overview](.\u002Fassets\u002Fhils_attn_overview.png)\n\n*Figure: Overview of HiLS-Attention. Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses compressed chunk keys to estimate a chunk-mass surrogate and factorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.*\n\n## Performance\n\n![HiLS-Attention Performance](.\u002Fassets\u002Fperformances.png)\n\n*Figure: After only 50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages: **strong ultra-long context extrapolation** beyond the YaRN-extended 4× length (a) and **faster inference** (b). Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range (c & d).*\n\n## TODO\n\nWe are actively working on releasing more resources. Stay tuned!\n\n- [x] Release training and evaluation code\n- [x] Release pre-trained model checkpoints\n- [ ] Release SGLang inference code for efficient long-context serving\n\n## Environment Setup\n\n### Install via uv (recommended)\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FHiLS-Attention.git\ncd HiLS-Attention\n\n# install uv\ncurl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.sh | sh\n\nuv sync\nsource .venv\u002Fbin\u002Factivate\n```\n\n### Install via pip\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FTencent-Hunyuan\u002FHiLS-Attention.git\ncd HiLS-Attention\n\nconda create -n hils python=3.11 -y\nconda activate hils\n\npip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu128\npip install -r requirements.txt\n```\n\n\n\n## Training\n\n\n\n### Training from Scratch\n\n```bash\nexport CORPUS_PATH=\u002Fpath\u002Fto\u002Ftokenized\u002Fdata\nexport OUTPUT_DIR=outputs\u002Fcheckpoints\u002Fhils_attn_8KA2K_HoPE_345M_prop3p1_qcal_r64\nbash scripts\u002Fpretrain\u002F345M_exp_dist\u002Fpretrain_hils_attn_8KA2K_HoPE_345M_prop3p1_qcal_r64.sh\n```\n\n\n\n### Continue Pre-Training\n\n```bash\nexport MODEL_PATH=\u002Fpath\u002Fto\u002Fbase\u002Fhf_ckpt\nexport CORPUS_PATH=\u002Fpath\u002Fto\u002Ftokenized\u002Fdata\nexport OUTPUT_DIR=outputs\u002Fcheckpoints\u002Folmo3_8KA2K_HoPE_qcal\nbash scripts\u002Fcpt\u002Fcpt_olmo3_8KA2K_HoPE_qcal.sh\n```\n\nFor landmark-token tuning, use `MODEL_PATH` to point to the base checkpoint directory:\n\n```bash\nexport MODEL_PATH=\u002Fpath\u002Fto\u002Fbase\u002Fcheckpoint\nexport CORPUS_PATH=\u002Fpath\u002Fto\u002Ftokenized\u002Fdata\nexport OUTPUT_DIR=outputs\u002Fcheckpoints\u002Folmo3_8KA2K_lmk_token_tuning\nbash scripts\u002Fcpt\u002Fcpt_olmo3_8KA2K_lmk_token_tuning.sh\n```\n\n\n\n## Evaluation\n\n\n\n### Checkpoint Format\n\nTraining saves distributed checkpoints (DCP). The PPL and RULER examples below load DCP checkpoints directly. HF conversion is only needed for HuggingFace-based generation or evaluation.\n\nConvert DCP to HF format when needed:\n\n```bash\nDCP_PATH=\u002Fpath\u002Fto\u002Fcheckpoints\u002Fglobal_step_xxx \\\nbash scripts\u002Fckpt_transfer\u002Fdcp_hf_transfer.sh\n```\n\nThe converted checkpoint is saved under:\n\n```text\n\u002Fpath\u002Fto\u002Fcheckpoints\u002Fglobal_step_xxx\u002Fhf_ckpt\n```\n\n\n\n### Perplexity Evaluation\n\n```bash\npython eval\u002Feval_ppl.py \\\n  --config_path configs\u002Fhils_attention\u002Fconfig_hils_attn_8KA2K_HoPE_345M_prop3p1_qcal_r64.json \\\n  --checkpoint_path \u002Fpath\u002Fto\u002Fcheckpoints\u002Fglobal_step_30000 \\\n  --use_dcp_checkpoint \\\n  --data_path \u002Fpath\u002Fto\u002Ftokenized\u002Feval\u002Fdata \\\n  --max_seq_len 8192 \\\n  --last_k_tokens 512  # compute PPL on the last 512 tokens only\n```\n\n\n\n### RULER Evaluation\n\n```bash\npython eval\u002Feval_ruler.py \\\n  --config_path configs\u002Fhils_attention\u002Fconfig_hils_attn_8KA2K_HoPE_345M_prop3p1_qcal_r64.json \\\n  --checkpoint_path \u002Fpath\u002Fto\u002Fcheckpoints\u002Fglobal_step_30000 \\\n  --corpus_path \u002Fpath\u002Fto\u002Ftokenized\u002Feval\u002Fdata \\\n  --max_seq_len 8192 \\\n  --task_id 0  # 0: S-N, 1: MK-MQ, 2: VT\n```\n\n\n\n### OLMo3 7B Evaluations\n\nOLMo3-7B evaluations use **HF checkpoints** (`global_step_xxx\u002Fhf_ckpt`) and the Olmo3 tokenizer under `configs\u002Folmo3_vocab\u002F`. Convert DCP checkpoints first (see [Checkpoint Format](#checkpoint-format) above).\n\nEach batch script lives under `scripts\u002Feval\u002F`. Edit the `MODELS` \u002F `MODEL_NAMES` block at the top of the script to point to your checkpoint and HiLS config (`configs\u002Folmo3_7B\u002F*.json`). Logs are written to `scripts\u002Feval\u002Flogs\u002F`.\n\n#### 1. LongBench v1 (long-context QA)\n\n```bash\nbash scripts\u002Feval\u002Feval_olmo3_longbench_v1.sh\n```\n\n\n| Setting      | Default           | Notes                                                               |\n| ------------ | ----------------- | ------------------------------------------------------------------- |\n| `MAX_LENGTH` | `65536`           | Max input length (middle truncation)                                |\n| `DATASETS`   | all 21 tasks      | Set comma-separated names to run a subset, e.g. `\"hotpotqa,qasper\"` |\n| `GPU_IDS`    | `0 1 2 3 4 5 6 7` | One GPU per model; jobs are queued automatically                    |\n\n\nPer-task predictions and scores are saved under `scripts\u002Feval\u002Flogs\u002Feval_longbench_v1_\u003Ctimestamp>\u002F\u003Cmodel_name>\u002F`.\n\n#### 2. OpenCompass (short-context benchmarks)\n\n11 benchmarks covering knowledge, reasoning, and code. Uses the Transformers backend with a multi-GPU job queue (one dataset per GPU by default).\n\n**One-time setup** — clone [OpenCompass](https:\u002F\u002Fgithub.com\u002Fopen-compass\u002Fopencompass) and install:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fopen-compass\u002Fopencompass.git scripts\u002Feval\u002Fopencompass\nbash scripts\u002Feval\u002Finstall_opencompass.sh\nexport OPENCOMPASS_PATH=scripts\u002Feval\u002Fopencompass\nexport PYTHONPATH=$PWD:$OPENCOMPASS_PATH:$PYTHONPATH\n```\n\n**Run:**\n\n```bash\nbash scripts\u002Feval\u002Feval_olmo3_opencompass.sh\n```\n\n\n| Benchmark                                 | Type                  |\n| ----------------------------------------- | --------------------- |\n| MMLU, GPQA, HellaSwag, ARC-c, BoolQ, RACE | Few-shot PPL          |\n| GSM8K, CMath                              | Math generation (CoT) |\n| HumanEval+, MBPP+, CRUXEval-O             | Code generation       |\n\n\nResults land in `scripts\u002Feval\u002Flogs\u002Feval_olmo3_opencompass_\u003Ctimestamp>\u002F`. A LaTeX summary table is written to `summary.log`. HumanEval+ \u002F MBPP+ are re-scored with `evalplus` after inference.\n\nFor MBPP+, place the evalplus-format jsonl at `data\u002Fmbpp_plus\u002Fmbpp_plus.jsonl` (see `eval\u002Fconfigs\u002Fdatasets\u002Fmbpp_plus_gen.py`).\n\n#### 3. PPL + RULER (long-context probing)\n\nBatch PPL and RULER over multiple sequence lengths. PPL uses Dolma3 tokenized data; RULER runs tasks 0 (S-N), 1 (MQ-N), and 2 (VT).\n\n```bash\n# both PPL and RULER (default)\nbash scripts\u002Feval\u002Feval_olmo3_ruler_ppl.sh\n\n# PPL only or RULER only\nEVAL_MODE=ppl  bash scripts\u002Feval\u002Feval_olmo3_ruler_ppl.sh\nEVAL_MODE=ruler bash scripts\u002Feval\u002Feval_olmo3_ruler_ppl.sh\n```\n\n\n| Setting                                 | Default                                           | Notes                                        |\n| --------------------------------------- | ------------------------------------------------- | -------------------------------------------- |\n| PPL lengths                             | 64 … 256K                                         | Skips lengths ≤ `chunk_size` for HiLS models |\n| RULER lengths                           | 8K, 16K, 32K, 128K                                |                                              |\n| `PPL_DATA_PATH`                         | `..\u002F..\u002Fdata\u002Fdolma3_mix-6T-1025-partial-tokenized` | Tokenized eval corpus                        |\n| `PPL_LAST_K_TOKENS`                     | `512`                                             | PPL computed on last 512 tokens only         |\n| `PPL_MAX_SAMPLES` \u002F `RULER_MAX_SAMPLES` | `100` \u002F `50`                                      |                                              |\n\n\nLogs and a LaTeX summary (`summary.log`) are saved to `scripts\u002Feval\u002Flogs\u002Feval_olmo3_\u003Cmode>_\u003Ctimestamp>\u002F`.\n\n## Citation\n\n```bibtex\n@misc{hu2026hierarchicalsparseattentionright,\n      title={Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling}, \n      author={Xiang Hu and Xinyu Wei and Hao Gu and Minshen Zhang and Tian Liang and Huayang Li and Lei Zhu and Yan Wang and Sirui Han and Yushi Bai and Kewei Tu and Haitao Mi and Leo Liang},\n      year={2026},\n      eprint={2607.02980},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.02980}, \n}\n```\n\n","HiLS-Attention 是一种面向超长上下文建模的分层稀疏注意力机制，通过端到端学习块级选择策略，在保持语言建模目标一致性的同时显著降低计算开销。其核心创新在于使用压缩块键估计块重要性，并将注意力分解为块间与块内两阶段 softmax，避免全量 QK 计算；支持原生稀疏训练，兼具强长程外推能力（支持远超 YaRN 扩展长度的上下文）和推理加速。适用于需要高效处理万级至百万级 token 上下文的大模型训练与推理场景，如长文档理解、代码补全、多轮对话摘要等。","2026-07-11 02:30:40","CREATED_QUERY"]