[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93508":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":8,"htmlUrl":8,"language":9,"languages":8,"totalLinesOfCode":8,"stars":10,"forks":11,"watchers":12,"openIssues":13,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":13,"stars7d":14,"stars30d":14,"stars90d":13,"forks30d":13,"starsTrendScore":14,"compositeScore":15,"rankGlobal":8,"rankLanguage":8,"license":16,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":17,"hasPages":17,"topics":19,"createdAt":8,"pushedAt":8,"updatedAt":20,"readmeContent":21,"aiSummary":8,"trendingCount":13,"starSnapshotCount":13,"syncStatus":22,"lastSyncTime":23,"discoverSource":24},93508,"SEED","jinyangwu\u002FSEED","jinyangwu",null,"Python",110,3,102,0,6,42.41,"MIT License",false,"main",[],"2026-07-23 04:02:10","\u003Ch1 align=\"center\">\n\u003Cimg src=\"figs\u002Fseed_icon.svg\" alt=\"SEED icon\" width=\"42\" style=\"vertical-align:middle;\">\nSEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning\n\u003C\u002Fh1>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fjinyangwu.github.io\u002Fseed\u002F\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-1F6FEB?style=for-the-badge&logo=googlechrome&logoColor=white\" alt=\"Project Page\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2607.14777\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FHF-Paper-FFD21E?style=for-the-badge&logo=huggingface&logoColor=111827\" alt=\"Hugging Face Paper\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.14777\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-Paper-B31B1B?style=for-the-badge&logo=arxiv&logoColor=white\" alt=\"arXiv Paper\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FJinyang23\u002FSeed-AlfWorld-3B\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FModel-Checkpoint-F59E0B?style=for-the-badge&logo=huggingface&logoColor=white\" alt=\"Model Checkpoint\">\n  \u003C\u002Fa>\n\u003C\u002Fp>\n\n## News\n\n- **2026-07-16**: We have released our paper and code.\n\nIf you have any questions ❓ or are interested in collaboration 🤝, please feel free to contact me at \nwu-jy23@mails.tsinghua.edu.cn.\n\n## Overview\n\n**SEED** is a **Self-Evolving On-Policy Distillation** framework for long-horizon\nLLM agents. Its core idea is to make one policy checkpoint play two synchronized\nroles: it acts in the environment to collect on-policy trajectories, then analyzes\nits own completed trajectories into hindsight skills, and finally distills the\nskill-induced change in action probabilities back into the ordinary policy. After\neach update, the improved policy becomes the next analyzer and generates the next\nround of hindsight skills, so the decision policy and hindsight supervision\nevolve together.\n\nSEED has two stages:\n\n1. **Hindsight-skill SFT.** Collect ordinary agent trajectories, annotate each\n   completed trajectory with an episode-level hindsight skill, and fine-tune the\n   backbone so the same model can analyze trajectories.\n2. **Self-evolving OPD during RL.** At each RL update, the frozen current policy\n   both samples on-policy trajectories and serves as the synchronized analyzer\n   that extracts hindsight skills. The same sampled action tokens are re-scored\n   under ordinary and skill-augmented contexts. The skill-induced log-probability\n   shift gates a dense OPD loss, which is optimized jointly with GRPO.\n\nAt inference time, SEED uses only the learned policy. It requires no analyzer,\nno skill bank, no retrieval module, and no skill-augmented prompt at deployment.\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\"figs\u002Fpipeline.png\" alt=\"SEED pipeline\" style=\"width:100%;\">\n  \u003Cbr>\n  \u003Cem>Figure 1: Overview of SEED.\u003C\u002Fem>\n\u003C\u002Fdiv>\n\n## Main Results\n\nAcross ALFWorld, Search-based QA, and WebShop, the experiments show three main\nfindings:\n\n1. **Dense hindsight supervision improves outcome-only RL.** SEED consistently\n   outperforms GRPO by converting trajectory-level hindsight into token-level\n   OPD signals.\n2. **Internalizing skills is better than prompting with skills.** SEED uses\n   hindsight skills only during training and still outperforms skill-prompted\n   evaluation baselines.\n3. **Self-evolving distillation beats static distillation.** Refreshing the\n   analyzer from the latest policy keeps hindsight supervision aligned with the\n   policy's evolving behaviors and failure modes.\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\"figs\u002Fresults.png\" alt=\"SEED results\" style=\"width:100%;\">\n  \u003Cbr>\n  \u003Cem>Figure 2: Main results.\u003C\u002Fem>\n\u003C\u002Fdiv>\n\n## Installation\n\n### Install veRL\n\n```bash\nconda create -n seed python==3.12 -y\nconda activate seed\n\npip3 install vllm==0.11.0\npip3 install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir\npip install -e .\n```\n\n### Install Supported Environments\n\n#### 1. ALFWorld\n\n```bash\npip3 install gymnasium==0.29.1\npip3 install stable-baselines3==2.6.0\npip3 install alfworld\n```\nDownload PDDL & Game files and pre-trained MaskRCNN detector (will be stored in ~\u002F.cache\u002Falfworld\u002F):\n```bash\nalfworld-download -f\n```\n#### 2. WebShop\n\nWebShop requires Python \u003C= 3.10, so begin by creating a separate environment:\n\n```bash\nconda create -n seed-webshop python==3.10 -y\nconda activate seed-webshop\n\ncd .\u002Fagent_system\u002Fenvironments\u002Fenv_package\u002Fwebshop\u002Fwebshop\n.\u002Fsetup.sh -d all\n\ncd repo_root\u002F\npip3 install torch==2.6.0 --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu124\npip3 install flash-attn==2.7.4.post1 --no-build-isolation\npip3 install -e .\npip3 install vllm==0.8.2\n```\n\n\n#### 3. Search-Based QA\n\n```bash\ncd .\u002Fagent_system\u002Fenvironments\u002Fenv_package\u002Fsearch\u002Fthird_party\npip install -e .\npip install gym==0.26.2\n```\n\nPrepare the Search-R1 style dataset:\n\n```bash\ncd repo_root\u002F\npython examples\u002Fdata_preprocess\u002Fpreprocess_search_r1_dataset.py\n```\n\nThe processed data is saved under `~\u002Fdata\u002FsearchR1_processed_direct` by default.\n\nBuild a separate retrieval environment for the local search server:\n\n```bash\nconda create -n retriever python=3.10 -y\nconda activate retriever\n\nconda install numpy==1.26.4\npip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu124\npip install transformers datasets pyserini huggingface_hub\nconda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y\npip install uvicorn fastapi\n```\n\nDownload the index:\n\n```bash\nconda activate retriever\n\nlocal_dir=~\u002Fdata\u002FsearchR1\npython examples\u002Fsearch\u002Fsearchr1_download.py --local_dir $local_dir\ncat $local_dir\u002Fpart_* > $local_dir\u002Fe5_Flat.index\ngzip -d $local_dir\u002Fwiki-18.jsonl.gz\n```\n\nStart the local flat e5 retrieval server:\n\n```bash\nconda activate retriever\n\nbash examples\u002Fsearch\u002Fretriever\u002Fretrieval_launch.sh > retrieval_server.log\n```\n\n## Training\n\nThe training and data-construction scripts load environment variables from the\nrepo-level `.env` file by default. Put model paths, analyzer endpoints, and\noptional logging keys there before running the scripts:\n\n```bash\n# Runtime devices and logging.\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\nWANDB_MODE=offline\n\n# Local model and data roots.\nMODELS_ROOT=\u002Fpath\u002Fto\u002Fmodels # the parent dir of models\nDATA_ROOT=\u002Fpath\u002Fto\u002Fdata-root # used for search-qa data\n\n# Used by Stage 1 hindsight-skill annotation.\nOPENAI_API_KEY=your_key_here\nOPENAI_BASE_URL=https:\u002F\u002Fyour-openai-compatible-endpoint\u002Fv1\nOPENAI_MODEL=glm-5.2\nOPENAI_API_RETRIES=5\nOPENAI_API_RETRY_DELAY=1.0\n```\n\n\n\n### Stage 1: Build Hindsight-Skill SFT Checkpoints\n\nThe paper-style workflow first builds episode-level skill SFT data and trains an\nanalyzer-capable policy checkpoint.\n\n```bash\n# ALFWorld\nbash scripts\u002Fsft\u002Falfworld\u002Fprepare_data.sh\nbash scripts\u002Fsft\u002Falfworld\u002Ftrain_sft.sh\n\n# WebShop\nbash scripts\u002Fsft\u002Fwebshop\u002Fprepare_data.sh\nbash scripts\u002Fsft\u002Fwebshop\u002Ftrain_sft.sh\n\n# Search-based QA\nbash scripts\u002Fsft\u002Fsearch\u002Fprepare_data.sh\nbash scripts\u002Fsft\u002Fsearch\u002Ftrain_sft.sh\n\n# EZPoints\nbash scripts\u002Fsft\u002Fezpoints\u002Fprepare_data.sh\nbash scripts\u002Fsft\u002Fezpoints\u002Ftrain_sft.sh\n\n# Sokoban\nbash scripts\u002Fsft\u002Fsokoban\u002Fprepare_data.sh\nbash scripts\u002Fsft\u002Fsokoban\u002Ftrain_sft.sh\n```\n\nThe default scale follows the paper: 180 tasks and 8 rollouts per task. The SFT\nscripts train for 3 epochs and export Hugging Face checkpoints under\n`$MODELS_ROOT`. All five datasets use the same\n`prepare_data.sh` \u002F `train_sft.sh` interface; see\n[`scripts\u002Fsft\u002FREADME.md`](scripts\u002Fsft\u002FREADME.md) for defaults and overrides.\n\n### Stage 2: Run Self-Evolving OPD RL\n\nAll SEED RL scripts live under `examples\u002Fseed_trainer\u002F` and assume the repo root\nas the working directory.\n\n```bash\nbash examples\u002Fseed_trainer\u002Frun_alfworld_sft_glm_self.sh\nbash examples\u002Fseed_trainer\u002Frun_webshop_sft_glm_self.sh\nbash examples\u002Fseed_trainer\u002Frun_search_sft_glm_self.sh\n```\n\nOther teacher-self SFT entrypoints use the same naming convention:\n\n```bash\nbash examples\u002Fseed_trainer\u002Frun_alfworld_sft_qwen_self.sh\nbash examples\u002Fseed_trainer\u002Frun_ezpoints_sft_gemini_self.sh\nbash examples\u002Fseed_trainer\u002Frun_sokoban_sft_gemini_self.sh\n```\n\n\n\n\n## Merge Checkpoints\n\nSee `scripts\u002Fmodel_merger.py` for FSDP\u002FMegatron merge examples using paths under\n`.\u002Fcheckpoints\u002F...`.\n\n## ⭐ Citation\n\nIf you find this project useful, welcome to cite us.\n\n```bibtex\n@article{wu2026seed,\n  title={SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning},\n  author={Wu, Jinyang and Yang, Shuo and Lu, Zhengxi and Zhang, Fan and\n          Shen, Yuhao and Feng, Lang and Luo, Haoran and Lian, Zheng and\n          Zhang, Shuai and Wen, Zhengqi and Tao, Jianhua},\n  journal={arXiv preprint arXiv:2607.14777},\n  year={2026}\n}\n```\n\n## Acknowledgement\n\nThis project builds on\n[veRL](https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl),\n[verl-agent](https:\u002F\u002Fgithub.com\u002FlangfengQ\u002Fverl-agent),\n[SDAR](https:\u002F\u002Fgithub.com\u002FZJU-REAL\u002FSDAR),\nand\n[OPID](https:\u002F\u002Fgithub.com\u002Fjinyangwu\u002FOPID). We thank the authors of those\nprojects.\n",2,"2026-07-20 02:30:03","CREATED_QUERY"]