[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-81020":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":13,"subscribersCount":13,"size":13,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":13,"forks30d":13,"starsTrendScore":17,"compositeScore":13,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":22,"hasPages":20,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":13,"starSnapshotCount":13,"syncStatus":16,"lastSyncTime":30,"discoverSource":31},81020,"T2PO","WillDreamer\u002FT2PO","WillDreamer","【ICML2026 Spotlight】 T2PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning","",null,"Python",36,0,29,1,2,6,7,"Apache License 2.0",false,"main",true,[24,25,26],"agent","llm","rl","2026-06-12 02:04:09","\n\u003Cdiv align=\"center\">\n\n\n  \u003Cdiv align=\"center\">\n\n  # \u003Cb>T\u003Csup>2\u003C\u002Fsup>PO\u003C\u002Fb>: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic RL\n\n  \u003C\u002Fdiv>\n\n  \u003Cdiv style=\"margin: 1.2em 0 0.7em 0;\">\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FWillDreamer\u002FT2PO\" align=\"center\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Fstars\u002FWillDreamer\u002FT2PO?style=social\" alt=\"GitHub stars\" \u002F>\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Fwillhx\u002Ft2po\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FHuggingFace-🤗-yellow\" alt=\"Huggingface\" \u002F>\n    \u003C\u002Fa>\n    \u003Ca href=\"http:\u002F\u002Farxiv.org\u002Fabs\u002F2605.02178\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-arXiv-red\" alt=\"arXiv Paper\" \u002F>\n    \u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fzhuanlan.zhihu.com\u002Fp\u002F2034956125624381457\">\n      \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FBlog-Post-green\" alt=\"Blog\u002FProject Post\" \u002F>\n    \u003C\u002Fa>\n  \u003C\u002Fdiv>\n  \n  \u003Ch3 align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F-🌟%20ICML%202026%20Spotlight%20Paper-116fb8?style=for-the-badge\" alt=\"ICML 2026 Spotlight\" \u002F>\n    \u003Cspan style=\"font-size:1.3em;\"> | \u003C\u002Fspan>\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F-Now%20Open%20Source!-3be5c3?style=for-the-badge\" alt=\"Now Open Source\" \u002F>\n  \u003C\u002Fh3>\n  \n  \u003Cdiv align=\"center\">\n\n\u003C\u002Fdiv>\n\n\n  \u003Cimg src=\"public\u002Fmain.png\" alt=\"Agentic RL Arena Framework\" width=\"830\" style=\"max-width:95%; border-radius:18px; margin-top: 10px; box-shadow:0 6px 32px rgba(28,111,184,.085);\"\u002F>\n  \n  \u003Cdiv align=\"center\">\n    \u003Cem>\n      \u003Cspan style=\"color:gray\">Figure 1: Overview of the \u003Cb>T\u003Csup>2\u003C\u002Fsup>PO\u003C\u002Fb> framework\u003C\u002Fspan>\n       ｜ \n      \u003Cspan style=\"color:crimson;\">\u003Cb>ICML 2026 Spotlight\u003C\u002Fb>\u003C\u002Fspan>\n    \u003C\u002Fem>\n  \u003C\u002Fdiv>\n\n\u003C\u002Fdiv>\n\n---\n\n**\u003Cspan style=\"color:#fd4f57\">Problem\u003C\u002Fspan>:** Hesitation is defeat! Multi-turn RL for LLM agents is **powerful**, but critically limited by \u003Cspan style=\"color:#0C8F8F; font-weight:bold;\">poor exploration\u003C\u002Fspan>.\n\n**\u003Cspan style=\"color:#116fb8\">Key idea\u003C\u002Fspan>:** Training fails mostly when agents repeat low-value actions or ignore task-level uncertainty.\n\n**\u003Cspan style=\"color:#3be5c3\">Our method\u003C\u002Fspan>:** **T\u003Csup>2\u003C\u002Fsup>PO** directly controls exploration at both the token and turn level using uncertainty signals, greatly improving stability and sample efficiency.\n\n---\n\n## 🛠️ \u003Cspan style=\"color:#116fb8;\">T\u003Csup>2\u003C\u002Fsup>PO Framework Design\u003C\u002Fspan>\n\n\u003Ctable width=\"100%\" style=\"max-width:720px; margin:18px auto; font-size:1.10rem; border-radius:10px; background:#fcfdfe;\">\n  \u003Ctr>\n    \u003Ctd>🔹 \u003Cb>Token-level:\u003C\u002Fb> \u003Cmark>T\u003Csup>2\u003C\u002Fsup>PO\u003C\u002Fmark> tracks marginal uncertainty and triggers interventions when it dips below a threshold.\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>🔸 \u003Cb>Turn-level:\u003C\u002Fb> \u003Cmark>T\u003Csup>2\u003C\u002Fsup>PO\u003C\u002Fmark> resamples turns with negligible exploration progress, preventing wasted updates.\u003C\u002Ftd>\n  \u003C\u002Ftr>\n  \u003Ctr>\n    \u003Ctd>📊 \u003Cb>Benchmarks:\u003C\u002Fb> Substantial gains on WebShop, ALFWorld, SearchQA and more—significantly better stability and learning efficiency.\u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n---\n\n## 🔥 Key Features\n\n- ✅ Support Training Multi-turn Embody Agents\n- ✅ Support Training Multi-turn Search Agents\n- ✅ Support Training Multi-turn Multi-modal Game Agents\n- ✅ Support Training Multi-turn Web Agents\n- ✅ Support Evaluating Commerical LLMs as Agents\n\n---\n\n## 💡 Getting Started\n\nOur work is based on the following main dependencies:\n\n```python\nPython=3.11, VeRL=0.4.0, PyTorch=2.6.0, and vLLM=0.8.5\n```\n\n\u003Cdetails>\n\u003Csummary>👉 \u003Cb>Click to expand installation guide\u003C\u002Fb> \u003Cem>(optional)\u003C\u002Fem>\u003C\u002Fsummary>\n\n```bash\n# (Optional) Install conda\nbash set_conda.sh\n\n# Install main dependencies\nbash setup_env.sh\n\n# Install extra requirements for specific tasks\nconda activate verl\npip install -r requirements_xxx.txt\n```\n\u003C\u002Fdetails>\n\n---\n\n## 🚀 Existing Support\n\n\u003Cdetails open>\n\u003Csummary>🤖 \u003Cb>Embodied Agents\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\n# 1. Build the environments\nbash prepare_all_embody.sh\n\n# 2. Run the demo code\nconda activate agentrl_embody\nbash examples\u002Fworld_agent_trainer\u002Ftrain_xxx.sh\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🛒 \u003Cb>Web Agents\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\n# 1. Build the webshop environments\nbash prepare_all_web.sh\n\n# 2. Run the demo code\nconda activate agentrl_web\nbash examples\u002Fshop_agent_trainer\u002Ftrain_xxxx.sh\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🕸️ \u003Cb>Search Agents\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\n# 1. Build the RAG server environments\nbash prepare_all_search.sh\n\n# 2. Run the demo code\nconda activate agentrl_search\nbash examples\u002Fsearch_agent_trainer\u002Ftrain_xxx.sh\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>🎮 \u003Cb>Multi-modal Game Agents\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\n# 1. Install the requirements\nbash prepare_all_game.sh\n\n# 2. Run the demo code\nbash examples\u002Fgame_agent_trainer\u002Ftrain_xxx.sh\n```\n\u003C\u002Fdetails>\n\n---\n\n## 🌊 Easy Extension\n\n✨ \u003Cb>Extensible by Design:\u003C\u002Fb>\u003Cbr>\n\u003Cul>\n  \u003Cli>All task recipes live in \u003Ccode>recipe\u003C\u002Fcode>. Wrap the VERL worker to plug in your own method. \u003Ca href=\"docs\u002Fextension.md\">[usage]\u003C\u002Fa>\u003C\u002Fli>\n  \u003Cli>Add new environments under \u003Ccode>agent_system\u003C\u002Fcode>.\u003C\u002Fli>\n  \u003Cli>Extra dependencies go into \u003Ccode>requirements_xxx.txt\u003C\u002Fcode>.\u003C\u002Fli>\n  \u003Cli>Third-party tools? Place them in \u003Ccode>AgentRL\u002Fsandbox\u003C\u002Fcode>.\u003C\u002Fli>\n\u003C\u002Ful>\n\n---\n\n## 📊 Further Analysis\n\n\u003Cdetails>\n\u003Csummary>📈 \u003Cb>Expand for MLFlow analysis setup\u003C\u002Fb>\u003C\u002Fsummary>\n\n```bash\n# Install requirements\npip install mlflow\n\n# Start server\nmlflow server \\\n  --host 0.0.0.0 --port 5000 \\\n  --backend-store-uri sqlite:\u002F\u002F\u002F\u002Ftmp\u002Fmlruns.db \\\n  --default-artifact-root \u002Ftmp\u002Fmlruns\n\nexport MLFLOW_TRACKING_URI=http:\u002F\u002F127.0.0.1:5000\n\n# Trainer config\nactor_rollout_ref.rollout.trace.backend: mlflow  # or weave\nactor_rollout_ref.rollout.trace.token2text: True\ntrainer.logger: ['console', 'mlflow']\n```\n\u003C\u002Fdetails>\n\n---\n## ✍️ Citation\n```bibtex\n@article{wang2026t,\n  title={T $^2$ PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning},\n  author={Wang, Haixin and Cui, Hejie and Zhang, Chenwei and Liu, Xin and Jin, Shuowei and Geng, Shijie and Zhang, Xinyang and Zalmout, Nasser and Shi, Zhenyu and Sun, Yizhou},\n  journal={arXiv preprint arXiv:2605.02178},\n  year={2026}\n}\n```\n\n\n","T2PO 是一个针对多轮次代理强化学习（Agentic RL）中探索控制问题的解决方案。该项目通过在令牌和回合级别上利用不确定性信号来指导探索，从而提高训练过程中的稳定性和样本效率。其核心技术特点包括：在令牌层面跟踪边缘不确定性并在低于阈值时触发干预；在回合层面重新采样那些探索进展微乎其微的回合，避免无效更新。T2PO 适用于需要高效稳定训练大型语言模型作为代理执行复杂任务的场景，如基于文本的游戏、对话系统等。项目采用 Python 编写，并已在 WebShop、ALFWorld 和 SearchQA 等多个基准测试中展现出显著优势。","2026-06-11 04:03:13","CREATED_QUERY"]