[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93889":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":15,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":16,"rankGlobal":10,"rankLanguage":10,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":10,"pushedAt":10,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":15,"starSnapshotCount":15,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},93889,"axrl","XYZ-AI-Lab\u002Faxrl","XYZ-AI-Lab","AxisRL is an agentic RL post-training framework built on SGLang rollout, Megatron training, and real-world agent workflows.","https:\u002F\u002Fxyz-lab.ai\u002F",null,"Python",566,21,8,0,48.03,"Apache License 2.0",false,"main",true,[],"2026-09-21 04:01:26","\u003Cdiv align=\"center\">\n  \u003Cpicture>\n    \u003Csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs\u002Ffigs\u002Faxisrl-logo-dark.svg\">\n    \u003Csource media=\"(prefers-color-scheme: light)\" srcset=\"docs\u002Ffigs\u002Faxisrl-logo-light.svg\">\n    \u003Cimg alt=\"AxisRL - Agentic Post-Training\" src=\"docs\u002Ffigs\u002Faxisrl-logo-light.svg\" width=\"680\">\n  \u003C\u002Fpicture>\n\n  \u003Cp>\n    \u003Cstrong>English\u003C\u002Fstrong> ·\n    \u003Ca href=\"docs\u002FREADME-cn.md\">简体中文\u003C\u002Fa>\n  \u003C\u002Fp>\n\n  \u003Cp>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FXYZ-AI-Lab\u002Faxrl\u002Factions\u002Fworkflows\u002Fci.yml\">\u003Cimg alt=\"CI\" src=\"https:\u002F\u002Fgithub.com\u002FXYZ-AI-Lab\u002Faxrl\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg\">\u003C\u002Fa>\n    \u003Cimg alt=\"Python 3.12+\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPython-3.12%2B-blue.svg\">\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fastral-sh\u002Fruff\">\u003Cimg alt=\"Ruff\" src=\"https:\u002F\u002Fimg.shields.io\u002Fendpoint?url=https:\u002F\u002Fraw.githubusercontent.com\u002Fastral-sh\u002Fruff\u002Fmain\u002Fassets\u002Fbadge\u002Fv2.json\">\u003C\u002Fa>\n    \u003Ca href=\"LICENSE\">\u003Cimg alt=\"License: Apache-2.0\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache_2.0-blue.svg\">\u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FXYZ-AI-Lab\u002Faxrl\u002Fpulls\">\u003Cimg alt=\"PRs Welcome\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPRs-welcome-brightgreen.svg\">\u003C\u002Fa>\n  \u003C\u002Fp>\n\n  \u003Cp>\n    \u003Ca href=\"https:\u002F\u002Fxyz-lab.ai\">XYZ AI Lab\u003C\u002Fa> ·\n    \u003Ca href=\"https:\u002F\u002Fxyz-lab.ai\u002Fblogs\u002Fai4ai-at-scale\u002F\">Technical Blog\u003C\u002Fa>\n  \u003C\u002Fp>\n\u003C\u002Fdiv>\n\n# AxisRL\n\nAxisRL is an agentic RL post-training framework built on SGLang rollout, Megatron training, and real-world agent workflows.\n\nAxisRL connects high-throughput rollout, large-scale training, weight synchronization, data movement, resource scheduling, and reproducible debugging inside one coherent framework. SGLang and Megatron remain the core serving and training engines; AxisRL handles the system layer around agentic post-training.\n\n## ✨ Highlights\n\n- Built on **SGLang** for high-throughput rollout and **Megatron** for large-scale distributed training.\n- Used in agent RL workflows with 300+ turn trajectories and training runs at hundreds-of-billions-parameter scale.\n- Provides configurable policy optimization objectives including PPO, GRPO\u002FGRPO2, GSPO, TOPR, TIS, and related variants.\n- Supports both white-box agent environments and black-box harness capture through an OpenAI-compatible proxy.\n- Reduces rollout\u002Ftraining idle time with partial rollout and a lightweight control plane.\n- Provides handle-based data movement, context packing, routing replay, mismatch analysis, and spike replay for rollout-trainer consistency.\n\n## 🧭 Why AxisRL?\n\nLLM post-training workloads are moving beyond single-turn question answering. In agentic RL, a model may interact with a long-lived environment, call tools, observe tool results, update context, and receive a reward only after several turns.\n\nThat changes the job of a post-training framework. It has to coordinate multi-turn rollout, environment state, tool calls, verifiers, reward collection, training sample construction, and weight synchronization. It also has to make training behavior observable, because small differences in tokenization, chat templates, logprobs, routing, packing, or weight sync can appear later as loss spikes, reward instability, or rollout-trainer mismatch.\n\nAxisRL is designed for this setting: real agent workflows, SGLang rollout, Megatron training, and the system contracts between them.\n\n## 🏗️ Architecture\n\n![AxisRL workflow](docs\u002Ffigs\u002Faxrl-workflow.png)\n\nAt a high level, an AxisRL run follows this loop:\n\n1. Rollout actors execute task-specific agent workflows.\n2. SGLang workers serve model generation.\n3. Environments, tools, verifiers, or external harnesses produce interaction records and rewards.\n4. Megatron workers consume training samples and run PPO or GRPO-family training.\n5. Updated weights are synchronized back to the rollout side for the next iteration.\n\nAxisRL keeps the driver lightweight. The driver manages scheduling, lifecycle, metrics, phase transitions, and metadata. Heavy payloads, such as routing replay data or future multimodal artifacts, move through a handle-based data path and are read by trainer workers on demand.\n\n## 🎯 Design Goals\n\n| Goal | Problem | AxisRL Approach |\n| --- | --- | --- |\n| Flexibility | Agent workflows differ in control flow, tools, rewards, context management, and resource needs. | Use recipes for task logic, support white-box environments and black-box harness capture, and manage heterogeneous components through resource groups. |\n| Efficiency | Long-tail trajectories, tool latency, verifiers, and repeated context can leave rollout or training resources idle. | Use partial rollout, thin control-plane scheduling, handle-based data movement, prefix-tree merge, MagiAttention, and off-policy stabilization tools such as TIS, sequence masking, and Icepop. |\n| Observability | Rollout and trainer paths can silently diverge in tokenization, masks, logprobs, routing, packing, or weight versions. | Test critical boundaries and provide mismatch analysis, routing replay checks, and spike replay for reproducible debugging. |\n\n## ⚙️ Installation\n\nThe recommended environment is the project Docker image. It includes SGLang, Megatron Core, MagiAttention, Ray, CUDA dependencies, and the Python packages used by the current recipes.\n\nPre-built image:\n\n```bash\ndocker pull leejunjie\u002Fsglang-mcore:cu130-sgl0.5.14-mcore0.18-magi\n```\n\nDockerfile:\n\n```text\ndocker\u002Fcuda\u002Fcu130-sgl0.5.14-mcore0.18-magi.Dockerfile\n```\n\nInstall AxisRL inside the container:\n\n```bash\npip install -e .\n```\n\nOptionally download the common models and datasets referenced by the current recipes and tests:\n\n```bash\npython axrl\u002Fexample\u002Fdownload_data.py\n```\n\nThe bulk download can be large because it includes multi-billion-parameter models. For a narrower run, adjust the recipe model and dataset paths instead of downloading everything.\n\n## 🚀 Quick Start\n\nThe recipe scripts below are the main entry points. They assume a GPU machine with enough resources for the default parallelism in each recipe. You can override most config fields from the command line with `--path.to.field=value`.\n\n### GSM8K GRPO\n\n```bash\nAXRL_OUTPUT_DIR_NAME=grpo_gsm8k \\\nbash axis_recipe\u002Fgrpo_gsm8k\u002Frun_train.sh \\\n  --online_rl_train.max_global_updates=4\n```\n\n### GSM8K PPO\n\n```bash\nAXRL_OUTPUT_DIR_NAME=ppo_gsm8k \\\nbash axis_recipe\u002Fppo_gsm8k\u002Frun_train.sh \\\n  --online_rl_train.max_global_updates=4\n```\n\n### Search-R1\n\nSearch-R1 uses a retrieval server in addition to rollout and training workers.\n\n```bash\nexport AXRL_SEARCH_PORT=18000\nbash axis_recipe\u002Fsearch_r1\u002Fstart_retriever.sh\npython axis_recipe\u002Fsearch_r1\u002Fsearch_r1_config.py\n\nAXRL_OUTPUT_DIR_NAME=search_r1 \\\npython -u axis_recipe\u002Fsearch_r1\u002Ftrain_search_r1.py \\\n  --config_path=axis_recipe\u002Fsearch_r1\u002Fsearch-r1-config.yaml \\\n  --online_rl_train.max_global_updates=4\n```\n\nFor the default full recipe script:\n\n```bash\nAXRL_SEARCH_PORT=18000 \\\nbash axis_recipe\u002Fsearch_r1\u002Frun_train.sh\n```\n\n### Black-Box RL With OpenHands and E2B\n\nThis recipe is still a work in progress. It demonstrates the black-box harness integration path with OpenHands\u002FE2B, but the config, launch scripts, and proxy interfaces may change.\n\nThe black-box RL recipe runs OpenHands inside E2B sandboxes. OpenHands calls AxisRL through an OpenAI-compatible proxy, and AxisRL captures model inputs, outputs, metadata, and rewards for training.\n\nPrerequisites:\n\n- `E2B_API_KEY` in the environment or in `.env`.\n- `cloudflared` on the training host for the default tunnel path.\n- An E2B template named `axrl-openhands`.\n\nBuild the E2B template once:\n\n```bash\ncd axis_recipe\u002Fblackbox_rl\u002Fe2b_template\ne2b template build --name axrl-openhands\ncd -\n```\n\nRun a small rollout smoke test:\n\n```bash\nAXRL_OUTPUT_DIR_NAME=blackbox-e2b-smoke \\\nAXRL_ROLLOUT_TEST_NUM_CASES=2 \\\nbash axis_recipe\u002Fblackbox_rl\u002Frun_rollout_test_distributed.sh\n```\n\nRun a short training job:\n\n```bash\nAXRL_OUTPUT_DIR_NAME=blackbox-e2b-train \\\nbash axis_recipe\u002Fblackbox_rl\u002Frun_train_distributed.sh \\\n  --online_rl_train.max_global_updates=4\n```\n\nMore details are in [axis_recipe\u002Fblackbox_rl\u002FREADME.md](axis_recipe\u002Fblackbox_rl\u002FREADME.md).\n\n## 🧩 Recipes\n\n| Recipe | Mode | Entry Point | Notes |\n| --- | --- | --- | --- |\n| GSM8K GRPO | White-box RL | `axis_recipe\u002Fgrpo_gsm8k\u002Frun_train.sh` | GRPO-style math training recipe. |\n| GSM8K PPO | White-box RL | `axis_recipe\u002Fppo_gsm8k\u002Frun_train.sh` | PPO math training recipe with actor and value workers. |\n| Search-R1 | White-box tool RL | `axis_recipe\u002Fsearch_r1\u002Frun_train.sh` | Retrieval-augmented multi-turn search recipe. |\n| Black-Box RL | Black-box harness RL | `axis_recipe\u002Fblackbox_rl\u002Frun_train_distributed.sh` | WIP OpenHands\u002FE2B recipe; config and proxy interfaces may change. |\n\nEach recipe owns task-specific logic such as the dataset, environment loop, verifier, reward computation, metrics, and training configuration. The shared AxisRL path handles rollout scheduling, sample construction, trainer input, weight sync, and debugging.\n\n## 🔧 Key Technical Ideas\n\n### White-Box and Black-Box Agent Workflows\n\nAxisRL supports two integration patterns.\n\n| Mode | Best Fit | AxisRL Handles | User Focus |\n| --- | --- | --- | --- |\n| White-box RL | Math, Search, simple tool environments | Agent loop control, rollout scheduling, training sample construction | Environment, tools, verifier, reward |\n| Black-box RL | OpenHands, browser tasks, complex external harnesses | Model I\u002FO and reward capture through an OpenAI-compatible proxy | Harness launch, adapters, verifier, reward collection |\n\nWhite-box recipes express the environment loop inside AxisRL. Black-box recipes let an existing harness call the model through an OpenAI-compatible API, while AxisRL captures the interaction and builds trainable samples from it.\n\n### Partial Rollout\n\nMulti-turn agent trajectories often have long-tail latency. Some samples finish quickly, while others may require many tool calls or slow verifier responses. AxisRL supports partial rollout so completed or partially completed samples can be handed to the trainer earlier, reducing waits caused by slow trajectories.\n\n### Thin Control Plane and Handle-Based Data Plane\n\nLarge rollout-side payloads should not have to pass through a centralized driver. AxisRL keeps the driver focused on scheduling and metadata, while heavy data moves through handles. Trainer workers read payloads on demand, which is useful for MoE routing replay data, complex rollout artifacts, and future multimodal intermediates.\n\n### Context Management and MagiAttention\n\nAgent contexts are not always simple linear sequences. A recipe may retain recent tool outputs, hide older tool results, or replace parts of context with placeholders. AxisRL uses prefix-tree merge and MagiAttention to reduce repeated attention compute while preserving the context each turn saw during rollout.\n\nThe key invariant is that context management and packing should not change training semantics. Whether a long trajectory is merged into one sample or split into multiple samples because of length limits, the training side should produce consistent gradients.\n\n### Rollout-Trainer Consistency\n\nTraining instability often comes from subtle differences between rollout and trainer execution paths. AxisRL puts critical boundaries under test, including tokenization, chat templates, weight sync, checkpointing, routing replay, RolloutTrace packing, prefix-tree merge, MagiAttention forward, the OpenAI proxy, and mismatch analysis.\n\nRollout Routing Replay (R3) is one example. In MoE post-training, R3 reduces expert routing mismatch between rollout and trainer, making KL and loss more stable. Routing payloads use the handle-based data path so the driver does not become a heavy-data relay.\n\n![R3 mismatch analysis](docs\u002Ffigs\u002Fr3-mismatch.png)\n\n### Mismatch Analysis and Spike Replay\n\nMismatch analysis compares token-level differences across rollout and trainer paths, backends, configurations, and routing replay settings. It helps identify whether a problem is broad drift, a small number of outliers, or something concentrated in a specific token range, sequence type, or context layout.\n\nSpike replay targets occasional gradient or loss spikes. AxisRL can save a snapshot of weights, optimizer state, data, and relevant routing information before a spike update. The same update can then be replayed for inspection instead of waiting for the next non-deterministic failure.\n\n## 📁 Repository Layout\n\n```text\naxrl\u002F                 Core framework code\naxis_recipe\u002F          Public recipes and task-specific integrations\ndocs\u002F                 Design notes, blog posts, and technical reports\ndocker\u002F               Dockerfiles for reproducible training environments\nscripts\u002F              Runtime helper scripts\ntests\u002F                Unit, integration, and consistency tests\n```\n\n## 📚 Documentation\n\n- [English blog](docs\u002Faxrl-blog-en.md)\n- [Chinese blog](docs\u002Faxrl-blog-cn.md)\n- [Black-box RL recipe](axis_recipe\u002Fblackbox_rl\u002FREADME.md)\n- [Config parsing notes](docs\u002Fconfig-parsing.md)\n\n## 🗺️ Roadmap\n\n- Add more real-world agent recipes and public case studies.\n- Improve asynchronous execution between rollout and training.\n- Extend support for multimodal rollout artifacts.\n","AxisRL 是一个面向智能体强化学习（Agentic RL）的后训练框架，专注于多轮交互场景下的高效策略优化。它基于 SGLang 实现高吞吐量多步 rollout，依托 Megatron 支持百B级参数的大规模分布式训练，并内置 rollout-训练协同调度、上下文打包、路由回放、不一致分析与轻量控制平面等系统级能力。支持 PPO、GRPO、GSPO 等多种策略优化目标，兼容白盒环境与黑盒工具调用（通过 OpenAI 兼容代理）。适用于需要长程决策、工具调用与延迟奖励建模的智能体训练场景，如复杂任务自动化、AI Agent 开发与仿真环境训练。",2,"2026-07-27 02:30:05","CREATED_QUERY"]