[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-11249":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":15,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":17,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":21,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":30,"readmeContent":31,"aiSummary":32,"trendingCount":16,"starSnapshotCount":16,"syncStatus":33,"lastSyncTime":34,"discoverSource":35},11249,"OpenSearch-VL","shawn0728\u002FOpenSearch-VL","shawn0728","🔍 OpenSearch-VL provides a fully open recipe for training strong multimodal deep search agents through high-quality data curation, diverse visual\u002Fsearch tools, and fatal-aware agentic reinforcement learning.","https:\u002F\u002Farxiv.org\u002Fpdf\u002F2605.05185",null,"Python",212,18,4,3,0,9,59,60.24,"Apache License 2.0",false,"main",[24,25,26,27,28,29],"deep-research-agent","multimodal","opensource","search-agent","think-with-image","vlm-agent","2026-06-12 04:00:54","\u003Cdiv align=\"center\">\n\n  \u003Cimg src=\".\u002Fimages\u002Flogo.png\" alt=\"OpenSearch-VL\" width=\"35%\">\n  \u003Ch1 style=\"margin: -18px 0 0; font-size: 1.8em;\">\n    An Open Recipe for Frontier Multimodal Search Agents\n  \u003C\u002Fh1>\n\n  \u003Cp>\u003Cb>Cold-Start Agentic SFT  ·  Multi-Turn Fatal-Aware GRPO  ·  Visual Tool Use\u003C\u002Fb>\u003C\u002Fp>\n  \n  [![Paper](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fpaper-A42C25?style=for-the-badge&logo=arxiv&logoColor=white)](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2605.05185)\n  [![alphaXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Fdiscussion-A42C25?style=for-the-badge&logo=arxiv&logoColor=white&color=blue)](https:\u002F\u002Fwww.alphaxiv.org\u002Fabs\u002F2605.05185)\n  [![Github](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FOpenSearch_VL-000000?style=for-the-badge&logo=github&logoColor=white)](https:\u002F\u002Fgithub.com\u002Fshawn0728\u002FOpenSearch-VL)\n  [![Hugging Face Collection](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FOpenSearch_VL_Collection-fcd022?style=for-the-badge&logo=huggingface&logoColor=000)](https:\u002F\u002Fhuggingface.co\u002FOpenSearch-VL)\n  [![Twitter](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FTwitter-%23000000.svg?style=for-the-badge&logo=twitter&logoColor=white)](https:\u002F\u002Fx.com\u002Fi\u002Fstatus\u002F2052822171932897362)\n\n  [![Awesome](https:\u002F\u002Fawesome.re\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002Fshawn0728\u002FOpenSearch-VL)\n  [![License: Apache 2.0](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-blue.svg)](https:\u002F\u002Fopensource.org\u002Flicenses\u002FApache-2.0)\n  [![Python](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPython-3.10%2B-blue.svg)](https:\u002F\u002Fwww.python.org\u002F)\n  ![](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flast-commit\u002Fshawn0728\u002FOpenSearch-VL?color=green)\n\n\u003C\u002Fdiv>\n\n## 📑 Table of Contents\n\n- [📖 Introduction](#-introduction)\n- [🗺️ Overview](#%EF%B8%8F-overview)\n- [🍭 Method Overview](#-method-overview)\n- [📊 Main Results](#-main-results)\n- [🔎 Case Study](#-case-study)\n- [📁 Repository Layout](#-repository-layout)\n- [🛠️ Prerequisites](#%EF%B8%8F-prerequisites)\n- [🏋️ Agentic SFT · `code\u002FSFT`](#%EF%B8%8F-agentic-sft--codesft)\n- [🚀 Agentic RL · `code\u002FRL`](#-agentic-rl--coderl)\n- [📊 Inference & Evaluation · `opensearch_vl`](#-inference--evaluation--opensearch_vl)\n- [🚧 TODO](#-todo)\n- [🙌 Acknowledgements](#-acknowledgements)\n\n---\n\n## 📖 Introduction\n\n**OpenSearch-VL** is a fully open recipe for training frontier multimodal deep-research agents with agentic reinforcement learning. In contrast to standard VLMs that answer in a single forward pass, the agent operates as a closed loop: it inspects the image, crops or enhances the regions of interest, issues web and image searches, visits the retrieved pages, and only then writes an answer grounded in the gathered evidence.\n\nReproducing top-tier multimodal search agents has so far been difficult because the underlying training data, trajectory-synthesis pipelines, and training recipes remain proprietary. This release aims to close that gap: we open-source the **data, code, and model checkpoints** required to reproduce the paper end-to-end.\n\nThe recipe addresses three challenges that we found to be largely independent in practice:\n\n- **Data.** A curation pipeline built on top of the Wikipedia hyperlink graph synthesizes image-grounded multi-hop VQA. *Fuzzy entity rewriting* and *source-anchored visual grounding* are used to suppress shortcut solutions in which a single retrieval step is sufficient. The pipeline yields two open datasets: **SearchVL-SFT-36k** for supervised fine-tuning and **SearchVL-RL-8k** for reinforcement learning.\n- **Tools.** A unified visual and retrieval tool environment (`crop`, `layout_parsing`, `text_search`, `image_search`, `web_search`, `visit`, `perspective_correct`, `super_resolution`, `sharpen`, `python_interpreter`) is shared across SFT data generation, RL rollout, and inference. This allows the agent both to recover from imperfect visual inputs and to acquire external knowledge through a consistent interface.\n- **Algorithm.** A multi-turn **fatal-aware GRPO** algorithm explicitly handles cascading tool failures during long rollouts. Tokens that follow a fatal step are masked out of the policy gradient, while *one-sided advantage clamping* preserves the credit assigned to valid pre-failure reasoning rather than penalizing the entire trajectory.\n\nAcross seven knowledge-intensive multimodal benchmarks—SimpleVQA, VDR, MMSearch, LiveVQA, BrowseComp-VL, FVQA, and InfoSeek—OpenSearch-VL improves the average score by more than **10 points** over the corresponding agentic baselines, and at the 30B \u002F 32B scale matches the accuracy of strong proprietary systems.\n\n---\n\n## 🗺️ Overview\n\nThis repository provides everything needed to **reproduce, fine-tune, and evaluate** OpenSearch-VL:\n\n| Component | Path | Description |\n|-----------|------|-------------|\n| **SFT Training** | [`SFT\u002F`](SFT\u002F) | Agentic cold-start with LLaMA-Factory + Ray + ZeRO-3 (full-parameter fine-tune of LLM + ViT + projector) |\n| **RL Training** | [`RL\u002F`](RL\u002F) | Asynchronous agentic RLOO\u002FGRPO on top of SFT, built on rLLM + verl + Megatron-LM + sglang |\n| **Inference & Evaluation** | [`opensearch_vl\u002F`](opensearch_vl\u002F) | Unified `run_infer.py --model {8b,30b-a3b,32b,claude}` rollout + GPT-4o judge for BrowseComp-VL, HLE, VDR-Bench |\n| **Models** | [OpenSearch-VL](https:\u002F\u002Fhuggingface.co\u002FOpenSearch-VL) | OpenSearch-VL-{8B, 30B-A3B, 32B} checkpoints |\n| **Datasets** | [OpenSearch-VL](https:\u002F\u002Fhuggingface.co\u002FOpenSearch-VL) | SearchVL-SFT-36k (cold-start) and SearchVL-RL-8k (RL) |\n\n### Workflow at a Glance\n\n```\n   ┌────────────────┐     ┌────────────────┐     ┌────────────────────┐\n   │ Qwen3-VL base  │ ─── │ Agentic SFT    │ ─── │ Async Agentic RL   │ ───▶ OpenSearch-VL\n   │ (HF weights)   │     │ (code\u002FSFT)     │     │ (code\u002FRL)          │\n   └────────────────┘     └────────────────┘     └────────────────────┘\n                                │                         │\n                                ▼                         ▼\n                        SearchVL-SFT-36k          SearchVL-RL-8k\n                        7-domain tool-use         Vision-DeepResearch-QA\n                        cold-start trajectories   (RLOO \u002F GRPO + fatal-aware)\n```\n\n### Tool Environment\n\nOpenSearch-VL is equipped with a heterogeneous tool set $\\mathcal{T} = \\mathcal{T}_v \\cup \\mathcal{T}_r$ shared by SFT, RL, and inference:\n\n| Category | Tools | Purpose |\n|---|---|---|\n| **Retrieval** ($\\mathcal{T}_r$) | `text_search`, `image_search`, `web_search`, `visit` | Acquire external textual \u002F visual evidence and visit pages |\n| **Image Enhancement** ($\\mathcal{T}_v$) | `sharpen`, `super_resolution`, `perspective_correct` | Repair blurry, low-resolution, or skewed inputs before retrieval |\n| **Attention & Parsing** ($\\mathcal{T}_v$) | `crop`, `layout_parsing` (OCR) | Localize regions of interest and decode fine-grained content |\n| **Computation** | `python_interpreter` | Numerical \u002F programmatic computation on retrieved evidence |\n\n### Quick Links\n\n- **Get started** → [Prerequisites](#-prerequisites)\n- **Train your own SFT model** → [Agentic SFT](#%EF%B8%8F-agentic-sft--codesft)\n- **Run agentic RL** → [Agentic RL](#-agentic-rl--coderl)\n- **Inference & benchmark** → [Inference & Evaluation](#-inference--evaluation--opensearch_vl)\n\n---\n\n## 🍭 Method Overview\n\n**Data Curation Pipeline.**\nStarting from the English Wikipedia hyperlink graph, we sample multi-hop entity paths and convert them into multi-hop VQA instances by (a) extracting canonical question–answer pairs along the path, (b) rewriting each intermediate entity into a fuzzy descriptor while certifying answer invariance and uniqueness, and (c) anchoring the question on a representative image of the **source** node — *not* the answer node — so that single-hop image lookup shortcuts are eliminated. The pipeline finishes with staged tool-demanding filtering and an enhancement subset (random degradations paired with the corresponding restoration tools) before synthesizing multi-turn expert trajectories with answer- and process-level rejection sampling.\n\n![data_pipeline](.\u002Fimages\u002Fdata_pipeline.png)\n\n**RL Training Pipeline.**\nStarting from the SFT-initialized checkpoint, we sample a group of multi-turn trajectories against the real environment $\\mathcal{E}$. Each trajectory is scored by a composite reward combining final-task accuracy ($r_{\\text{acc}}$), process-level search-query quality ($r_{\\text{query}}$), and a multiplicative format gate ($r_{\\text{fmt}}$). To preserve valid reasoning when a trajectory eventually triggers cascading tool failures, we apply **fatal-aware token masking** to truncate the sequence at the fatal step $f_i$ and **one-sided advantage clamping** ($\\hat{A}_i = \\max(\\widetilde{r}_i, 0)$ for fatal trajectories) during policy optimization, preventing the suppression of viable early reasoning.\n\n![rl_pipeline](.\u002Fimages\u002Frl_pipeline.png)\n\n\n\n---\n\n## 📊 Main Results\n\nOpenSearch-VL is built on three Qwen3-VL variants and evaluated on **7 multimodal knowledge-intensive QA \u002F web-search benchmarks** under the same Pass@1 + GPT-4o judge protocol as VDR-Bench.\n\n\n**Highlights.**\n*OpenSearch-VL-8B* is the strongest open 8B agent (**+3.9** Avg over SenseNova-MARS-8B). *OpenSearch-VL-30B-A3B* improves the Qwen3-VL agentic baseline by **+13.8** Avg, with large gains on **VDR (+13.3)**, **MMSearch (+24.5)**, **FVQA (+10.2)**, and **InfoSeek (+16.2)**. *OpenSearch-VL-32B* surpasses Gemini-2.5-Pro and Claude-4-Sonnet direct-reasoning baselines on most benchmarks.\n\n**Fatal-aware GRPO ablation.**\nVanilla search-augmented GRPO improves SFT 64.6 → 67.6 Avg; the hard-masking baseline of Vision-DeepResearch saturates at 67.7; **fatal masking** alone reaches 69.1; our **full method with one-sided advantage clamping reaches 71.8** — a +4.2 gain over vanilla GRPO and the best score on every benchmark.\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\".\u002Fimages\u002Fturn_acc_combined.png\" width=\"65%\" alt=\"turn_acc_combined\">\n\u003C\u002Fdiv>\n\nFatal-aware GRPO sustains a **higher number of tool-use turns *and* a higher batch accuracy** than vanilla GRPO and the Hard-Mask baseline — encouraging productive exploration rather than prematurely suppressing difficult rollouts.\n\n---\n\n## 🔎 Case Study\n\nThe example below illustrates a representative OpenSearch-VL trajectory on a knowledge-intensive visual question: **“In what year did this bridge open?”** The answer cannot be read directly from the image or reliably produced by parametric knowledge alone. Instead, the agent progressively grounds the query through tool use.\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\".\u002Fimages\u002Fcase_study.png\" width=\"88%\" alt=\"OpenSearch-VL case study\">\n\u003C\u002Fdiv>\n\n**Tool-use flow.**\n1. **Visual inspection** — The agent identifies the roadside sign as the most useful visual clue.\n2. **Crop** — It zooms into the sign to obtain a cleaner local view.\n3. **Image search** — The cropped region helps identify the structure as the **Kessock Bridge**.\n4. **Text search \u002F verification** — A targeted search verifies the official opening year as **1982**.\n\nThis case highlights the core behavior encouraged by OpenSearch-VL: the agent does not guess from a single model pass, but chains visual perception, image retrieval, and textual evidence acquisition until the answer is grounded.\n\n---\n\n## 📁 Repository Layout\n\n```\ncode\u002F\n├── SFT\u002F                           # agentic supervised fine-tuning (LLaMA-Factory fork)\n│   ├── examples\u002Fagentic_full\u002F     # training YAMLs for Qwen2.5-VL \u002F Qwen3-VL \u002F Qwen3.5-VL\n│   ├── examples\u002Fdeepspeed\u002F        # ZeRO-2 \u002F ZeRO-3 configs\n│   ├── data\u002Fdataset_info.json     # 7 agentic-SFT datasets (relative paths)\n│   ├── src\u002Fllamafactory\u002F          # LLaMA-Factory source (trainer, data loader, CLI)\n│   └── README.md                  # SFT-specific instructions\n│\n├── RL\u002F                            # reinforcement learning on the SFT checkpoint\n│   ├── rllm\u002F                      # rLLM + verl + vision_deepresearch_async_workflow\u002F\n│   ├── Megatron-LM\u002F               # Megatron backend\n│   ├── mbridge\u002F                   # HF ↔ Megatron parallelism bridge\n│   └── README.md                  # RL-specific instructions\n│\n├── opensearch_vl\u002F                 # inference + benchmark evaluation (first-party package)\n│   ├── run_infer.py               # unified entrypoint (--model 8b|32b|30b-a3b|claude)\n│   ├── run_infer.sh               # env-driven launcher around run_infer.py\n│   ├── eval_with_gpt4o.py         # GPT-4o judge for BrowseComp-VL \u002F HLE \u002F VDR-Bench\n│   ├── run_eval.sh                # env-driven judge driver across all benchmarks\n│   ├── .env.example               # template for inference \u002F judge \u002F search keys\n│   └── opensearch_infer\u002F          # modular package: config, prompts, auth, tools,\n│                                  # search, runners (Claude + Qwen3-VL dense \u002F MoE),\n│                                  # message converters, multi-turn pipeline\n│\n└── README.md                      # (this file)\n```\n\n---\n\n## 🛠️ Prerequisites\n\n| Component   | Minimum                                                       |\n| ----------- | ------------------------------------------------------------- |\n| Python      | 3.10+                                                         |\n| CUDA        | 12.1+ (12.4 recommended)                                      |\n| PyTorch     | ≥ 2.4 with CUDA support                                       |\n| GPU         | ≥ 1× H100 \u002F H800 \u002F A100-80GB for 8B (multi-node for 30B \u002F 32B) |\n| NCCL \u002F RDMA | InfiniBand \u002F RoCE recommended for multi-node; see `RL\u002Frllm\u002F.env.example` |\n\nThe three components share most of their Python dependencies (PyTorch, `transformers`, `transformer_engine`, `flash-attn`, `deepspeed`, `ray`, `qwen-vl-utils`, `sglang`) — we recommend installing each sub-project into its **own virtual environment**.\n\n### External API keys\n\nAll keys are optional; components gracefully no-op if unset.\n\n| Variable | Used by | Purpose |\n| --- | --- | --- |\n| `API_GATEWAY_HOST` \u002F `API_GATEWAY_USER` \u002F `API_GATEWAY_KEY` | RL | Optional HMAC-secured gateway that proxies Serper + Jina behind one credential (set on RL workers). |\n| `API_HOST` \u002F `API_USER` \u002F `API_KEY` | opensearch_vl | Same gateway, named to match the inference package's env vars. |\n| `SERPER_API_KEY` | RL, opensearch_vl | [Serper.dev](https:\u002F\u002Fserper.dev) text & image search (used when no gateway is configured). |\n| `JINA_API_KEY` | RL, opensearch_vl | [Jina AI](https:\u002F\u002Fjina.ai) reader (page visit \u002F content extraction). |\n| `QWEN_API_BASE` \u002F `QWEN_MODEL_NAME` | opensearch_vl | OpenAI-compatible chat-completions server used for search summarization (defaults to a local Qwen3-32B). |\n| `LAYOUT_PARSING_API_URL` \u002F `LAYOUT_PARSING_TOKEN` | RL, opensearch_vl | PP-StructureV3-compatible OCR \u002F layout endpoint. |\n| `CLAUDE_API_HOST` \u002F `CLAUDE_API_USER` \u002F `CLAUDE_API_KEY` | opensearch_vl | Optional HMAC-secured gateway for the Claude Opus 4.5 backend. |\n| `JUDGE_API_BASE_URL` \u002F `JUDGE_APP_ID` \u002F `JUDGE_APP_KEY` \u002F `JUDGE_MODEL_MARKER` | opensearch_vl | OpenAI-compatible GPT-4o judge used by `eval_with_gpt4o.py`. |\n| `QWEN3VL_8B_PATH` \u002F `QWEN3VL_32B_PATH` \u002F `QWEN3VL_30B_A3B_PATH` | opensearch_vl | Local checkpoints for the three Qwen3-VL variants (overrideable via `--checkpoint`). |\n| `FVQA_IMAGE_DIR` | opensearch_vl | Optional fallback directory of `\u003Ccase_id>.\u003Cext>` images used when a benchmark URL is unreachable. |\n| `WANDB_API_KEY` | SFT, RL | W&B logging. |\n\nTwo templates are provided: [`RL\u002Frllm\u002F.env.example`](RL\u002Frllm\u002F.env.example) for the RL workers, and [`opensearch_vl\u002F.env.example`](opensearch_vl\u002F.env.example) for inference + judge. Copy whichever applies and source it before launching.\n\n---\n\n## 🏋️ Agentic SFT · `code\u002FSFT`\n\nCold-starts the base VLM on **7 tool-use datasets** (FVQA, Palace, WebQA, LiveVQA, WikiArt, Wiki-zh, Wiki-en — together forming **SearchVL-SFT-36k**, with an average of 6.3 tool-invocation turns per trajectory). We perform a **full-parameter fine-tune of the LLM + vision tower + projector** with DeepSpeed ZeRO-3, distributed via Ray.\n\n### Install\n\n```bash\ncd code\u002FSFT\npip install -e \".[torch,metrics,deepspeed,ray]\"\npip install qwen-vl-utils pillow av decord torchvision flash-attn\n```\n\n### Data layout\n\nDownload the **SearchVL-SFT-36k** bundle from the [HuggingFace collection](https:\u002F\u002Fhuggingface.co\u002FOpenSearch-VL) and place the 7 sub-sets under `code\u002FSFT\u002Fdata\u002F` so that the **relative** `file_name` values in [`data\u002Fdataset_info.json`](SFT\u002Fdata\u002Fdataset_info.json) resolve:\n\n```\nSFT\u002Fdata\u002F\n├── dataset_info.json\n├── new_fvqa\u002Ffvqa_llama_factory_clean.json\n├── palace\u002Fpalace_llama_factory_filtered.json\n├── WebQA\u002Fwebqa_llama_factory_filtered.json\n├── new_livevqa\u002Flivevqa_llama_factory_filtered.json\n├── wikiart\u002Fwikiart_llama_factory_filtered.json\n├── wiki_en\u002Fwiki_en_llama_factory_filtered.json\n└── wiki_zh\u002Fwiki_zh_llama_factory_filtered.json\n```\n\nEach JSON is in **ShareGPT format** with `conversations`, `images`, `system`, and `tools` columns.\n\n### Launch\n\n```bash\ncd code\u002FSFT\n\n# Multi-node Ray (16 nodes × 8 GPU by default):\nUSE_RAY=1 llamafactory-cli train \\\n    examples\u002Fagentic_full\u002Fqwen3_vl_full_sft_8b_ray.yaml\n\n# Single-node smoke test:\nFORCE_TORCHRUN=1 NNODES=1 NPROC_PER_NODE=8 llamafactory-cli train \\\n    examples\u002Fagentic_full\u002Fqwen3_vl_full_sft_8b_ray.yaml\n```\n\n### Available training configs\n\nEdit `ray_num_workers`, `placement_strategy`, and NCCL \u002F IB vars to match your cluster.\n\n| YAML | Model | # workers |\n| ---- | ----- | --------- |\n| `qwen3_vl_full_sft_8b_ray.yaml`      | Qwen3-VL-8B-Instruct      | 256 |\n| `qwen3_vl_full_sft_30_3b_ray.yaml`   | Qwen3-VL-30B-A3B-Instruct | 256 |\n| `qwen3_vl_full_sft_32b_ray.yaml`     | Qwen3-VL-32B-Instruct     | 256 |\n| `qwen3_5vl_full_sft_27b_ray.yaml`    | Qwen3.5-VL-27B-Instruct   | 256 |\n| `qwen3_5vl_full_sft_35b_3b_ray.yaml` | Qwen3.5-VL-35B-A3B        | 256 |\n| `qwen2_5_vl_full_sft_7b_ray.yaml`    | Qwen2.5-VL-7B-Instruct    | 256 |\n| `qwen2_5_vl_full_sft_32b_ray.yaml`   | Qwen2.5-VL-32B-Instruct   | 256 |\n| `qwen2_5_vl_full_sft_72b_ray.yaml`   | Qwen2.5-VL-72B-Instruct   | 256 |\n\n### Shared hyper-parameters\n\n| Hyperparameter | Value |\n| --- | --- |\n| Cutoff length | `32000` |\n| Precision | `bf16` |\n| Learning rate | `2e-5` (cosine, `warmup_ratio=0.1`) |\n| Epochs | `8` |\n| Per-device batch | `1` (with `gradient_checkpointing: true`) |\n| DeepSpeed | ZeRO-3 (`examples\u002Fdeepspeed\u002Fds_z3_config.json`) |\n| Frozen modules | none (`freeze_vision_tower: false`, `freeze_multi_modal_projector: false`) |\n\nCheckpoints land in `saves\u002F\u003Cmodel>\u002Ffull\u002Fsft_data_v1\u002F` (override via `output_dir` \u002F `ray_storage_path` in the YAML).\n\n> Full details, dataset format, and cluster notes: [`code\u002FSFT\u002FREADME.md`](SFT\u002FREADME.md).\n\n---\n\n## 🚀 Agentic RL · `code\u002FRL`\n\n**Asynchronous agentic RLOO \u002F GRPO \u002F PPO** on top of the SFT checkpoint, using [rLLM](https:\u002F\u002Fgithub.com\u002Frllm-org\u002Frllm)'s `AgentWorkflowEngine`, [verl](https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl) as the policy-optimization backend, and [Megatron-LM](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM) + [mbridge](https:\u002F\u002Fgithub.com\u002FISEEKYAN\u002Fmbridge) for large-scale model parallelism. Trajectories are rolled out by sglang; Megatron handles actor \u002F ref updates.\n\n### Install\n\n```bash\ncd code\u002FRL\u002Frllm    && pip install -e .\ncd ..\u002FMegatron-LM  && pip install -e .\ncd ..\u002Fmbridge      && pip install -e .\npip install \"sglang[all]\" transformer_engine flash-attn \\\n            ray==2.34.* hydra-core omegaconf wandb \\\n            pillow requests python-dotenv\n```\n\nCopy the env template:\n\n```bash\ncp RL\u002Frllm\u002F.env.example RL\u002Frllm\u002F.env   # edit keys as needed\n```\n\n### Data preparation\n\nThe workflow expects `rllm.data.DatasetRegistry` to hold a dataset named `Vision-DeepResearch-QA` (i.e. **SearchVL-RL-8k**). Two helpers in `RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Fdata_prepare\u002F` handle the conversion:\n\n```bash\ncd RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Fdata_prepare\n\n# 1) Extract embedded image bytes → PNG + JSONL\nDATA_ROOT=.\u002Fdata\u002FVision-DeepResearch-RL-Data \\\n    bash convert_parquet2jsonl.sh\n\n# 2) Register it with rLLM as \"Vision-DeepResearch-QA\" (90 \u002F 10 split)\nJSONL_PATH=.\u002Fdata\u002FVision-DeepResearch-RL-Data\u002Fvision-deepresearch_RL_Demo_1k.jsonl \\\n    bash register_rl_dataset.sh\n```\n\n### Launch\n\nAll run scripts `cd` into `rllm\u002F`, auto-source `.env`, and call `python -m vision_deepresearch_async_workflow.train_deepresearch_workflow_megatron` with the right Hydra overrides.\n\n```bash\n# Primary configuration in the paper: 8B dense, 8 nodes × 8 GPU\nbash RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Frun\u002Fqwen3-vl-8b-multi-node.sh\n\n# Other presets:\nbash RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Frun\u002Fqwen3-vl-8b-single-node.sh    # 1-node smoke test\nbash RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Frun\u002Fqwen3-vl-30b-3b-multi-node.sh # 30B-A3B MoE\nbash RL\u002Frllm\u002Fvision_deepresearch_async_workflow\u002Frun\u002Fqwen3-vl-32b-multi-node.sh    # 32B dense\n```\n\n### Key hyper-parameters (8B multi-node)\n\n| Field | Value |\n| --- | --- |\n| Advantage estimator | `rloo` (set `grpo` \u002F `reinforce_plus_plus` to swap) |\n| KL coefficient | `0.001` |\n| Clip ratio (high) | `0.28` |\n| Train prompt batch | `256` (group size `n_resp_per_prompt=8`, mini-batch `64`) |\n| Max prompt \u002F response length | `4096` \u002F `70000` |\n| Megatron parallelism | `TP=4 \u002F PP=2 \u002F CP=8` (dense) |\n| sglang rollout | `TP=4`, `gpu_memory_utilization=0.85` |\n| Reward composition | $r = r_{\\text{fmt}} \\cdot [\\,0.8\\, r_{\\text{acc}} + 0.2\\, r_{\\text{query}}\\,]$ |\n| Fatal threshold $K$ | `3` consecutive tool-execution errors |\n\nCheckpoints go to `checkpoints\u002F${project_name}\u002F${exp_name}\u002F`; trajectories can be dumped to `$TRAJ_DUMP_DIR` (default `.\u002Ftrajectory_dumps\u002F\u003Cexp>\u002F`).\n\n### Reproducing the paper\n\n| Variant | Script | Cluster |\n| --- | --- | --- |\n| OpenSearch-VL-8B          | `qwen3-vl-8b-multi-node.sh`       | 8 × 8 H100 \u002F H800 |\n| OpenSearch-VL-30B-A3B     | `qwen3-vl-30b-3b-multi-node.sh`   | 8 × 8 H100 \u002F H800 |\n| OpenSearch-VL-32B         | `qwen3-vl-32b-multi-node.sh`      | 16 × 8 H100 \u002F H800 |\n\n> Full details, environment variables, and cluster notes: [`code\u002FRL\u002FREADME.md`](RL\u002FREADME.md).\n\n---\n\n## 📊 Inference & Evaluation · `opensearch_vl`\n\nModular Python package that drives the trained model as a **tool-using Visual Investigation Agent**, plus a **GPT-4o judge** for standardized benchmark scoring. The same agent loop, tool environment and search\u002Fvisual utilities are shared across all three Qwen3-VL backends and the optional Claude Opus 4.5 backend; the variant is selected with a single `--model` flag.\n\n### Package layout\n\n```\nopensearch_vl\u002F\n├── run_infer.py                  # unified entrypoint (--model 8b|32b|30b-a3b|claude)\n├── run_infer.sh                  # env-driven wrapper around run_infer.py\n├── run_eval.sh                   # env-driven judge driver across all benchmarks\n├── eval_with_gpt4o.py            # GPT-4o judge for BrowseComp-VL \u002F HLE \u002F VDR-Bench\n├── .env.example                  # full env-variable template\n└── opensearch_infer\u002F\n    ├── config.py                 # env-driven settings + ModelSpec registry\n    ├── prompts.py                # Visual Investigation Agent system prompt\n    ├── auth.py                   # HMAC helper + Claude gateway client\n    ├── cos_upload.py             # optional COS uploader bootstrap\n    ├── image_io.py               # image download \u002F decode \u002F cache utilities\n    ├── image_engines.py          # PIL + OpenCV crop \u002F OCR \u002F enhance pipelines\n    ├── search.py                 # text_search \u002F image_search \u002F layout_parsing\n    ├── tools.py                  # JSON tool schema + parsing + dispatcher\n    ├── messages.py               # Gemini ↔ Claude \u002F Qwen3-VL converters\n    ├── runners.py                # ClaudeRunner + Qwen3VLRunner (dense + MoE)\n    └── pipeline.py               # per-case multi-turn agent loop\n```\n\n### Rollout\n\nOne entrypoint, four backends. Each call accepts a parquet of questions + images and writes one trajectory JSON per sample:\n\n| `--model`   | Backend                                              |\n| ----------- | ---------------------------------------------------- |\n| `8b`        | OpenSearch-VL-8B (Qwen3-VL-8B base, dense)           |\n| `32b`       | OpenSearch-VL-32B (Qwen3-VL-32B base, dense)         |\n| `30b-a3b`   | OpenSearch-VL-30B-A3B (Qwen3-VL-30B-A3B base, MoE)   |\n| `claude`    | Claude Opus 4.5 via HMAC gateway (no GPU required)   |\n\nMulti-GPU model parallelism is enabled automatically when `--gpus 0,1,...` lists more than one device (`device_map=\"auto\"`); single-GPU placement uses `device_map={\"\": \"cuda:N\"}`. The MoE scatter dtype patch for 30B-A3B is applied automatically.\n\n```bash\n# Source the env template first; only the entries you need have to be filled in.\ncp opensearch_vl\u002F.env.example ~\u002F.opensearch-vl.env\nsource ~\u002F.opensearch-vl.env\n\n# Dense Qwen3-VL-8B on a single GPU\npython opensearch_vl\u002Frun_infer.py --model 8b --gpus 0 \\\n    --data-path  \u002Fpath\u002Fto\u002Fbenchmark.parquet \\\n    --output-dir .\u002Foutputs\u002Fopensearch_vl_8b \\\n    --start 0 --end 1000\n\n# MoE Qwen3-VL-30B-A3B with 4-way model parallel (auto-applies the scatter dtype patch)\npython opensearch_vl\u002Frun_infer.py --model 30b-a3b --gpus 0,1,2,3 \\\n    --checkpoint \u002Fpath\u002Fto\u002FOpenSearch-VL-30B-A3B \\\n    --data-path  \u002Fpath\u002Fto\u002Fbenchmark.parquet \\\n    --output-dir .\u002Foutputs\u002Fopensearch_vl_30b_a3b\n\n# Claude Opus 4.5 (CLAUDE_API_HOST \u002F _USER \u002F _KEY required)\npython opensearch_vl\u002Frun_infer.py --model claude \\\n    --data-path  \u002Fpath\u002Fto\u002Fbenchmark.parquet \\\n    --output-dir .\u002Foutputs\u002Fclaude_opus\n```\n\nThe shell wrapper `opensearch_vl\u002Frun_infer.sh` reads the same parameters from environment variables (`MODEL`, `GPUS`, `DATA_PATH`, `OUTPUT_DIR`, `LIMIT`, `CATEGORY`, ...) for one-line invocations.\n\n### Benchmark evaluation\n\n[`eval_with_gpt4o.py`](opensearch_vl\u002Feval_with_gpt4o.py) consumes the trajectory directory produced above and calls a GPT-4o-class judge to compute per-sample correctness using the **VDR-Bench evaluation prompt**:\n\n```bash\npython opensearch_vl\u002Feval_with_gpt4o.py \\\n    --traj_dir    .\u002Foutputs\u002Fopensearch_vl_8b\u002Fbc_vl_level1 \\\n    --benchmark   bc_vl            # one of: hle | bc_vl | vdr\n    --max_workers 20\n```\n\n`--answer_file` is required for VDR-Bench (pass the `.parquet` with `id` \u002F `answer` columns).\n\n[`run_eval.sh`](opensearch_vl\u002Frun_eval.sh) is a thin driver that chains the five reported evaluations (BrowseComp-VL L1, BrowseComp-VL L2, HLE, VDR-Bench testmini × 2 models). Configure trajectory directories via env variables (`TRAJ_BC_VL_LEVEL1`, `TRAJ_BC_VL_LEVEL2`, `TRAJ_HLE`, `TRAJ_VDR_PRIMARY`, `TRAJ_VDR_SECONDARY`, `VDR_ANSWER_PARQUET`) and run:\n\n```bash\nbash opensearch_vl\u002Frun_eval.sh --workers 20\n```\n\n> Full inference details: [`opensearch_vl\u002FREADME.md`](opensearch_vl\u002FREADME.md).\n\n---\n\n## 🚧 TODO\n\n- [x] Release **OpenSearch-VL-{8B, 30B-A3B, 32B}** checkpoints on the [HuggingFace collection](https:\u002F\u002Fhuggingface.co\u002FOpenSearch-VL).\n- [x] Release **SearchVL-SFT-36k** and **SearchVL-RL-8k** datasets (full bundle + image assets).\n- [ ] Release the **data curation pipeline** (Wikipedia path sampling, fuzzy entity rewriting, source-anchor visual grounding) as a standalone toolkit.\n- [ ] Public **demo** for interactive multi-turn deep-research rollouts.\n\nAll artifacts have entered the final approval stage. Stay tuned — once the approval process is complete, we will release them **ASAP**.\n\n---\n\n## 🙌 Acknowledgements\n\nThis repository bundles and builds on several outstanding open-source projects; each sub-directory retains its upstream `LICENSE`:\n\n- [**LLaMA-Factory**](https:\u002F\u002Fgithub.com\u002Fhiyouga\u002FLLaMA-Factory) — SFT trainer and CLI (`code\u002FSFT\u002F`).\n- [**rLLM**](https:\u002F\u002Fgithub.com\u002Frllm-org\u002Frllm) and [**verl**](https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl) — agentic RL framework (`code\u002FRL\u002Frllm\u002F`).\n- [**Megatron-LM**](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM) and [**mbridge**](https:\u002F\u002Fgithub.com\u002FISEEKYAN\u002Fmbridge) — model-parallel backend (`code\u002FRL\u002FMegatron-LM\u002F`, `code\u002FRL\u002Fmbridge\u002F`).\n- [**sglang**](https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang) — async rollout engine.\n- [**Qwen-VL**](https:\u002F\u002Fgithub.com\u002FQwenLM\u002FQwen3-VL) — base VLM checkpoints.\n- We also thank the authors of [Search-R1](https:\u002F\u002Fgithub.com\u002FPeterGriffinJin\u002FSearch-R1) and [Vision-DeepResearch](https:\u002F\u002Fgithub.com\u002FAlibaba-NLP\u002FVision-DeepResearch) whose ideas inspired our multi-turn search-augmented RL formulation.\n\nProject-specific additions are released under the root [`LICENSE`](LICENSE) (Apache 2.0).\n\n---\n\n## 📮 Contact\n\nFor questions, feedback, or collaboration opportunities, feel free to reach out: csfufu0728@gmail.com\n\n\n## 📄Citation\n\nIf you find our works useful for your research, please consider citing:\n```\n@article{chen2026opensearch,\n  title={OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents},\n  author={Chen, Shuang and Feng, Kaituo and Chen, Hangting and Huang, Wenxuan and Dai, Dasen and Shou, Quanxin and Lin, Yunlong and Yue, Xiangyu and Gao, Shenghua and Pang, Tianyu},\n  journal={arXiv preprint arXiv:2605.05185},\n  year={2026}\n}\n```\n\n## ⭐️ Star HistoryMore actions\n\n[![Star History Chart](https:\u002F\u002Fapi.star-history.com\u002Fsvg?repos=shawn0728\u002FOpenSearch-VL&type=Date)](https:\u002F\u002Fstar-history.com\u002F#shawn0728\u002FOpenSearch-VL&Date)\n","OpenSearch-VL 是一个用于训练前沿多模态深度搜索代理的完全开源方案。该项目通过高质量的数据整理、多样化的视觉和搜索工具以及致命感知的代理强化学习，使模型能够以闭环方式处理图像，包括裁剪或增强感兴趣区域、执行网络和图像搜索、访问检索到的页面，并基于收集到的证据生成答案。其核心技术特点包括冷启动代理SFT、多轮次致命感知GRPO及视觉工具使用。适用于需要复杂多模态理解和交互的场景，如高级图像搜索、研究辅助等。项目采用Python开发，遵循Apache License 2.0许可协议。",2,"2026-06-11 03:31:31","CREATED_QUERY"]