[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-1609":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":15,"forks30d":15,"starsTrendScore":19,"compositeScore":20,"rankGlobal":9,"rankLanguage":9,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":24,"hasPages":22,"topics":25,"createdAt":9,"pushedAt":9,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":15,"starSnapshotCount":15,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},1609,"GR00T-VisualSim2Real","NVlabs\u002FGR00T-VisualSim2Real","NVlabs","GR00T-VisualSim2Real: Open-source sim-to-real framework for humanoid visual loco-manipulation. Train in simulation, deploy zero-shot on real robots with RGB + proprioception for tasks like pick-and-place and door opening.",null,"Python",304,29,5,3,0,9,23,116,27,83.93,"Apache License 2.0",false,"main",true,[],"2026-06-12 04:00:10","\u003Cdiv align=\"center\">\n\n# GR00T-VisualSim2Real\n\n\n\u003Ctable>\n  \u003Ctr>\n    \u003Ctd align=\"center\" valign=\"top\" width=\"50%\">\n      \u003Cp>\u003Cb>VIRAL\u003C\u002Fb>\u003Cbr>\u003Cbr>\u003Cb>Visual Sim-to-Real at Scale for\u003Cbr>Humanoid Loco-Manipulation\u003C\u002Fb>\u003C\u002Fp>\n      \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2511.15200\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2511.15200-b31b1b.svg\" alt=\"VIRAL Paper\">\u003C\u002Fa>\n      \u003Ca href=\"https:\u002F\u002Fviral-humanoid.github.io\u002F\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-blue.svg\" alt=\"VIRAL Project Page\">\u003C\u002Fa>\n      \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FNVlabs\u002FGR00T-VisualSim2Real\u002Ftree\u002Fmain\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode-viral-lightgrey.svg\" alt=\"VIRAL Code\">\u003C\u002Fa>\n      \u003Cbr>\u003Cbr>\n      \u003Cimg src=\".\u002Fmedia\u002Fviral-teaser.gif\" width=\"100%\">\n    \u003C\u002Ftd>\n    \u003Ctd align=\"center\" valign=\"top\" width=\"50%\">\n      \u003Cp>\u003Cb>DoorMan\u003C\u002Fb>\u003Cbr>\u003Cbr>\u003Cb>Opening the Sim-to-Real Door for\u003Cbr>Humanoid Pixel-to-Action Policy Transfer\u003C\u002Fb>\u003C\u002Fp>\n      \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2512.01061\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2512.01061-b31b1b.svg\" alt=\"DoorMan Paper\">\u003C\u002Fa>\n      \u003Ca href=\"https:\u002F\u002Fdoorman-humanoid.github.io\u002F\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-blue.svg\" alt=\"DoorMan Project Page\">\u003C\u002Fa>\n      \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FNVlabs\u002FGR00T-VisualSim2Real\u002Ftree\u002Fdoorman\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode-doorman-lightgrey.svg\" alt=\"DoorMan Code\">\u003C\u002Fa>\n      \u003Cbr>\u003Cbr>\n      \u003Cimg src=\".\u002Fmedia\u002Fdoorman-teaser.gif\" width=\"100%\">\n    \u003C\u002Ftd>\n  \u003C\u002Ftr>\n\u003C\u002Ftable>\n\n\n\u003C\u002Fdiv>\n\n\u003Cbr\u002F>\n\n## Overview\nThis repository contains the application code for **VIRAL** (Visual Sim-to-Real for Humanoid Loco-Manipulation) and **DoorMan**. The system enables humanoid robots (e.g., Unitree G1) to perform complex tasks like opening heavy doors in the real world through a teacher-student simulation framework.\n\nThis repository contains the official code for:\n\n- **VIRAL**: *Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation*\n    [Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2511.15200)  |  [Project](https:\u002F\u002Fviral-humanoid.github.io\u002F)  |  [Code](https:\u002F\u002Fgithub.com\u002FNVlabs\u002FGR00T-VisualSim2Real\u002Ftree\u002Fmain)\n- **DoorMan**: *Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer*\n    [Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2512.01061)  |  [Project](https:\u002F\u002Fdoorman-humanoid.github.io\u002F)  |  [Code](https:\u002F\u002Fgithub.com\u002FNVlabs\u002FGR00T-VisualSim2Real\u002Ftree\u002Fdoorman)\n\n---\n\n\n# VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation\n\nA reinforcement learning framework for humanoid robot loco-manipulation on the **Unitree G1** robot. The codebase supports:\n\n- **Teacher Training** -- PPO-based policy training with privileged state observations\n- **Student Training** -- Vision-based policy distillation (DAgger) from a trained teacher using RGB camera input\n- **Evaluation** -- Evaluate trained teacher or student checkpoints, with optional ONNX export for deployment\n\nBuilt on [Isaac Lab](https:\u002F\u002Fisaac-sim.github.io\u002FIsaacLab\u002F) (Isaac Sim 5.1), [TRL](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl), and [Hydra](https:\u002F\u002Fhydra.cc\u002F) for configuration management.\n\n## Table of Contents\n\n- [GR00T-VisualSim2Real](#gr00t-visualsim2real)\n  - [Overview](#overview)\n- [VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation](#viral-visual-sim-to-real-at-scale-for-humanoid-loco-manipulation)\n  - [Table of Contents](#table-of-contents)\n  - [Prerequisites](#prerequisites)\n  - [Installation](#installation)\n    - [1. Create conda environment](#1-create-conda-environment)\n    - [2. Install Isaac Sim 5.1](#2-install-isaac-sim-51)\n    - [3. Install Isaac Lab](#3-install-isaac-lab)\n    - [4. Install this package](#4-install-this-package)\n    - [5. Verify installation](#5-verify-installation)\n  - [Usage](#usage)\n    - [Teacher Training (PPO)](#teacher-training-ppo)\n    - [Teacher Evaluation](#teacher-evaluation)\n    - [Student Training (DAgger Distillation)](#student-training-dagger-distillation)\n    - [Student Evaluation](#student-evaluation)\n  - [Configuration](#configuration)\n    - [Experiment Tracking](#experiment-tracking)\n  - [ONNX Export](#onnx-export)\n  - [Project Structure](#project-structure)\n  - [License](#license)\n  - [Copyright \\& Attribution](#copyright--attribution)\n  - [Citation](#citation)\n\n## Prerequisites\n\n- Ubuntu 22.04\n- NVIDIA GPU with driver >= 535\n- [Isaac Sim 5.1](https:\u002F\u002Fdocs.omniverse.nvidia.com\u002Fisaacsim\u002Flatest\u002Finstallation\u002Finstall_workstation.html)\n- [Isaac Lab](https:\u002F\u002Fisaac-sim.github.io\u002FIsaacLab\u002F)\n- Conda or Mamba\n\n## Installation\n\n### 1. Create conda environment\n\n```bash\nconda create -n viral python=3.11 -y\nconda activate viral\n```\n\n### 2. Install Isaac Sim 5.1\n\n```bash\npip install torch==2.7.0 torchvision==0.22.0 --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu128\n\npip install isaacsim==5.1.0.0 isaacsim-rl==5.1.0.0\n```\n\n### 3. Install Isaac Lab\n\nClone or download [Isaac Lab](https:\u002F\u002Fgithub.com\u002Fisaac-sim\u002FIsaacLab) and install:\n\n```bash\npip install setuptools poetry-core flatdict\n\npip install --no-build-isolation -e \u003Cpath-to-IsaacLab>\u002Fsource\u002Fisaaclab\npip install --no-build-isolation -e \u003Cpath-to-IsaacLab>\u002Fsource\u002Fisaaclab_assets \\\n    -e \u003Cpath-to-IsaacLab>\u002Fsource\u002Fisaaclab_tasks \\\n    -e \"\u003Cpath-to-IsaacLab>\u002Fsource\u002Fisaaclab_rl[all]\"\n\npip install numpy==1.26.0\n```\n\nVerify the install:\n\n```bash\npython -c \"import isaaclab; print(isaaclab.__file__)\"\n```\n\n### 4. Install this package\n\n```bash\ncd \u003Cpath-to-this-repo>\npip install -e .\n\npip install numpy==1.26.0   # pip may upgrade it; pin again\n```\n\n### 5. Verify installation\n\n```bash\npython -c \"from gr00t.rl.envs.base_task.base_task import BaseTask; print('OK')\"\n```\n\n## Usage\n\n### Teacher Training (PPO)\n\nTrain a teacher policy using privileged state observations:\n\n```bash\nHYDRA_FULL_ERROR=1 accelerate launch --num_processes 1 \\\n    gr00t\u002Frl\u002Ftrain_agent_trl.py \\\n    +exp=loco_manip\u002Fwalk_stand_place_grasp_turn_homie \\\n    num_envs=48 \\\n    project_name=wsdpt_teacher\n```\n\n> **Tip:** Add `headless=False` to open the Isaac Sim GUI and watch training live.\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\".\u002Fmedia\u002Fviral-teacher-gif.gif\" width=\"60%\">\u003Cbr\u002F>\n  \u003Cem>Teacher policy running in Isaac Sim\u003C\u002Fem>\n\u003C\u002Fp>\n\n| Argument | Description |\n|---|---|\n| `num_envs` | Number of parallel environments (higher = faster, more VRAM) |\n| `project_name` | Weights & Biases project name |\n| `headless` | `True` (default) for headless; `False` to open GUI |\n| `env.config.reset_from_dataset.enable` | Reset from demonstration dataset |\n\n### Teacher Evaluation\n\n```bash\npython gr00t\u002Frl\u002Feval_agent_trl.py \\\n    +checkpoint=logs_rl\u002F\u003Cexperiment_dir>\u002Fmodel_step_044500.pt\n```\n\n### Student Training (DAgger Distillation)\n\nTrain a vision-based student policy by distilling from a trained teacher:\n\n1. Update the teacher checkpoint path in the experiment config:\n\n```yaml\n# gr00t\u002Frl\u002Fconfig\u002Fexp\u002Floco_manip\u002Fwsdpt_student_for_teacher_v8q8.002_resnet_rgb_delay.yaml\nteacher_actor_path: logs_rl\u002F\u003Cyour_teacher_experiment>\u002Fmodel_step_XXXXXX.pt\n```\n\n2. Launch training:\n\n```bash\nHYDRA_FULL_ERROR=1 accelerate launch --num_processes 1 \\\n    gr00t\u002Frl\u002Ftrain_agent_trl.py \\\n    +exp=loco_manip\u002Fwsdpt_student_for_teacher_v8q8.002_resnet_rgb_delay \\\n    num_envs=8 \\\n    headless=True \\\n    experiment_name=wsdpt_student \\\n    project_name=wsdpt_student_debug\n```\n\nIf you add `headless=False`, you can see the student policy running in Isaac Sim:\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\".\u002Fmedia\u002Fviral-student-gif.gif\" width=\"60%\">\u003Cbr\u002F>\n  \u003Cem>Student policy running in Isaac Sim\u003C\u002Fem>\n\u003C\u002Fp>\n\n### Student Evaluation\n\n```bash\npython gr00t\u002Frl\u002Feval_agent_trl.py \\\n    +checkpoint=logs_rl\u002F\u003Cstudent_experiment_dir>\u002Fmodel_step_XXXXXX.pt\n```\n\n## Configuration\n\nThis project uses [Hydra](https:\u002F\u002Fhydra.cc\u002F) for configuration. Configs are composed from YAML files in `gr00t\u002Frl\u002Fconfig\u002F`. Override any value from the command line:\n\n```bash\n# Number of environments\n... num_envs=16\n\n# Reward weights\n... rewards.reward_scales.tracking_lin_vel=1.0\n\n# Training hyperparameters\n... algo.config.actor_learning_rate=1e-4\n```\n\n### Experiment Tracking\n\nTraining logs to [Weights & Biases](https:\u002F\u002Fwandb.ai\u002F) by default:\n\n```bash\nwandb login\n```\n\nCheckpoints are saved to `logs_rl\u002F\u003Cexperiment_name>\u002F` at intervals controlled by the `save_frequency` callback parameter.\n\n## ONNX Export\n\nDuring evaluation with `num_envs=1`, the policy is automatically exported as ONNX for deployment:\n\n```bash\npython gr00t\u002Frl\u002Feval_agent_trl.py \\\n    +checkpoint=\u003Cpath_to_checkpoint.pt> \\\n    num_envs=1\n```\n\nThe exported model is saved to `\u003Cexperiment_dir>\u002Fexported\u002F`.\n\n\n## Project Structure\n\n```\ngr00t\u002Frl\u002F\n├── train_agent_trl.py          # Training entry point (teacher & student)\n├── eval_agent_trl.py           # Evaluation entry point\n├── config\u002F                     # Hydra YAML configs\n│   ├── base.yaml               #   Base training config\n│   ├── base_eval.yaml          #   Base evaluation config\n│   ├── exp\u002Floco_manip\u002F         #   Experiment configs\n│   ├── algo\u002F                   #   Algorithm configs (PPO, DAgger)\n│   ├── env\u002F                    #   Environment configs\n│   ├── robot\u002Fg1\u002F               #   Robot configs (G1 43-DOF)\n│   ├── rewards\u002F                #   Reward function configs\n│   ├── terrain\u002F                #   Terrain configs\n│   ├── obs\u002F                    #   Observation configs\n│   └── domain_rand\u002F            #   Domain randomization configs\n├── envs\u002F                       # Environment implementations\n│   ├── base_task\u002F              #   Base task classes\n│   └── loco_manip\u002F             #   Loco-manipulation task\n├── trl\u002F                        # TRL-based trainers and modules\n│   ├── trainer\u002F                #   PPO and distillation trainers\n│   ├── modules\u002F                #   Actor-critic network modules\n│   ├── callbacks\u002F              #   Training callbacks\n│   └── utils\u002F                  #   Training utilities\n├── agents\u002Fmodules\u002F             # Neural network building blocks\n├── simulator\u002Fisaacsim\u002F         # Isaac Sim interface\n├── data\u002F                       # Task data (robot assets, scenarios)\n└── utils\u002F                      # General utilities\n```\n\n## License\n\nThis project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.\n\n## Copyright & Attribution\n\nCopyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n\nThis project includes third-party open-source software. Please refer to individual source files for specific licenses and copyright headers.\n\n## Citation\n\nIf you find this work useful, please cite:\n\n```bibtex\n@inproceedings{he2025viral,\n    title={VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation},\n    author={He, Tairan and Wang, Zi and Xue, Haoru and Ben, Qingwei and Luo, Zhengyi and Xiao, Wenli and Yuan, Ye and Da, Xingye and Castañeda, Fernando and Sastry, Shankar and Liu, Changliu and Shi, Guanya and Fan, Linxi and Zhu, Yuke},\n    booktitle={Proceedings of the IEEE\u002FCVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    year={2026}\n}\n\n@inproceedings{xue2025opening,\n  title={Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer},\n  author={Xue, Haoru and He, Tairan and Wang, Zi and Ben, Qingwei and Xiao, Wenli and Luo, Zhengyi and Da, Xingye and Casta{\\~n}eda, Fernando and Shi, Guanya and Sastry, Shankar and others},\n  booktitle={Proceedings of the IEEE\u002FCVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year={2026}\n}\n```\n","GR00T-VisualSim2Real 是一个开源的仿真到现实框架，用于人形机器人的视觉定位和操作任务。该项目通过在仿真环境中训练模型，并利用RGB图像和本体感觉数据实现零样本迁移至真实机器人上执行诸如拾取放置和开门等任务。它基于Isaac Sim 5.1构建，采用PPO算法进行教师模型的策略训练，并通过DAgger方法从已训练好的教师模型中蒸馏出基于视觉的学生模型策略。适用于需要在复杂环境中执行精细操作的人形机器人应用场景，如服务机器人、救援机器人等领域。",2,"2026-06-11 02:44:58","CREATED_QUERY"]