[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92999":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":19,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":27,"readmeContent":28,"aiSummary":29,"trendingCount":15,"starSnapshotCount":15,"syncStatus":30,"lastSyncTime":31,"discoverSource":32},92999,"PoseVLA","hetolin\u002FPoseVLA","hetolin","Universal Pose Pretraining for Generalizable Vision-Language-Action Policies(RSS2026)","https:\u002F\u002Fhetolin.github.io\u002FPoseVLA",null,"Python",60,1,51,0,4,41.3,"MIT License",false,"main",true,[23,24,25,26],"co-training","manipulation","vla","vlm","2026-07-22 04:02:07","\u003Cdiv align=\"center\">\n\n# PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies (RSS2026)\n\n[\u002F\u002F]: # (Purely HuggingFace + Accelerate + DeepSpeed + Hydra based — concise code, multi-node ready, easy to extend.)\n\n[![arXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2602.19710-b31b1b.svg)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2602.19710)\n[![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject_Page-PoseVLA-2ea44f.svg)](https:\u002F\u002Fhetolin.github.io\u002FPoseVLA\u002F)\n[![ModelScope Model](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FModelScope-Model-624aff)](https:\u002F\u002Fwww.modelscope.ai\u002Fmodels\u002Fhanyangyu1021\u002FPoseVLA-robotwin\u002Ffiles)\n\n[\\[🚀 Quick Start\\]](#-quick-start) [\\[🌟 Pre-train\\]](docs\u002FPRETRAIN.md) [\\[🤖 Post-train (Robotwin)\\]](docs\u002FPOSTTRAIN.md) [\\[🕹 RoboTwin Eval\\]](robotwin\u002FPoseVLA\u002FREADME.md) [\\[🐛 Troubleshooting\\]](docs\u002FPRETRAIN.md#-troubleshooting)\n\n\u003Cimg src=\"assets\u002Fteaser.png\" width=\"100%\" \u002F>\n\n\u003C\u002Fdiv>\n\n---\n\n## News 🚀🚀🚀\n- `2026\u002F06`: Initial release of **PoseVLA**: supports Omni3D \u002F Omni6D \u002F BOP \u002F GraspClutter6D for 3D tasks and Agibot \u002F InternData-A1 for robot actions.\n\n---\n\n## 🏆 Main Results on RoboTwin (50 tasks, Average %)\n\n| Method                | Easy | Hard |\n|:----------------------| :---: | :---: |\n| π0                    | 67.00 | 65.12 |\n| π0.5                  | 79.48 | 76.16 |\n| PaliGemma_expert      | 35.40 | 33.36 |\n| Pose-VLA (In Paper)   | 79.91 | 79.10 |\n| **PoseVLA (In Repo)** | **89.40** | **88.60** |\n\n---\n\n## 📖 Documents\n\nPoseVLA is split into two training stages, each with its own document:\n\n| Stage | Entry script | Config | Doc |\n| --- | --- | --- | --- |\n| **Pre-train** (joint VLM + Action on large-scale data) | [train_pretrain.py](train_pretrain.py) | [config\u002Fbase.yaml](config\u002Fbase.yaml) | [docs\u002FPRETRAIN.md](docs\u002FPRETRAIN.md) |\n| **Robotwin data conversion** (raw → HDF5 + normalization) | [utils\u002Fprocess_data_all.py](utils\u002Fprocess_data_all.py) | [config\u002Fdataset\u002Frobotwin.yaml](config\u002Fdataset\u002Frobotwin.yaml) | [docs\u002FROBOTWIN_DATA.md](docs\u002FROBOTWIN_DATA.md) |\n| **Post-train \u002F Fine-tune** (Robotwin and other downstream robots) | [train_posttrain.py](train_posttrain.py) | [config\u002Fbase_posttrain.yaml](config\u002Fbase_posttrain.yaml) | [docs\u002FPOSTTRAIN.md](docs\u002FPOSTTRAIN.md) |\n| **RoboTwin simulation eval** | [robotwin\u002FPoseVLA\u002Feval_policy.py](robotwin\u002FPoseVLA\u002Feval_policy.py) | [robotwin\u002FPoseVLA\u002Fdeploy_policy.yml](robotwin\u002FPoseVLA\u002Fdeploy_policy.yml) | [robotwin\u002FPoseVLA\u002FREADME.md](robotwin\u002FPoseVLA\u002FREADME.md) |\n\nThe project supports three orthogonal training modes that can be freely combined:\n\n- **VLM training** — learn 3D object detection via **Next-Token Prediction (NTP)** on Omni3D, Omni6D, BOP, GraspClutter6D, …\n- **Action training** — learn robot actions via **Flow Matching** on HDF5 \u002F LeRobot-format data (Agibot, InternData-A1, Robotwin, …).\n- **Co-Training** — VLM and Action data are interleaved within the same optimization step.\n\n\u003Cdetails>\n\u003Csummary>\u003Cb>📁 Project Structure\u003C\u002Fb> (click to expand)\u003C\u002Fsummary>\n\n```\nPoseVLA\u002F\n├── train_pretrain.py           # Pre-train entry (hydra + accelerate + deepspeed)\n├── train_posttrain.py          # Post-train \u002F fine-tune entry (Robotwin etc.)\n├── eval_detection.py           # Evaluation \u002F mAP entry (Omni3D and other 3D tasks)\n│\n├── posevla\u002F                    # PoseVLA model implementation (π0 \u002F π0.5 based)\n│   ├── configuration_posevla.py    # PoseVLAConfig\n│   ├── modeling_posevla.py         # PoseVLAPolicy (PaliGemma + Action Expert + Flow Matching)\n│   ├── paligemma_with_expert.py    # PaliGemma + Expert dual-stream architecture\n│   ├── patch_embed.py              # Vision patch embedding (SigLIP + prior fusion)\n│   ├── convert_jax_model_to_pytorch.py  # JAX → PyTorch weight converter\n│   └── _lerobot_compat.py         # LeRobot compatibility layer\n│\n├── data\u002F\n│   ├── factory.py              # Unified VLM \u002F Action DataLoader factory\n│   ├── collators.py            # DataCollators (action \u002F detection)\n│   ├── ds_raw\u002F                 # Raw dataset readers\n│   │   ├── agibot.py \u002F interndata_a1.py \u002F robotwin.py\n│   │   ├── droid.py \u002F rdt.py \u002F umi.py \u002F xtrainer.py\n│   │   └── mix.py             # Multi-source mixed dataset reader\n│   └── ds_train\u002F               # Training Datasets\n│       ├── detection\u002F          # 3D detection datasets (VLM)\n│       │   ├── dataset_bop.py \u002F dataset_omni3d.py \u002F dataset_omni6d.py\n│       │   ├── dataset_clutter.py\n│       │   └── graspclutter6dAPI.py\n│       └── robot\u002F              # Robot action datasets (VLA)\n│           ├── dataset_hdf5_action.py   # HDF5 action dataset\n│           ├── dataset_lerobot.py       # LeRobot format dataset\n│           └── dataset_agibot.py        # Agibot dataset\n│\n├── utils\u002F                      # Shared utilities\n│   ├── process_data_all.py     # Robotwin raw → HDF5 conversion\n│   ├── mapping_token.py        # Text ↔ 3D scene encoding \u002F decoding utilities\n│   ├── transform_utils.py      # SE(3) \u002F pose math helpers\n│   ├── image_corrupt.py        # Image augmentation\n│   ├── vis.py                  # Visualization helpers\n│   └── logger.py               # WandB \u002F training-state logging\n│\n├── config\u002F                     # Hydra configs\n│   ├── base.yaml               # Pre-train main config\n│   ├── base_posttrain.yaml     # Post-train (Robotwin) main config\n│   ├── zero0.json \u002F zero2.json \u002F zero3_offload.json   # DeepSpeed configs\n│   ├── dataset\u002F                # Action training dataset configs (hdf5, lerobot, robotwin)\n│   ├── dataset_bop\u002F            # BOP series\n│   ├── dataset_clutter\u002F        # GraspClutter6D\n│   ├── dataset_omni6d\u002F         # Omni6D\n│   ├── dataset_omni3d\u002F         # Omni3D (train\u002Fval\u002Ftest)\n│   ├── dataset_lerobot\u002F        # LeRobot grouped configs\n│   └── dataset_meta\u002F           # Per-source sample lists (json)\n│\n├── scripts\u002F\n│   ├── launch\u002F                 # Training launch scripts\n│   │   ├── pretrain.sh         # Multi-GPU launch for pre-training (train_pretrain.py)\n│   │   └── posttrain.sh        # Multi-GPU launch for post-training (train_posttrain.py)\n│   ├── download\u002F               # Dataset download \u002F extract \u002F yaml-gen scripts\n│   │   ├── agibot.sh\n│   │   ├── interndata_a1.sh\n│   │   ├── interndata_a1_unzip.sh\n│   │   └── interndata_a1_generate_yaml.py\n│   └── stats\u002F                  # Dataset normalization stats\n│       ├── compute_dataset_stat_hdf5_abs_joint.py\n│       ├── compute_dataset_stat_hdf5_rel_ee.py\n│       ├── norm_robotwin.py    # Robotwin EEP \u002F qpos normalization stats\n│       └── normalize.py\n│\n├── bin_stats\u002F                  # Non-uniform tokenizer bin boundaries\n│   └── nonuniform_bins.pkl     # Pre-computed quantization bins for 3D tokenization\n│\n├── docs\u002F                       # Stage-level documents\n│   ├── PRETRAIN.md             # Pre-train guide\n│   ├── POSTTRAIN.md            # Post-train \u002F fine-tune guide (Robotwin)\n│   └── ROBOTWIN_DATA.md        # Robotwin raw → HDF5 conversion guide\n│\n├── robotwin\u002FPoseVLA\u002F           # RoboTwin simulation deploy + eval\n└── google\u002Fpaligemma-3b-pt-224\u002F # Local PaliGemma tokenizer \u002F config\n```\n\n\u003C\u002Fdetails>\n\n---\n\n## 🚀 Quick Start\n\n### 1. Clone\n\n```bash\ngit clone git@github.com:hetolin\u002FPoseVLA.git\ncd PoseVLA\n```\n\n### 2. Conda env (installation from scratch)\n\n- Python 3.10.12\n- PyTorch 2.7.0 + CUDA 12.6 (TF32 \u002F bf16 works best on Ampere \u002F Hopper GPUs)\n\n```bash\nconda create -n vla python==3.10.12\nconda activate vla\n\n# Install PyTorch first — it is intentionally NOT pinned in requirements.txt,\n# because the correct wheel depends on your local CUDA version.\n# CUDA 12.6:\npip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 \\\n    --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu126\n\n# Install lerobot (pinned commit, compatible with PoseVLA)\n# Fix: rename `pyav` -> `av` (package renamed on PyPI)\n# Fix: remove `rerun-sdk` (requires numpy>=2, conflicts with imgaug)\ngit clone https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Flerobot.git third_party\u002Flerobot \\\n  && cd third_party\u002Flerobot \\\n  && git checkout 638d411cd3acf32c28d8c2120f3c41bda8bb15d4 \\\n  && sed -i 's\u002Fpyav\u002Fav\u002F' pyproject.toml \\\n  && sed -i '\u002Frerun-sdk\u002Fd' pyproject.toml \\\n  && pip install -e . \\\n  && cd ..\u002F..\n\n# (Optional) Install bop_toolkit (used for BOP dataset, for 3D pretraining only)\ngit clone https:\u002F\u002Fgithub.com\u002Fthodan\u002Fbop_toolkit.git third_party\u002Fbop_toolkit \\\n  && cd third_party\u002Fbop_toolkit \\\n  && pip install -e . \\\n  && cd ..\u002F..\n\n# NOTE: keep the `lerobot` line commented out in requirements.txt — we install it manually.\npip install -r requirements.txt\n\n```\n\n### 3. Pretrained weights\n\nCreate a `pretrain\u002F` directory under the project root and place any of the following weights (pick what you need — multiple loading branches are available in [train_pretrain.py](train_pretrain.py)):\n\n```\npretrain\u002F\n├── paligemma-3b-pt-224\u002F   # Vanilla PaliGemma VLM\n├── lerobot_pi0\u002F           # π0\n└── pi05_base\u002F             # π0.5\n```\n\nThe tokenizer is already bundled under [google\u002Fpaligemma-3b-pt-224\u002F](google\u002Fpaligemma-3b-pt-224\u002F); the config points to it via `model.tokenizer_model_path`.\n\n### 4. Environment variables\n\nSet the project parent directory before launching Hydra-based training scripts. At minimum:\n\n```bash\nexport ROOT=\"\u002Fyour\u002Fhome\"\nexport DEV_PATH=\"${ROOT}\u002Frobot_code\"\nexport PYTHONPATH=\"$PYTHONPATH:${DEV_PATH}\"\nexport HF_HOME=${ROOT}\u002F.cache\u002Fhuggingface\nexport HF_LEROBOT_HOME=${HF_HOME}\u002Flerobot\nexport HYDRA_FULL_ERROR=1\n```\n\nW&B auto-login: the script writes `\u002Froot\u002F.netrc` — replace `API_KEY` with your own.\n\n### 5. Next step\n\n- For **pre-training** the PoseVLA backbone on large-scale 3D + action data, follow [docs\u002FPRETRAIN.md](docs\u002FPRETRAIN.md).\n- For **post-training \u002F fine-tuning** on RoboTwin (or your own robot), follow [docs\u002FPOSTTRAIN.md](docs\u002FPOSTTRAIN.md).\n- For **simulation evaluation** on RoboTwin, follow [robotwin\u002FPoseVLA\u002FREADME.md](robotwin\u002FPoseVLA\u002FREADME.md).\n\n---\n\n## TODO List\n\n- [x] Release the training \u002F co-training code for PoseVLA.\n- [x] Release the 3D evaluation entry (`eval_detection.py`) with mAP \u002F PR-curve aggregation.\n- [x] Release support for both **π0** and **π0.5** Action Experts in a single codebase (switchable via `training.pi05`).\n- [x] Release the Robotwin post-training entry (`train_posttrain.py`) and RoboTwin deployment scripts.\n- [x] Release pretrained PoseVLA checkpoints.\n\n## 🙋 FAQs\n\nIf you encounter any issues, feel free to open an issue on GitHub or reach out through discussions. Feedback and contributions are very welcome! 🚀\n\n## Acknowledgement\n\nPoseVLA is built with reference to the following projects:\n[lerobot](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Flerobot),\n[Transformers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers),\n[Google PaliGemma](https:\u002F\u002Fhuggingface.co\u002Fgoogle\u002Fpaligemma-3b-pt-224),\n[π0 \u002F OpenPI](https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi),\n[Omni3D](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fomni3d),\n[BOP Toolkit](https:\u002F\u002Fgithub.com\u002Fthodan\u002Fbop_toolkit),\nand [GraspClutter6D](https:\u002F\u002Fgithub.com\u002FSeungBack\u002FGraspClutter6D).\nThanks for their awesome work.","PoseVLA 是一个面向具身智能的通用姿态预训练框架，旨在构建可泛化的视觉-语言-动作（VLA）策略。其核心通过联合预训练视觉语言模型（VLM）与机器人动作建模，支持3D位姿估计（Omni6D\u002FBOP等）和机器人操作动作生成，并创新性地采用共训练（co-training）机制，在同一优化步中交替处理VLM和动作数据；技术上基于HuggingFace生态，兼容DeepSpeed与Hydra，支持多节点分布式训练。适用于机器人仿真评估（如RoboTwin）、跨任务泛化动作策略学习及3D场景理解驱动的具身控制研究。",2,"2026-07-11 02:30:35","CREATED_QUERY"]