[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93066":3},{"id":4,"name":5,"fullName":6,"owner":5,"repo":5,"description":7,"homepage":8,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":15,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":16,"rankGlobal":9,"rankLanguage":9,"license":17,"archived":18,"fork":18,"defaultBranch":19,"hasWiki":20,"hasPages":18,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":14,"starSnapshotCount":14,"syncStatus":12,"lastSyncTime":25,"discoverSource":26},93066,"egosteer","egosteer\u002Fegosteer","A world-model-enhanced VLA for steerable dexterous manipulation, along with optimized training infra.","https:\u002F\u002Fegosteer.github.io",null,"Python",54,2,53,0,1,41.53,"Apache License 2.0",false,"main",true,[],"2026-07-22 04:02:08","\u003Ch1 align=\"center\">EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos\u003C\u002Fh1>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosteer\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-arXiv-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white\" alt=\"Paper\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fegosteer.github.io\u002F\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-1a73e8?style=for-the-badge&logo=googlechrome&logoColor=white\" alt=\"Project Page\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fegosteer\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Data-Hugging%20Face-FFD21E?style=for-the-badge&labelColor=555555\" alt=\"Data\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FEgoSteer\u002Fmodels\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Model-Hugging%20Face-FFD21E?style=for-the-badge&labelColor=555555\" alt=\"Model\">\u003C\u002Fa>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosmith\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FEgoSmith-Data%20Pipeline-24292e?style=for-the-badge&logo=github&logoColor=white\" alt=\"EgoSmith\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fegosteer\u002Frobot-stack\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FRobot%20Stack-Teleop%20%26%20Deploy-2ea44f?style=for-the-badge&logo=github&logoColor=white\" alt=\"Robot Stack\">\u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosteer\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FEgoSteer-this%20repo-6f42c1?style=for-the-badge&logo=github&logoColor=white\" alt=\"EgoSteer\">\u003C\u002Fa>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002FTeaser.png\" width=\"100%\">\n\u003C\u002Fp>\n\nOur **full-stack system** integrates [EgoSmith](https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosmith), [Robot Stack](https:\u002F\u002Fgithub.com\u002Fegosteer\u002Frobot-stack), and [EgoSteer](https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosteer) (this repo) to learn from large-scale egocentric human videos and facilitate data-efficient real-robot post-training, enabling steerable dexterous manipulation across over 40 tasks alongside few-shot adaptation to complex, long-horizon tasks.\n\nThis repository contains **EgoSteer**, a **world-model-enhanced** Vision-Language-Action (VLA) policy built on a Qwen3-VL backbone with a flow-matching action expert. It provides a complete pipeline for training, evaluating, and serving the policy, runs on the RealMan robot out of the box, and easily extends to other embodiments.\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002Fmodel-arch.png\" alt=\"EgoSteer model architecture\" width=\"100%\">\n\u003C\u002Fp>\n\n## Installation\n\nClone the repository first, then set up an environment for training or inference:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosteer.git\ncd egosteer\n```\n\n### Full Training Environment\n\n```bash\nbash scripts\u002Finstall.sh   # In the \"Add NVIDIA repository\" step, pick the entry matching your OS.\n```\n\nThe installation script sets up system dependencies, a Python 3.10 conda environment, PyTorch with CUDA 12.8, FlashAttention, and other Python packages required for training and inference.\n\n### Minimal Inference Environment\n\nTo host a policy server only, you can manually set up the environment. Packages like pdsh and numactl are not required; use the commands below to install a minimal environment:\n\n```bash\nconda create -y -n egosteer python=3.10\nconda activate egosteer\n\npip install torch==2.10.0+cu128 torchvision==0.25.0+cu128 torchaudio==2.10.0+cu128 --extra-index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu128\n\npip install packaging ninja psutil\nMAX_JOBS=2 pip install flash-attn --no-build-isolation\npip install -r requirements.txt\n```\n\nYou can also get this environment from Docker instead of conda; see [Docker Environment](#docker-environment) under Deployment.\n\n## Pretrained Weights\n\nEgoSteer is built on two pretrained models, Qwen3-VL-2B and DINOv3, which are required for training and inference. Download them from Hugging Face, [Qwen3-VL-2B-Instruct](https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-VL-2B-Instruct) and [DINOv3](https:\u002F\u002Fhuggingface.co\u002Ffacebook\u002Fdinov3-vitl16-pretrain-lvd1689m), then set each local path in the configs below.\n\nSet the **Qwen3-VL-2B-Instruct** path in:\n\n- `pretrained.model_name_or_path` in [qwen3_vl_2b.yaml](src\u002Fconfig\u002Fmodel\u002Fqwen3_vl_2b.yaml)\n- `policy.pretrained_vlm_path` in [inference.yaml](src\u002Fconfig\u002Fexperiment\u002Finference.yaml)\n\nSet the **DINOv3** path in:\n\n- `world_model.frozen_teacher.model_name_or_path` in [frozen_regression.yaml](src\u002Fconfig\u002Fworld_model\u002Ffrozen_regression.yaml)\n- `policy.teacher_path` in [inference.yaml](src\u002Fconfig\u002Fexperiment\u002Finference.yaml)\n\nWe also release the trained EgoSteer models below. Use **EgoSteer-3B-Base** as the base for fine-tuning on your own data, and **EgoSteer-3B-RealMan** for off-the-shelf deployment on the RealMan robot.\n\n| Model Type | Model Name | Parameters | Description |\n|------------|------------|------------|-------------|\n| **EgoSteer Pretrained** | [EgoSteer-3B-Base](https:\u002F\u002Fhuggingface.co\u002FEgoSteer\u002FEgoSteer-3B-Base) | 3B | Base EgoSteer model trained on 9.6k hours of egocentric human videos, ready for fine-tuning |\n| **EgoSteer Generalist** | [EgoSteer-3B-RealMan](https:\u002F\u002Fhuggingface.co\u002FEgoSteer\u002FEgoSteer-3B-RealMan) | 3B | A generalist model post-trained on real-world data collected on the RealMan robot |\n\n## Quick Start\n\nThe fastest way to verify your setup is to train on the small example dataset.\n\n**1. Download and unpack the example data** at the repository root. The package\nis hosted on [Google Drive](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1iBPKZrHYjw3kmC0XYHfj7mpcElmBtCBR\u002Fview?usp=sharing)\nand contains both VLA and VLM dataset shards. It ships no normalizer, so you\ncompute it in step 2. Quick Start trains on the VLA shards only. For the VLM\ndata format and how to enable VLA + VLM joint training, see [`data.md`](data\u002Fdata.md).\n\n```bash\n# Option A: Command Line (gdown handles the Google Drive confirmation step)\npip install gdown\ngdown 1iBPKZrHYjw3kmC0XYHfj7mpcElmBtCBR -O example_data.zip\n\n# Option B: Manual Download\n# Download \"example_data.zip\" manually from the link above, \n# and place it at the repository root.\n\n# Unpack the dataset (required for both options)\nunzip -q example_data.zip\n```\n\nThis unpacks to:\n\n```text\nexample_data\u002F\n├── vla\u002F\n│   ├── train\u002Fshard-*.tar\n│   └── val\u002Fshard-*.tar\n└── vlm\u002F\n    ├── train\u002Fshard-*.tar\n    └── val\u002Fshard-*.tar\n```\n\n**2. Compute the state\u002Faction normalizer** from the shards:\n\n```bash\nbash scripts\u002Fcompute_norm_stats.sh\n#    -> outputs\u002Fnormalizer\u002Fexample\u002Fnormalizer.pkl\n#    -> outputs\u002Fnormalizer\u002Fexample\u002Fnormalizer.json\n```\n\n**3. Launch single-node training.** The default config already points at\n`example_data\u002F` and the normalizer above, so no further edits are needed. Make\nsure the Qwen3-VL \u002F DINOv3 paths from [Pretrained Weights](#pretrained-weights)\nare set:\n\n```bash\nbash scripts\u002Ftrain_egosteer_fsdp2_single_node.sh\n```\n\nThe script launches [train.py](train.py) through `torchrun`, sets the distributed\nenvironment variables, and uses [numa_bind_wrapper.sh](scripts\u002Fnuma_bind_wrapper.sh) when applicable.\n\n**4. Monitor on Weights & Biases.** Training logs losses, learning rates, and eval metrics such as `eval\u002Fl1_loss` to [Weights & Biases](https:\u002F\u002Fwandb.ai), under the project and entity set in [logging\u002Fdefault.yaml](src\u002Fconfig\u002Flogging\u002Fdefault.yaml). Run `wandb login` first, or set `logging.mode: offline` to turn it off.\n\n## Fine-tuning with Your Own Data\n\nTo adapt EgoSteer to your own data or embodiment, fine-tune from the released\n**EgoSteer-3B-Base** weights.\n\n**1. Prepare your data and normalizer.** Convert your data to the EgoSteer\nWebDataset format and compute a normalizer over it. See [`data.md`](data\u002Fdata.md)\nfor the shard\u002Fsample layout, coordinate conventions, and conversion guide. Then\npoint the shard paths in [vla_wds.yaml](src\u002Fconfig\u002Fdataset_paths\u002Fvla_wds.yaml) to\nyour data, and compute the normalizer:\n\n```bash\nbash scripts\u002Fcompute_norm_stats.sh   # writes outputs\u002Fnormalizer\u002Fexample\u002Fnormalizer.pkl\n```\n\nIn general, fine-tuning uses a normalizer fitted to *your* data rather than the\nreleased one. The exception is when the target robot is a humanoid whose\nembodiment gap from humans is small: because the action space is the unified\nhuman-to-robot representation, you can then reuse the pre-computed normalizer\ndirectly and skip [compute_norm_stats.sh](scripts\u002Fcompute_norm_stats.sh). The pre-computed normalizer is provided \nalongside **EgoSteer-3B-Base** in the same Hugging Face repository as `normalizer.pkl`.\n\n**2. Download the base model.** Get `model_bf16.pt` for [EgoSteer-3B-Base](https:\u002F\u002Fhuggingface.co\u002FEgoSteer\u002FEgoSteer-3B-Base) from\nHugging Face.\n\n**3. Configure the run.** In [default.yaml](src\u002Fconfig\u002Ftraining\u002Fdefault.yaml):\n\n```yaml\ntraining:\n  finetune_checkpoint_path: \u002Fpath\u002Fto\u002Fmodel_bf16.pt        # the downloaded EgoSteer-3B-Base weights\n  normalizer_path: \u002Fpath\u002Fto\u002Fyour\u002Fnormalizer.pkl\n```\n\nEgoSteer trains on an infinite WebDataset stream, so run length is measured in optimizer update steps rather than data epochs. The same `training` section also sets the run length, learning-rate schedule, evaluation interval, and checkpoint interval; follow the inline comments there.\n\n**4. Launch training.**\n\n```bash\n# Single node\nbash scripts\u002Ftrain_egosteer_fsdp2_single_node.sh\n```\n\nFor a manually managed multi-node cluster, edit the node list and network\nsettings in [train_egosteer_fsdp2.sh](scripts\u002Ftrain_egosteer_fsdp2.sh), then launch from the master node:\n\n```bash\n# Multi node\nbash scripts\u002Ftrain_egosteer_fsdp2.sh\n```\n\nPasswordless SSH must be configured across all nodes; the launcher uses `pdsh`\nover SSH to start one `torchrun` worker group per node, each node must share \nidentical source paths, Python environments, data directories, and checkpoint storage.\n\nOn a container-cloud platform (e.g., Volcano, Tencent Cloud), use\n[train_egosteer_fsdp2_cloud.sh](scripts\u002Ftrain_egosteer_fsdp2_cloud.sh) instead. This script is launched once\ninside the container of **each** node and reads the multi-node topology from\nenvironment variables the platform injects, so no SSH fan-out is needed. These\nvariables specify the master address\u002Fport, node rank, node count, and GPUs per node. Before using it,\nedit the `XXX_*` variable names near the top of the script to match the ones\nyour platform actually exports, then set it as the container start command:\n\n```bash\n# Run inside every node's container (platform sets the topology env vars)\nbash scripts\u002Ftrain_egosteer_fsdp2_cloud.sh\n```\n\n### Resuming Training\n\nTo resume an interrupted run from one of its own DCP checkpoint directories,\nset the following in [default.yaml](src\u002Fconfig\u002Ftraining\u002Fdefault.yaml):\n\n```yaml\ntraining:\n  resume: true\n  resume_checkpoint_path: \u002Fpath\u002Fto\u002Fcheckpoint        # a DCP checkpoint directory\n```\n\nUnlike fine-tuning, resuming restores the full training state, namely the model,\noptimizer, LR schedule, and step counter, so the run continues exactly where it stopped.\n\n### Checkpoint Layout\n\nEach training run writes outputs to a Hydra run directory. The saved\n`.hydra\u002Fconfig.yaml` in that same directory is the training config used by\nevaluation and serving.\n\n```text\noutputs\u002F\u003Cdate>\u002F\u003Ctime>_\u003Cname>_\u003Cexperiment>\u002F\n├── .hydra\u002F\n│   ├── config.yaml              # pass this to eval \u002F serving as the training config\n│   ├── hydra.yaml\n│   └── overrides.yaml\n├── checkpoints\u002F\n│   └── update_step=10000\u002F       # top-k DCP checkpoint directory\n│       ├── .metadata\n│       └── __*_*.distcp\n├── step_checkpoints\u002F\n│   └── update_step_10000\u002F       # periodic DCP checkpoint directory\n│       ├── .metadata\n│       └── __*_*.distcp\n└── ...\n```\n\nUse the checkpoint directory itself as `checkpoint_path`, not an individual file inside it.\n\n## Deployment\n\nFor real-robot inference, EgoSteer runs as a **WebSocket policy server**: it\nloads the policy and returns action chunks for incoming observations. The\nrobot-side client lives in the [Robot Stack](https:\u002F\u002Fgithub.com\u002Fegosteer\u002Frobot-stack)\nrepository. It captures camera\u002Fstate observations, streams them to the server, and\nexecutes the returned actions on hardware. A full\ndeployment uses **both**: the policy server described below and the Robot Stack\ncontroller talking to it.\n\nThe hardware platforms used for dexterous manipulation are shown below.\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"assets\u002FRealMan.png\" alt=\"RealMan hardware platform\" width=\"34%\">\n  \u003Cimg src=\"assets\u002FG1.png\" alt=\"G1 hardware platform\" width=\"34%\">\n\u003C\u002Fp>\n\nAll serving configuration lives in [inference.yaml](src\u002Fconfig\u002Fexperiment\u002Finference.yaml); edit\nthe fields there following their inline comments. We provide two ways to serve the\npolicy: directly from a local checkout, or from a Docker environment container\nthat mounts a local checkout.\n\n### Native Serving\n\nEdit the `EDIT` fields in `inference.yaml`, then launch the WebSocket policy\nserver from a local checkout:\n\n```bash\nbash scripts\u002Frun_server.sh\n```\n\n### Docker Environment\n\nThe Docker image contains the inference environment: CUDA, PyTorch, and\nthe Python\u002Fsystem dependencies. Run it from a local checkout so\n`create_container.sh` can mount this repository into the container, matching the\nworkflow used by the robot stack.\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fegosteer\u002Fegosteer.git\ncd egosteer\n\n# Pull the prebuilt environment image:\ndocker pull egosteerai\u002Finference-server:latest\n\n.\u002Fcreate_container.sh egosteer .\u002Fegosteer-checkpoints\n\n# Inside the container:\nbash scripts\u002Frun_server.sh\n```\n\nIf Docker Hub is unavailable in your network, pull the same image from another\nregistry and tag it as the default image name used by `create_container.sh`:\n\n```bash\ndocker pull docker-registry.psibot.net\u002Fegosteer\u002Finference-server:latest\ndocker tag docker-registry.psibot.net\u002Fegosteer\u002Finference-server:latest egosteerai\u002Finference-server:latest\n```\n\nor:\n\n```bash\ndocker pull ghcr.io\u002Fegosteer\u002Finference-server:latest\ndocker tag ghcr.io\u002Fegosteer\u002Finference-server:latest egosteerai\u002Finference-server:latest\n```\n\nYou can also build docker from `Dockerfile` with `docker build -t egosteerai\u002Finference-server:1.0.0 -t egosteerai\u002Finference-server:latest .`\n\nThe checkpoint root passed as the second argument is mounted to\n`\u002Froot\u002Fworkspace\u002Fcheckpoints` inside the container. A typical serving layout is:\n\n```text\negosteer-checkpoints\u002F\n  egosteer-realman\u002F\n    config.yaml\n    model_bf16.pt\n    normalizer-relative-10k-pretrain\u002F\n      normalizer.pkl\n  Qwen3-VL-2B-Instruct\u002F\n    ...\n  dinov3-vitl16-pretrain-lvd1689m\u002F\n    ...\n```\n\nWith this standard layout the container needs **no extra configuration**. If\nyour bundle uses a different layout, edit the paths in\n[inference.yaml](src\u002Fconfig\u002Fexperiment\u002Finference.yaml) directly before starting\nthe server.\n\nThe container starts as a long-running environment shell with this repository\nmounted at `\u002Froot\u002Fworkspace\u002Fegosteer`. Because the container uses host\nnetworking, no Docker port mapping is needed. To choose a different port, edit\n`serving.port` in [inference.yaml](src\u002Fconfig\u002Fexperiment\u002Finference.yaml) before\nstarting the server.\n\n## Evaluation\n\nOffline evaluation is configured by [eval_config.yaml](src\u002Fconfig\u002Feval_config.yaml).\n\n```bash\nbash scripts\u002Frun_eval.sh \u003Ccheckpoint_path> \u003Ctrain_config_path>\n```\n\nExample:\n\n```bash\nbash scripts\u002Frun_eval.sh \\\n    outputs\u002F\u003Cyour-run>\u002Fcheckpoints\u002Fupdate_step=10000 \\\n    outputs\u002F\u003Cyour-run>\u002F.hydra\u002Fconfig.yaml\n```\n\nThe second argument is the `config.yaml` saved by Hydra in the same training output directory as the checkpoint. For example, a checkpoint under `outputs\u002F\u003Cyour-run>\u002Fcheckpoints\u002Fupdate_step=10000\u002F` should normally use `outputs\u002F\u003Cyour-run>\u002F.hydra\u002Fconfig.yaml`.\n\nEvaluation computes action metrics and can generate a self-contained HTML report with visual overlays.\n\n## Acknowledgements\n\nEgoSteer builds on a number of excellent open-source projects, and we thank their authors and communities.\n\n**Third-party code:**\n\n- [openpi](https:\u002F\u002Fgithub.com\u002FPhysical-Intelligence\u002Fopenpi)\n- [diffusion_policy](https:\u002F\u002Fgithub.com\u002Freal-stanford\u002Fdiffusion_policy)\n- [Hugging Face Transformers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers)\n- [PyTorch3D](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fpytorch3d)\n- [transforms3d](https:\u002F\u002Fgithub.com\u002Fmatthew-brett\u002Ftransforms3d)\n- [TorchTitan](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan)\n- [TorchTNT](https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftnt)\n- [kornia](https:\u002F\u002Fgithub.com\u002Fkornia\u002Fkornia)\n- [pytorch_HMR](https:\u002F\u002Fgithub.com\u002FMandyMo\u002Fpytorch_HMR)\n\n**Pretrained models & assets:**\n\n- [Qwen3-VL](https:\u002F\u002Fgithub.com\u002FQwenLM\u002FQwen3-VL)\n- [DINOv3](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdinov3)\n\nSee [NOTICE](NOTICE) for the full copyright and license notices.\n\n## Citation\n\nIf you find EgoSteer useful in your research, please consider citing:\n\n```bibtex\n@article{egosteer2026,\n  title   = {EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos},\n  author  = {EgoSteer Team},\n  journal = {arXiv preprint arXiv:XXXX.XXXXX},\n  year    = {2026},\n}\n```\n","EgoSteer 是一个面向主视角视频的可操控灵巧操作视觉-语言-动作（VLA）系统，通过引入世界模型增强策略建模能力，并结合流匹配动作专家实现高精度动作生成。其核心基于 Qwen3-VL 多模态骨干网络，提供端到端训练、评估与部署流水线，支持 RealMan 等真实机器人开箱即用，具备少样本适配复杂长程任务的能力。适用于机器人灵巧操作研究、主视角模仿学习、真实场景数据高效微调等场景。","2026-07-11 02:30:49","CREATED_QUERY"]