[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93071":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":11,"openIssues":13,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":14,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":15,"rankGlobal":9,"rankLanguage":9,"license":16,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":19,"hasPages":19,"topics":20,"createdAt":9,"pushedAt":9,"updatedAt":21,"readmeContent":22,"aiSummary":23,"trendingCount":14,"starSnapshotCount":14,"syncStatus":13,"lastSyncTime":24,"discoverSource":25},93071,"vla-corrector","ZJU-OmniAI\u002Fvla-corrector","ZJU-OmniAI","a lightweight detect-and-correction inference for vla",null,"Python",51,4,2,0,42.1,"Apache License 2.0",false,"main",true,[],"2026-07-22 04:02:08","\u003Cdiv align=\"center\">\n  \u003Cimg src=\"docs\u002Fassets\u002Fbranding\u002Fvla_corrector_logo.png\" alt=\"VLA-Corrector logo\" width=\"240\">\n\n  # VLA-Corrector\n\n  \u003Cp>\n    \u003Cstrong>English\u003C\u002Fstrong> | \u003Ca href=\"README_zh.md\">中文\u003C\u002Fa>\n  \u003C\u002Fp>\n\n  ### VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizon\n\n  \u003Cp>\n    Yi Pan\u003Csup>1\u003C\u002Fsup>, Miao Pan\u003Csup>1\u003C\u002Fsup>, Qi Lu\u003Csup>1\u003C\u002Fsup>, Jiaming Huang\u003Csup>1\u003C\u002Fsup>,\n    Man Zhang\u003Csup>1\u003C\u002Fsup>, Siteng Huang\u003Csup>2\u003C\u002Fsup>, Xin Li\u003Csup>2\u003C\u002Fsup>, Jie Zhang\u003Csup>1\u003C\u002Fsup>,\n    Yongliang Shen\u003Csup>1\u003C\u002Fsup>, Xuhong Zhang\u003Csup>1\u003C\u002Fsup>, Wenqi Zhang\u003Csup>1\u003C\u002Fsup>\u003Cbr>\n    \u003Csup>1\u003C\u002Fsup>Zhejiang University   ·   \u003Csup>2\u003C\u002Fsup>Alibaba DAMO Academy\u003Cbr>\n    \u003Ca href=\"mailto:panyi0304@gmail.com\">panyi0304@gmail.com\u003C\u002Fa>   ·  \n    \u003Ca href=\"mailto:zhangwenqi@zju.edu.cn\">zhangwenqi@zju.edu.cn\u003C\u002Fa>\n  \u003C\u002Fp>\n\n  \u003Cbr\u002F>\n\n  [![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject_Page-1A73E8?style=for-the-badge&logo=googlechrome&logoColor=white)](https:\u002F\u002Fzju-omniai.github.io\u002Fvla-corrector\u002F)\n  [![Code](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode-181717?style=for-the-badge&logo=github&logoColor=white)](https:\u002F\u002Fgithub.com\u002FZJU-OmniAI\u002Fvla-corrector)\n  [![arXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2607.01804-B31B1B?style=for-the-badge&logo=arxiv&logoColor=white)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01804)\n  [![BibTeX](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCite-BibTeX-4C8C4A?style=for-the-badge&logo=googlescholar&logoColor=white)](#citation)\n\n  \u003Cbr\u002F>\n\u003C\u002Fdiv>\n\n\u003Cdiv align=\"center\">\n  \u003Cvideo src=\"https:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Feb2b70f7-f8d9-4d18-b85a-7014949fde7b\" controls muted width=\"80%\">\u003C\u002Fvideo>\n\u003C\u002Fdiv>\n\n---\n\n**VLA-Corrector** is a lightweight detect-and-correct inference framework for action-chunked Vision-Language-Action (VLA) policies. It addresses the open-loop blind spot created by fixed action horizons: fresh observations arrive during execution, but the policy continues following queued actions until the horizon ends.\n\nVLA-Corrector keeps the VLA backbone frozen and adds an external latent dynamics corrector. A Latent-space Vision Monitor (LVM) detects persistent mismatch between predicted and observed visual feature evolution; the system then truncates stale actions and invokes corrective replanning via Online Gradient Guidance (OGG).\n\n## Paper Figure\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"docs\u002Fassets\u002Fimages\u002Fmethod_overview.png\" alt=\"VLA-Corrector method overview\" width=\"92%\">\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\u003Cb>VLA-Corrector method overview.\u003C\u002Fb>\u003C\u002Fp>\n\n## Abstract\n\nAction-chunked VLA policies reduce policy-call frequency and preserve temporal coherence by executing several future actions before querying the policy again. This design can fail in contact-rich manipulation, where small perturbations, pose drift, or slippage may compound inside the open-loop blind spot.\n\nVLA-Corrector mitigates this issue with an event-triggered adaptive action horizon. During stable execution it preserves the efficiency of long chunks. When latent visual dynamics indicate persistent drift, it truncates the current queue and applies OGG only to the next recovery query. The trainable component is an external lightweight corrector, not the full VLA backbone.\n\n## Method\n\nThe paper organizes VLA-Corrector into four parts:\n\n1. **External latent dynamics corrector:** predicts short-horizon latent residuals from frozen VLA visual features and executed actions.\n2. **Latent-space Vision Monitor:** compares expected and observed latent visual evolution during action-chunk execution.\n3. **Event-triggered truncation:** discards remaining queued actions when persistent drift indicates that the chunk has become stale.\n4. **Online Gradient Guidance:** guides the single recovery replan immediately after an interrupt event.\n\nThe paper reports residual MLP correctors with approximately **38--42M parameters**, referred to as a lightweight ~40M MLP corrector.\n\n## Results\n\nThe following values summarize the paper's reported results. See the paper for complete protocols, task splits, and appendix tables.\n\n| Setting | Baseline | + VLA-Corrector | Reported change |\n| --- | ---: | ---: | ---: |\n| MetaWorld, PI0.5 avg. success | 48.70 | 64.35 | +15.65 |\n| MetaWorld, SmolVLA avg. success | 61.90 | 66.65 | +4.75 |\n| MetaWorld, X-VLA avg. success | 55.55 | 59.60 | +4.05 |\n| LIBERO, PI0.5 few-shot avg. success | 94.00 | 97.80 | +3.80 |\n| AgileX PiPER real-world avg. success | 55.6 | 73.3 | +17.7 |\n\nAdditional analysis in the paper reports that truncation alone improves MetaWorld average success from 48.70% to 60.35%, while truncation plus OGG reaches 64.35%. The paper also reports that 83.7% of truncations occur in manually labeled critical phases.\n\n## Installation\n\n```bash\nconda env create -f environment.yml\nconda activate lerobot\npython -m pip install -e . --no-build-isolation\n```\n\nFor PushT simulation smoke tests:\n\n```bash\npython -m pip install -e '.[pusht]' --no-build-isolation\n```\n\nAlternatively:\n\n```bash\npython -m pip install -r requirements.txt\n```\n\nThe exported environment name is `lerobot`. You can edit the `name:` field in `environment.yml` before creating the environment.\n\n## Data and Checkpoints\n\nThis repository does **not** include datasets, raw demonstration data, training outputs, Hugging Face pretrained weights, fine-tuned VLA checkpoints, trained corrector checkpoints, wandb logs, or caches. The project page includes only compressed silent real-robot clips for visualization.\n\nPrepare or specify these paths yourself:\n\n```text\n\u003CDATASET_DIR>             # Source LeRobot, MetaWorld, or LIBERO dataset\n\u003CEXTRACTED_CACHE_DIR>     # Extracted latent cache from siglip_dynamics.extract\n\u003CPOLICY_CHECKPOINT>       # Base or fine-tuned PI0.5, SmolVLA, or X-VLA policy checkpoint\n\u003CCORRECTOR_CHECKPOINT>    # Trained latent dynamics corrector checkpoint directory\n\u003COUTPUT_DIR>              # Local output directory, usually under outputs\u002F\n```\n\nKnown model names referenced by the code include:\n\n| Component | Hugging Face repository referenced in source | Source location |\n| --- | --- | --- |\n| PI0.5 base policy | [`lerobot\u002Fpi05_base`](https:\u002F\u002Fhuggingface.co\u002Flerobot\u002Fpi05_base) | `tests\u002Fpolicies\u002Fpi0_pi05\u002Ftest_pi05_original_vs_lerobot.py` |\n| SmolVLA base policy | [`lerobot\u002Fsmolvla_base`](https:\u002F\u002Fhuggingface.co\u002Flerobot\u002Fsmolvla_base) | `src\u002Flerobot\u002Fpolicies\u002Fsmolvla\u002Fmodeling_smolvla.py` |\n| X-VLA WidowX policy | [`lerobot\u002Fxvla-widowx`](https:\u002F\u002Fhuggingface.co\u002Flerobot\u002Fxvla-widowx) | `tests\u002Fpolicies\u002Fxvla\u002Ftest_xvla_original_vs_lerobot.py` |\n| SmolVLA VLM backbone | [`HuggingFaceTB\u002FSmolVLM2-500M-Video-Instruct`](https:\u002F\u002Fhuggingface.co\u002FHuggingFaceTB\u002FSmolVLM2-500M-Video-Instruct) | `src\u002Flerobot\u002Fpolicies\u002Fsmolvla\u002Fconfiguration_smolvla.py` |\n| PI0\u002FPI0.5 tokenizer backbone | [`google\u002Fpaligemma-3b-pt-224`](https:\u002F\u002Fhuggingface.co\u002Fgoogle\u002Fpaligemma-3b-pt-224) | `src\u002Flerobot\u002Fpolicies\u002Fpi05\u002Fprocessor_pi05.py` |\n| Fast action tokenizer | [`lerobot\u002Ffast-action-tokenizer`](https:\u002F\u002Fhuggingface.co\u002Flerobot\u002Ffast-action-tokenizer) | `src\u002Flerobot\u002Fscripts\u002Flerobot_train_tokenizer.py` |\n\nFine-tuned checkpoints are not included. Please specify your own checkpoint paths with `--policy.path` and `--safety_model_path`.\n\n## Corrector Training\n\nLatent extraction:\n\n```bash\npython -m siglip_dynamics.extract \\\n  --dataset-path \u003CDATASET_DIR> \\\n  --dataset-repo-id \u003CDATASET_REPO_ID> \\\n  --dataset-loader parquet \\\n  --dataset-format \u003Cmetaworld_or_libero> \\\n  --output-path \u003CEXTRACTED_CACHE_DIR> \\\n  --encoder-backend \u003Cpi05_or_smolvla_or_xvla> \\\n  --use-normalized-delta-action \\\n  --encoder-policy-path \u003CPOLICY_CHECKPOINT> \\\n  --encoder-local-files-only\n```\n\nCorrector training:\n\n```bash\ntorchrun --nproc_per_node=1 -m siglip_dynamics.train \\\n  --model-type mlp \\\n  --h-window 1 \\\n  --k-step-list 10 \\\n  --dataset-path \u003CEXTRACTED_CACHE_DIR> \\\n  --batch-size 512 \\\n  --epochs 30 \\\n  --train-loss-type cosine \\\n  --checkpoint-dir \u003CCORRECTOR_CHECKPOINT>\n```\n\n## Evaluation\n\nMain modified evaluation entry point:\n\n```bash\npython -m lerobot.scripts.lerobot_eval_modified_detection --help\n```\n\nPI0.5-style modified evaluation:\n\n```bash\nexport MUJOCO_GL=egl\nexport PYOPENGL_PLATFORM=egl\nexport EGL_PLATFORM=surfaceless\n\npython -m lerobot.scripts.lerobot_eval_modified_detection \\\n  --policy.path=\u003CPOLICY_CHECKPOINT> \\\n  --policy.device=cuda \\\n  --policy.n_action_steps=10 \\\n  --policy.chunk_size=50 \\\n  --policy.compile_model=false \\\n  --env.type=metaworld \\\n  --env.task=\u003CTASK_SPLIT> \\\n  --env.episode_length=300 \\\n  --eval.batch_size=1 \\\n  --eval.n_episodes=20 \\\n  --eval.use_async_envs=false \\\n  --env.max_parallel_tasks=1 \\\n  --seed=1000 \\\n  --safety_model_path=\u003CCORRECTOR_CHECKPOINT> \\\n  --safety_k=10 \\\n  --guidance_eta=1 \\\n  --guidance_apply_every=1 \\\n  --guidance_loss_objective=attract_delta_z_correction \\\n  --guidance_compare_baseline=true \\\n  --meltdown_cooldown_steps=10 \\\n  --output_dir=\u003COUTPUT_DIR> \\\n  --save_analysis=false \\\n  --save_raw_video=false \\\n  --save_summary_csv=true \\\n  --save_summary_json=true\n```\n\nSmolVLA and X-VLA use the same entry point with backbone-specific policy arguments. Full evaluation requires simulator dependencies, GPU resources, datasets, policy checkpoints, and trained corrector checkpoints.\n\n## Repository Structure\n\n```text\n.\n├── src\u002Flerobot\u002F                 # LeRobot-based codebase and modified VLA policies\n├── src\u002Fsiglip_dynamics\u002F         # Latent extraction and corrector training\n├── docs\u002F                        # English project page\n├── media\u002F                       # Non-Pages media materials\n├── examples\u002F\n├── tests\u002F\n├── environment.yml\n└── requirements.txt\n```\n\n## Citation\n\nThe paper is available on arXiv: [https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01804](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01804).\n\n```bibtex\n@article{pan2026vla,\n  title={VLA-Corrector: Lightweight Detect-and-Correct Inference for Adaptive Action Horizon},\n  author={Pan, Yi and Pan, Miao and Lu, Qi and Huang, Jiaming and Zhang, Man and Huang, Siteng and Li, Xin and Zhang, Jie and Shen, Yongliang and Zhang, Xuhong and others},\n  journal={arXiv preprint arXiv:2607.01804},\n  year={2026}\n}\n```\n\n## Acknowledgements\n\nThis repository builds on LeRobot and the Hugging Face ecosystem, and references VLA backbones and benchmarks including PI0.5, SmolVLA, X-VLA, MetaWorld, and LIBERO. Please also cite the corresponding upstream projects when using this code.\n","VLA-Corrector 是一个面向视觉-语言-动作（VLA）模型的轻量级检测与校正推理框架，用于动态调整固定动作步长带来的开环盲区问题。其核心通过冻结原始VLA主干网络，引入外部潜空间动力学校正模块：利用潜空间视觉监控器（LVM）持续比对预测与实际视觉特征演化，触发事件驱动的动作截断与在线梯度引导（OGG）重规划。该方案无需微调主干模型，兼顾实时性与鲁棒性，适用于机器人接触式操作、精细抓取等易受扰动影响的具身智能任务。","2026-07-11 02:30:49","CREATED_QUERY"]