[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-1997":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":14,"stars7d":16,"stars30d":17,"stars90d":15,"forks30d":15,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":20,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":28,"readmeContent":29,"aiSummary":30,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":31,"discoverSource":32},1997,"SIM1","InternRobotics\u002FSIM1","InternRobotics","Official implementation of \"SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds \"","https:\u002F\u002Finternrobotics.github.io\u002Fsim1.github.io\u002F",null,"Python",145,8,1,0,2,7,3,2.86,false,"main",[23,24,25,26,27],"deformable-manipulation","lerobot","robotics","simulation","vision-language-action-model","2026-06-12 02:00:35","\u003Cdiv id=\"top\" align=\"center\">\n\n# SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds\n\n\u003Cp align=\"center\">\n  \u003Cimg src=\"media\u002Fteaser.gif\" width=\"900px\" alt=\"SIM1 teaser\">\n\u003C\u002Fp>\n\n[![Demo](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDemo-SIM1-0366d6?style=flat&logo=googlechrome&logoColor=white)](https:\u002F\u002Fsim1-demo.intern-robotics.com\u002F) [![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-blue.svg)](LICENSE) [![arXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2604.08544-b31b1b.svg)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.08544) [![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject%20Page-SIM1-0366d6?style=flat&logo=githubpages&logoColor=white)](https:\u002F\u002Finternrobotics.github.io\u002Fsim1.github.io\u002F) [![Hugging Face · Assets](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗%20Sim1-Assets-yellow)](https:\u002F\u002Fhuggingface.co\u002FInternRobotics\u002FSim1_Assets) [![Hugging Face · Dataset](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗%20Sim1-Dataset-yellow)](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FInternRobotics\u002FSim1_Dataset)\n\n\u003C\u002Fdiv>\n\nA research project from [InternRobotics](https:\u002F\u002Fgithub.com\u002FInternRobotics).\n\n[YouTube 1](https:\u002F\u002Fyoutu.be\u002FtsPLa-1Lygw) · [YouTube 2](https:\u002F\u002Fyoutu.be\u002FLXStHGWHh18) · [YouTube 3](https:\u002F\u002Fyoutu.be\u002Fzesn7aK9sgQ)\n\n**Sim1** is a physics-aligned simulator and data stack for dual-arm cloth manipulation in simulation: teleoperation, diffusion-based data generation, replay, filtering, and optional photorealistic rendering, built on [Newton](https:\u002F\u002Fnewton-physics.github.io\u002Fnewton\u002F) and NVIDIA [Warp](https:\u002F\u002Fnvidia.github.io\u002Fwarp\u002F). This repository contains the full pipeline from interactive control and synthetic trajectory generation to rendering and LeRobot-style dataset export.\n\n---\n\n\n## Table of Contents\n\n1. [Installation](#installation)\n2. [Quick Start — Interactive Teleoperation](#quick-start--interactive-teleoperation)\n3. [Quick Start — Data Generation](#quick-start--data-generation)\n4. [Rendering Pipeline](#rendering-pipeline)\n5. [Data Conversion](#data-conversion)\n6. [Project Structure](#project-structure)\n7. [TODO List](#todo-list)\n8. [Citation](#citation)\n9. [License](#license)\n\n---\n\n## Installation\n\n### Prerequisites\n\nUse Python 3.11 with conda (environment name `sim1`) and CUDA toolkit >= 12.4 if you want GPU acceleration.\n\n\u003Cdetails>\n\u003Csummary>Reference: Newton Installation Guide\u003C\u002Fsummary>\n\n[Newton — manual setup with pip (virtual environment)](https:\u002F\u002Fnewton-physics.github.io\u002Fnewton\u002F0.2.2\u002Fguide\u002Finstallation.html#method-3-manual-setup-using-pip-in-a-virtual-environment)\n\n\u003C\u002Fdetails>\n\n---\n\n### Step 1 — Create the conda environment\n\n```bash\nconda create -n sim1 python=3.11 -y\nconda activate sim1\n```\n\n---\n\n### Step 2 — Clone the repository\n\nClone with submodules so `components\u002Frender\u002FMeisterRender` ([SIM1MeisterRender](https:\u002F\u002Fgithub.com\u002FInternRobotics\u002FSIM1MeisterRender), `main` branch) is checked out automatically:\n\n```bash\ngit clone --recurse-submodules https:\u002F\u002Fgithub.com\u002FInternRobotics\u002FSIM1.git sim1\ncd sim1\n```\n\n---\n\n### Step 3 — Install dependencies\n\nWith `sim1` active, from the repository root:\n\n```bash\nconda activate sim1\nbash setup.sh\n```\n\nAll Python dependencies (simulation, DataGen, asset download helpers, optional full render stack, and post-install checks) are installed by [`setup.sh`](setup.sh) only. No separate render dependency install is required. Open that file for the full list, optional environment variables (`SIM1_SKIP_RENDER`, `TORCH_INDEX_URL`), and the exact `pip` commands. For render usage notes (not dependency installation), see [`components\u002Frender\u002FREADME.md`](components\u002Frender\u002FREADME.md).\n\n---\n\n### Step 4 - Download assets (required before data generation)\n\nRun the download script once from the repository root. It downloads required assets into `assets\u002F`.\n\n```bash\n# From the repository root (after setup.sh)\nbash download_assets.sh\n```\n\n---\n\n### Verify installation\n\n```bash\nconda activate sim1\npython -c \"import newton; print('Newton version:', newton.__version__)\"\npython -c \"import warp as wp; print('Warp OK')\"\npython -c \"import torch, torchvision; print('torch', torch.__version__, 'cuda', torch.cuda.is_available())\"\n```\n\nNewton smoke test (MuJoCo humanoid + `nv_humanoid.xml`; needs a display for the GL viewer):\n\n```bash\ncd newton\npython newton\u002Fexamples\u002Frobot\u002Fexample_robot_humanoid.py\n```\n\nEquivalent: `python -m newton.examples robot_humanoid` (from the same `newton\u002F` directory). The MJCF asset is `newton\u002Fexamples\u002Fassets\u002Fnv_humanoid.xml`.\n\nAfter `bash download_assets.sh`, you should see at least:\n- `assets\u002Facone\u002Facone.urdf`\n- `assets\u002Fcloth\u002Fshort-shirt.usdc`\n- `assets\u002Fmodel\u002Fflow_ckpt_three.pth`\n- `assets\u002Fsim_teleoperated_npz\u002F` (reference NPZ subset)\n\n---\n\n## Quick Start — Interactive Teleoperation\n\nLaunch a real-time interactive simulation with keyboard-driven dual-arm control:\n\n```bash\npython apps\u002Fteleoperation_app.py --task lift_manip_shirt\n```\n\n### Keyboard Controls\n\nMatches the startup prompt in [`apps\u002Fteleoperation_app.py`](apps\u002Fteleoperation_app.py) (using the same shields style as the [Demo](https:\u002F\u002Fsim1-demo.intern-robotics.com\u002F) badge above):\n\n| Key | Action |\n|---|---|\n| `W\u002FS`, `A\u002FD`, `Q\u002FE`, `X`, `Z\u002FC` | Left gripper: front\u002Fback, left\u002Fright, down\u002Fup, toggle grip, pitch |\n| `U\u002FJ`, `H\u002FK`, `Y\u002FI`, `N`, `B\u002FM` | Right gripper: front\u002Fback, left\u002Fright, down\u002Fup, toggle grip, pitch |\n| Arrow keys, mouse left-drag, scroll | Camera: move, look, zoom |\n\n### WebSocket Streaming (Remote Teleoperation)\n\nFor headless or remote machines, enable WebSocket streaming:\n\n```bash\npython apps\u002Fteleoperation_app.py --task lift_manip_shirt --stream --host 0.0.0.0 --ws-port 8765 --http-port 8080\n```\n\nThen open `http:\u002F\u002F\u003Cserver-ip>:8080` in a browser to view and control the simulation remotely.\n\n---\n\n## Quick Start — Data Generation\n\n`run_pipeline.sh` runs the full data path in one shot: generate → Kalman smooth → replay (NPZ + USD) → filter. Generation uses the diffusion-policy path; robot URDF and cloth USD come from the default Hugging Face bundle location `assets\u002F` (see [Step 4 - Download assets](#step-4--download-assets)). The script prints `HF assets : …` on startup. For extra diversity at replay, add `--position-randomize`.\n\n### Generate data\n\nFrom the repository root (conda env, assets, and clone steps are in [Installation](#installation) and [Step 4 — Download assets](#step-4--download-assets)):\n\n```bash\nbash run_pipeline.sh --num 100\n```\n\nOptional: `bash run_pipeline.sh --num 100 --position-randomize`\n\nBy default, DataGen reads references from `assets\u002Fsim_teleoperated_npz` (downloaded from Hugging Face). You can override the reference source with `--ref_npz_folder`. Generated trajectories are written under `--data_folder` (`gen\u002F`, `gen\u002Fkf\u002F`). Replay outputs are saved under `replay\u002Fpipeline_output_XXXX\u002F` (the script prints the path). Use that session folder as `--root_dir` for [Rendering Pipeline](#rendering-pipeline).\n\n\u003Cdetails>\n\u003Csummary>All \u003Ccode>run_pipeline.sh\u003C\u002Fcode> options (advanced)\u003C\u002Fsummary>\n\n| Option | Description | Default |\n|---|---|---|\n| `--data_folder DIR` | Output data root (`gen\u002F`, `gen\u002Fkf\u002F`) | with `--ref_npz_folder`: `.\u002Fdataset\u002Fexample`; otherwise auto: `\u003CSIM1_ASSETS_ROOT>\u002Fsim_teleoperated_npz` (fallback `.\u002Fdataset\u002Fexample`) |\n| `--ref_npz_folder DIR` | Reference NPZ source for DataGen (`\u003CDIR>\u002Fnpz\u002F*.npz` or `\u003CDIR>\u002F*.npz`) | unset (use `--data_folder` layout) |\n| `--num N` | Trajectories to generate (DP pipeline only) | 10 |\n| `--workers N` | Parallel workers (smooth + filters) | 8 |\n| `--skip_smooth` \u002F `--skip_replay` \u002F `--skip_filter` | Skip a stage | off |\n| `--folder_name NAME` | `replay\u002F\u003CNAME>_XXXX\u002F` base name | `pipeline_output` |\n| `--position-randomize` | Random cloth pose at replay; joint filter also runs EE reachability (FK). Omit → joint filter uses `--no-check-ee` (jump \u002F mutation only) | off |\n| `--ref_usd PATH` | Reference USD for aligned cloth filter (with randomization); auto-picked if omitted | auto |\n| `--skip_asset_check` | Do not verify the HF bundle (`SIM1_ASSETS_ROOT`) before running | off |\n\n\u003C\u002Fdetails>\n\n### What runs internally\n\n```\nrun_pipeline.sh\n│\n├─ 1. Generate    →  apps\u002Fdatagen_app.py --use_dp --mode fine (DP only; fixed in script)\n│                     → \u003Cdata_folder>\u002Fgen\u002F*.npz\n├─ 2. Smooth      →  scripts\u002Fsmooth_trajectory_multi_thread.py (Kalman; fixed variances in script)\n│                     → \u003Cdata_folder>\u002Fgen\u002Fkf\u002F*.npz\n├─ 3. Replay      →  apps\u002Freplay_app.py [--position-randomize]\n│                     → replay\u002F\u003Cfolder_name>_NNNN\u002F{npz,usd}\u002F\n└─ 4. Filter      →  filter_joint_unreachable.py (joint jump + first-5 mutation; + EE FK if --position-randomize)\n                     → filter_cloth_quality.py (aligned + --ref-usd if randomize, else direct)\n```\n\n### Session layout\n\n```\nreplay\u002F\n└── pipeline_output_0001\u002F\n    ├── npz\u002F\n    ├── usd\u002F\n    ├── npz_bad_cloth\u002F\n    ├── usd_bad_cloth\u002F\n    ├── npz_unreachable\u002F          # joint \u002F EE rejects (see filter_joint_unreachable.py logs)\n    └── cloth_filter_summary.txt\n```\n\n\u003Cdetails>\n\u003Csummary>Manual step-by-step (only if you are not using \u003Ccode>run_pipeline.sh\u003C\u002Fcode>)\u003C\u002Fsummary>\n\n1. Generate: `python apps\u002Fdatagen_app.py --data_folder assets\u002Fsim_teleoperated_npz --num 100 --use_dp --mode fine`  \n2. Smooth: `python scripts\u002Fsmooth_trajectory_multi_thread.py assets\u002Fsim_teleoperated_npz\u002Fgen assets\u002Fsim_teleoperated_npz\u002Fgen\u002Fkf --method kalman --workers 8`  \n3. Replay: `python apps\u002Freplay_app.py assets\u002Fsim_teleoperated_npz\u002Fgen\u002Fkf --folder_name my_replay`  \nOptional cloth position randomization at replay: add `--position-randomize` (then use the matching manual filters as in Step 4 above).  \n4a. Joint \u002F EE filter: `python scripts\u002Ffilter_joint_unreachable.py .\u002Freplay\u002Fmy_replay_0001\u002Fnpz --usd-dir .\u002Freplay\u002Fmy_replay_0001\u002Fusd --workers 8` (add `--no-check-ee` to skip EE FK; joint checks always run)  \n4b. Cloth quality: `python scripts\u002Ffilter_cloth_quality.py .\u002Freplay\u002Fmy_replay_0001` (add `--ref-usd ...` if you used randomization)\n\n\u003C\u002Fdetails>\n\n---\n\n## Rendering Pipeline\n\nConvert simulation USD output to photorealistic data: `main.py` runs Steps 1–3 by default (USD → blend → cameras → `blend_out\u002F`). Step 4 (MeisterRender path tracing + LMDB) writes under `out_updated\u002F\u003Crecord_id>\u002F`; run it via `batch_step4.sh`, or inline with `main.py --step4`.\n\nMeisterRender lives in the git submodule `components\u002Frender\u002FMeisterRender` ([InternRobotics\u002FSIM1MeisterRender](https:\u002F\u002Fgithub.com\u002FInternRobotics\u002FSIM1MeisterRender), `main`). Use `git clone --recurse-submodules` in [Step 2](#step-2--clone-the-repository) so it is checked out automatically.\n\nEnvironment: use the same `sim1` env; the render stack is installed by `setup.sh` unless you set `SIM1_SKIP_RENDER=1` (see comments in `setup.sh`). For render usage and step notes, see [`components\u002Frender\u002FREADME.md`](components\u002Frender\u002FREADME.md).\n\n```bash\nconda activate sim1\n\n# One-click render on latest replay\u002Fpipeline_output_XXXX\nbash components\u002Frender\u002Frun_latest.sh\n\n# Optional: use another replay prefix\n# bash components\u002Frender\u002Frun_latest.sh --session-prefix my_run\n```\n\n### Asset Configuration\n\nRendering resolves the HF bundle via `SIM1_ASSETS_ROOT` (default `\u003Crepo>\u002Fassets\u002F`). HDRI \u002F table \u002F cloth glTF roots default to `assets\u002Frandom\u002F{bg,table,mat}\u002F` inside that bundle (`scripts\u002Fsim1_asset_paths.py`); no extra `export` is required for the usual layout.\n\n---\n\n## Data Conversion\n\nAfter Step 4 rendering, trajectories are stored under `replay\u002F\u003Csession>\u002Fout_updated\u002F\u003Crecord_id>\u002F` as LMDB + `meta_info.pkl`. To convert them into a LeRobot v2 dataset for training, use `components\u002Flmdb2lerobot\u002F`.\n\nOne-time environment (separate conda env `lerobot`, Python 3.12 — see full docs for details):\n\n```bash\nbash components\u002Flmdb2lerobot\u002Fsetup_conda_lerobot.sh\nconda activate lerobot\n```\n\nSingle session → LeRobot dataset:\n\n```bash\n# Auto-detect latest replay\u002Fpipeline_output_XXXX, then:\n#   src = \u003Clatest>\u002Fout_updated\n#   out = \u003Clatest>\u002Flerobot_dataset\nbash components\u002Flmdb2lerobot\u002Frun_local.sh\n\n# Optional: explicit paths still supported\n# bash components\u002Flmdb2lerobot\u002Frun_local.sh \\\n#   --src .\u002Freplay\u002Fmy_session\u002Fout_updated \\\n#   --out .\u002Freplay\u002Fmy_session\u002Flerobot_dataset\n```\n\nThis runs LMDB→LeRobot, sim2real, then removes near-static frames by default (`--keep-static-frames` to skip).\n\nBatch \u002F multi-GPU (optional): `components\u002Flmdb2lerobot\u002Frun_batch.sh` — see [`components\u002Flmdb2lerobot\u002FREADME.md`](components\u002Flmdb2lerobot\u002FREADME.md).\n\n---\n\n## Project Structure\n\n```\nsim1\u002F\n├── setup.sh                    # Dependency installation (setup.sh)\n├── download_assets.sh          # Hugging Face -> assets\u002F (Sim1_Assets + Sim1_Dataset\u002Fsim_teleoperated_npz only)\n├── run_pipeline.sh             # Data generation pipeline (generate→smooth→replay→filter)\n├── apps\u002F\n│   ├── teleoperation_app.py    # Interactive teleoperation entry point\n│   ├── datagen_app.py          # SIM1-DataGen entry (diffusion-policy mode)\n│   ├── datagen_fine_app.py     # Optional fine-grained DataGen entry\n│   └── replay_app.py           # Trajectory replay (headless)\n├── replay_batch.sh             # Batch replay script\n│\n├── newton\u002F                     # Newton physics engine (local install)\n│   ├── pyproject.toml\n│   └── newton\u002F                 # Newton source code\n│\n├── assets\u002F                     # Robot URDFs, meshes, render assets\n├── configs\u002F                    # Task configuration files\n├── envs\u002F                       # Simulation environments\n├── tasks\u002F                      # Task definitions (cloth manipulation)\n├── stream\u002F                     # WebSocket streaming server + web UI\n│\n├── components\u002F\n│   ├── datagen\u002F                # SIM1-DataGen core (splitter, selector, diffusion)\n│   │   ├── datagen_core.py     # DataGenerator class\n│   │   ├── splitter.py         # Trajectory splitter\n│   │   ├── selector.py         # Segment selector\n│   │   ├── traj_df\u002F            # Diffusion model for trajectory generation\n│   │   └── configs\u002F            # Task split configurations\n│   ├── function\u002F               # Utility functions (FK, IK, video, analysis)\n│   ├── randomization\u002F          # Environment randomization\n│   ├── recorder\u002F               # Dual-arm data recorder\n│   ├── render\u002F                 # USD → Blender → MeisterRender (git submodule) pipeline\n│   └── lmdb2lerobot\u002F           # LMDB → LeRobot v2 (+ sim2real + remove_static_frames by default)\n│\n├── scripts\u002F                    # Post-processing scripts\n│   ├── smooth_trajectory_multi_thread.py   # Kalman smooth (used by run_pipeline.sh)\n│   ├── filter_joint_unreachable.py         # Joint jump + optional EE reachability (see --no-check-ee)\n│   ├── filter_cloth_quality.py             # Cloth-quality filter (used by run_pipeline.sh)\n│   └── convert_ee_quat.py                  # EE pose conversion (used by datagen)\n│\n├── module_train\u002F               # Training modules\n│   ├── trajectory_discriminator\u002F\n│   └── trajectory_generator\u002F\n│\n└── dataset\u002F                    # Example datasets (npz, segments, etc.)\n```\n\n---\n\n## TODO List\n\n- [x] Simulation assets — Robot URDFs, cloth meshes, render assets.\n- [x] Public datasets — Open-sourced a subset of trajectories and rendered data.\n- [x] Data generation pipeline — Supports one-command generate → smooth → replay → filter.\n- [x] Training utilities — Includes policy and trajectory training modules.\n- [ ] Open-source 10,000 trajectories.\n- [ ] Upgrade to latest Newton — Bump bundled `newton\u002F` to upstream; adapt API changes in envs\u002Ftasks\u002Fcomponents.\n- [ ] Integrate libuipc solver — Optional libuipc cloth\u002Fdeformable backend for richer contact and friction.\n\n---\n\n## Citation\n\nIf you use Sim1 (code, assets, or datasets) in research, please cite the paper below.\n\n```bibtex\n@misc{zhou2026sim1physicsalignedsimulatorzeroshot,\n      title={SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds}, \n      author={Yunsong Zhou and Hangxu Liu and Xuekun Jiang and Xing Shen and Yuanzhen Zhou and Hui Wang and Baole Fang and Yang Tian and Mulin Yu and Qiaojun Yu and Li Ma and Hengjie Li and Hanqing Wang and Jia Zeng and Jiangmiao Pang},\n      year={2026},\n      eprint={2604.08544},\n      archivePrefix={arXiv},\n      primaryClass={cs.RO},\n      url={https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.08544}, \n}\n```\n\n---\n\n## License\n\nUnless otherwise noted, this repository is released under [Apache License 2.0](https:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0), while language data is released under [CC BY-NC-SA 4.0](https:\u002F\u002Fcreativecommons.org\u002Flicenses\u002Fby-nc-sa\u002F4.0\u002F). [Newton](https:\u002F\u002Fgithub.com\u002Fnewton-physics\u002Fnewton) and other third-party components follow their own licenses; see, for example, [newton\u002FLICENSE.md](newton\u002FLICENSE.md).\n","SIM1是一个面向可变形物体（如布料）的双臂操作物理对齐模拟器。它集成了遥操作、基于扩散的数据生成、重放、过滤以及可选的照片级渲染等功能，依托于Newton物理引擎和NVIDIA Warp技术构建。项目支持从交互式控制到合成轨迹生成，再到渲染及LeRobot风格数据集导出的全流程。适用于需要在虚拟环境中进行高精度软体物体操控的研究与开发场景，特别是在机器人学习、视觉-语言-动作模型训练等方面具有广泛应用潜力。","2026-06-11 02:47:17","CREATED_QUERY"]