[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-93248":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":9,"rankLanguage":9,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":15,"starSnapshotCount":15,"syncStatus":14,"lastSyncTime":25,"discoverSource":26},93248,"ardy","nv-tlabs\u002Fardy","nv-tlabs","Official implementation of ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation (SIGGRAPH 2026).",null,"Python",152,20,1,2,0,33,47.27,"Apache License 2.0",false,"main",[],"2026-07-22 04:02:08","# ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Fsil\u002Fprojects\u002Fardy\u002F\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-blue\" alt=\"Project Page\">\u003C\u002Fa>\n  \u003Ca href=\"LICENSE\">\u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-76B900.svg\" alt=\"License\">\u003C\u002Fa>\n  \u003Cimg src=\".\u002Fassets\u002Fbanner.png\" alt=\"Banner\" width=\"100%\">\n\u003C\u002Fp>\n\nARDY is an autoregressive diffusion model designed for interactive motion generation, supporting online text prompting and flexible long-horizon kinematic constraints (root paths\u002Fwaypoints, full-body keyframes, and sparse joint positions\u002Frotations) with real-time responsiveness.\n\nThis repo provides code, checkpoints, and demos to work with the pre-trained ARDY models introduced in the [SIGGRAPH paper](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Fsil\u002Fprojects\u002Fardy\u002F).\n\n## Setup\n> This repo has mainly been tested on Ubuntu Linux 22.04 with RTX 4090, nvidia-driver-575, and Python 3.11.\n\nIn a fresh Python 3.10+ environment (conda or venv), first install a build of **PyTorch** (>= 2.4) that matches your machine's CUDA, then install ARDY in editable mode with all optional features:\n\n```bash\nconda create -n ardy python=3.11 -y\nconda activate ardy\n# Install PyTorch for your CUDA version first — see https:\u002F\u002Fpytorch.org\u002Fget-started\u002Flocally\u002F\n# For example:\npip install torch torchvision --index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu126\npip install -e \".[all]\"\n```\n\n\u003Cdetails>\n\u003Csummary>Install details: why PyTorch first, build requirements, and partial installs\u003C\u002Fsummary>\n\n- **Why install PyTorch first:** ARDY only requires `torch>=2.4`, so installing PyTorch yourself first lets you pick the CUDA build that matches your GPU\u002Fdriver. The ARDY install then keeps the compatible PyTorch you already have rather than pulling a default wheel.\n- **Build requirements:** the install builds the bundled motion-correction C++ extension, which requires CMake >= 3.15 and a C++17 compiler (`sudo apt install cmake build-essential` on Ubuntu).\n- **Partial installs:** the optional dependency groups can also be installed individually:\n  - `pip install -e .` — core model inference only\n  - `pip install -e \".[demo]\"` — adds the interactive demo (viser, gradio)\n  - `pip install -e \".[trt]\"` — adds TensorRT acceleration\n- **TensorRT requirements:** the `[trt]` extra requires an NVIDIA driver >= 525 (CUDA 12-capable — the CUDA runtime itself is bundled via pip) and access to `pypi.nvidia.com` during install. On setups that don't meet these requirements, install `.[demo]` instead and select a non-TensorRT acceleration mode in the demo.\n\n\u003C\u002Fdetails>\n\n### Set up Hugging Face token for text encoder\n\nThe text encoder relies on the gated [meta-llama\u002FMeta-Llama-3-8B-Instruct](https:\u002F\u002Fhuggingface.co\u002Fmeta-llama\u002FMeta-Llama-3-8B-Instruct) model, which requires:\n\n- Your Hugging Face account has been granted access to the model page.\n- You provide a Hugging Face token at runtime.\n\nAfter you receive access to the Llama repo, create an access token in [Hugging Face token settings](https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Ftokens), then either log in on the command line:\n\n```bash\nhf auth login\n```\n\nor paste the token in `~\u002F.cache\u002Fhuggingface\u002Ftoken`. If you do not have `hf` installed, install it first:\n\n```bash\npip install --upgrade huggingface_hub\n```\n\n### Checkpoints\n\nARDY checkpoints are available trained on various skeletons with differing FPS and prediction horizon length.\n\n> Note: models will be downloaded automatically when used in the demo, so there is no need to download them manually\n\n| Model | Skeleton | Training Data | FPS | Horizon | Release Date | Hugging Face | License |\n|:-------|:-------------|:------:|:------:|:------:|:------:|:-------------:|:-------------:|\n| **ARDY-Core-RP-20FPS-Horizon40** | Core | [Bones Rigplay 1](https:\u002F\u002Fbones.studio\u002Fdatasets#rp01) | 20 | 40 | July 10, 2026 | [Link](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FARDY-Core-RP-20FPS-Horizon40) | [NVIDIA Open Model](https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fagreements\u002Fenterprise-software\u002Fnvidia-open-model-agreement\u002F) |\n| **ARDY-Core-RP-20FPS-Horizon8** | Core | [Bones Rigplay 1](https:\u002F\u002Fbones.studio\u002Fdatasets#rp01) | 20 | 8 | July 10, 2026 | [Link](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FARDY-Core-RP-20FPS-Horizon8) | [NVIDIA Open Model](https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fagreements\u002Fenterprise-software\u002Fnvidia-open-model-agreement\u002F) |\n| **ARDY-G1-RP-25FPS-Horizon52** | Unitree G1 | [Bones Rigplay 1](https:\u002F\u002Fbones.studio\u002Fdatasets#rp01) | 25 | 52 | July 10, 2026 | [Link](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FARDY-G1-RP-25FPS-Horizon52) | [NVIDIA Open Model](https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fagreements\u002Fenterprise-software\u002Fnvidia-open-model-agreement\u002F) |\n| **ARDY-G1-RP-25FPS-Horizon8** | Unitree G1 | [Bones Rigplay 1](https:\u002F\u002Fbones.studio\u002Fdatasets#rp01) | 25 | 8 | July 10, 2026 | [Link](https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FARDY-G1-RP-25FPS-Horizon8) | [NVIDIA Open Model](https:\u002F\u002Fwww.nvidia.com\u002Fen-us\u002Fagreements\u002Fenterprise-software\u002Fnvidia-open-model-agreement\u002F) |\n\n\n**Coming soon!** We are working to train a version of ARDY on Rigplay 1 with the [SOMA body model](https:\u002F\u002Fgithub.com\u002FNVlabs\u002FSOMA-X) skeleton.\n\n### [Optional] Dataset\n\nDownloading data is only required for running the kinematically constrained generation demos.\n\n**Bones SEED motion data**:\nThe kinematically constrained generation demo samples constraints from motion sequences in the [Bones SEED](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fbones-studio\u002Fseed) dataset. The motion data are provided in CSV format for G1. Corresponding text descriptions are retrieved from the metadata CSV during sampling.\n\nPlease download the Bones SEED dataset and put them under the `datasets\u002Fbones-seed\u002F` directory in the root of the repo. The directory structure should be as follows:\n```\ndatasets\u002Fbones-seed\u002F\n  g1\u002Fcsv\u002F\n  metadata\u002F\n    seed_metadata_v004.csv\n```\n\n---\n\n## Interactive Demo\n\nWe provide an interactive demo that enables real-time humanoid character control through a combination of streaming text prompts and interactive spatial constraints. Below are some features of the interactive demo:\n\n- Online motion generation visualized in web browser viewport\n- Streaming text-to-motion\n- Kinematically constrained motion generation (full body keyframes, root trajectory and waypoints, end-effector joints)\n- Mouse and keyboard-based locomotion controls\n\n### Launch Interactive Demo\n\n```bash\npython scripts\u002Frun_demo.py\n```\n\nIf you plan to be frequently re-launching the demo, it can be helpful to launch the text encoder API service in the background. This way, the demo will not need to instantiate the text encoder every time it is launched. In a separate terminal, run:\n```bash\npython scripts\u002Frun_text_encoder_server.py\n```\n\n### Quick Start\n- **Open the UI**: In your browser, go to `http:\u002F\u002Flocalhost:2333`. You can use Left-drag to rotate, Right-drag to pan, and Scroll to zoom to control the camera with mouse.\n- **Load a model**: Choose a checkpoint from the **Model Directory** dropdown in the **Model** tab and click **Load Model**. The first load may take a few minutes if TensorRT compilation is enabled. A default checkpoint is loaded if no checkpoint is manually selected.\n- **Start playback**: Press `Space` key or click **Play** button to start\u002Fstop playing the motion.\n- **Text prompts**: Either click a preset in **Prompt List**, or type your own in **Prompt Text** and click **Update Text Prompt**.\n- **Mouse-based locomotion control**: Press `p` to enable waypoint mode (or click the **Enable Waypoint Mode** checkbox in the **Waypoint** folder under **Generate** tab), then click in the viewport to place waypoints for locomotion control.\n- **Keyboard-based velocity control**: Press `t` key to enable target-velocity control. Use arrow keys to steer: left\u002Fright rotate the direction; up\u002Fdown increase or decrease the speed.\n- **Kinematic constraints**: Press `z` key to sample kinematic constraints from a dataset sequence specified in **Motion file path** (in the **Constraints** folder under **Generate** tab). Use the constraint-type checkboxes to control which categories are sampled.\n- **Export\u002Fload sessions**: Use the **IO** tab for basic export\u002Fload of constraints and motion.\n\nMore detailed information about the interactive demo is available in the [GUI Reference](#gui-reference) section.\n\n### GUI Reference\n\n\u003Cdetails>\n\u003Csummary>Click to expand for detailed GUI usage\u003C\u002Fsummary>\n\n#### Hotkeys\n\n| Key | Action |\n|-----|--------|\n| Space | Toggle play\u002Fpause |\n| j | Previous frame |\n| k | Next frame |\n| r | Reset automatic camera to follow current frame |\n| t | Toggle target velocity control and arrow key overlay |\n| p | Toggle waypoint control mode |\n| z | Sample kinematic constraints from specified motion file |\n| Arrow keys | Control target velocity direction\u002Fspeed (when target velocity control is enabled) |\n| h | Show \u002F hide keyboard shortcut overlay |\n\n#### Tab: Playback\n\n- **Native FPS** - The motion FPS of the loaded model.\n- **Enable Auto-Replan** - If disabled, motion will not be automatically generated and stop at the last generated frame.\n\n#### Tab: Text\n\n- **Prompt List** - Preset prompt buttons containing example prompts. Clicking directly updates the active text prompt and embedding. You can select the text prompts on the fly and can also input your own text prompt using the **Prompt Text** field.\n- **Prompt Text** - Enter a text prompt here; it will not take effect until the Update button is clicked.\n- **Update Text Prompt** - Click to update the active text prompt and embedding.\n\n#### Tab: Generate\n\n- **Restart** - Clear all motion and restart generation from the first frame.\n- **Restart From Now** - Clear motion after current frame and continue generation from the current frame.\n- **Clear All Constraints** - Clear all kinematic constraints.\n- **Initial Body Transform** - A gizmo to control initial root position and orientation.\n- **Constraints**\n  - Motion file path: path to a CSV (G1) motion file from `datasets\u002Fbones-seed\u002F`. The kinematic constraints will be sampled from this motion file.\n  - Random Motion File button: pick a random motion file from `datasets\u002Fbones-seed\u002F` matching the loaded skeleton and populate the Motion file path field.\n  - Crop to 10s: if enabled, randomly crop the loaded motion to 10 seconds if the motion is longer than 10 seconds.\n  - Constraints types checkboxes [full body, hands, feet, hands+feet, 2D waypoints, 2D trajectory]: if enabled, sample constraints for the selected category.\n  - Max keyframe number: maximum number of keyframe constraints allowed for sampling\n  - Continue from current frame: if enabled, sample new constraints without restarting; otherwise clear motion history before generation.\n  - Sample Constraints button (hotkey: `z`): If pressed, the kinematic constraints will be sampled from the specified motion file. Note that the kinematic constraints sampling only supports the G1 skeleton which is included in the Bones Seed dataset.\n- **Waypoint** (mouse-based locomotion control)\n  - Waypoint mode (hotkey: `p`): toggle mouse-click-based waypoint control mode\n  - Dense root: if enabled, use interpolation to generate dense trajectory constraints from waypoints\n  - Waypoint interval: time interval between current frame and the waypoint keyframe, measured in number of frames.\n- **Target Velocity** (keyboard-based velocity control; works better with the **Over-the-shoulder** auto camera enabled)\n  - Enable checkbox: enable target velocity control.\n  - Target root velocity: target root velocity in XZ plane.\n  - Use target heading: if enabled, use root position + orientation as constraints; if disabled, only use root position as constraints.\n- **Post Process**\n  - Enable Post-Processing: apply optional motion correction to reduce foot skating and improve constraint following (slower; disabled by default).\n  - Root Margin: margin for root position correction.\n  - Contact Threshold: threshold for foot contact detection.\n\n#### Tab: Visualize\n\n- **Show Foot Contacts** - Color the foot joints purple on frames where the model predicts the foot to be in contact with the ground. Only visible when **Show Skeleton** is enabled.\n- **Show Hand+Foot Orientations** - Overlay rotation-axis gizmos on the hand\u002Fwrist and foot joints, for both the generated motion and any end-effector keyframes in the timeline. Useful for verifying that orientation constraints are being followed.\n- **Show Reference Motion** - Display the reference motion specified in **Motion file path** as a red semi-transparent mesh character\n- **Hide Distant Constraints** - If enabled, constraints outside the future cropping range specified in **Future Crop Length** are hidden\n- **Auto Camera** - A dropdown list to select from three auto camera types that updates according to current frame's human body pose.\n\n#### Tab: Model\n\n- **Acceleration Mode** - Choose TensorRT, torch.compile, or none\n- **Text Encoder** - Set device + precision for the LLM2Vec text encoder. Options: `cuda \u002F bfloat16` (default, fastest, ~14 GB VRAM), `cuda \u002F float32` (VRAM demanding), `cpu \u002F bfloat16`, `cpu \u002F float32` (slower prompt encoding but requires less VRAM).\n- **TRT Export Settings** - Min\u002Fopt\u002Fmax tokens and export button. These settings are used to configure the exportation of the TensorRT engines for the denoiser and decoder and may slightly affect the generation speed.\n- **History Crop Length** - Maximum number of history frames input to the model (min: patch size, max: FPS x 10s - generation horizon, step size: patch size, default: min). Smaller history crop length facilitates faster adaptation to new prompts and kinematic constraints, while larger history crop length allows for longer context which helps complex semantic motion generation and smoother transitions.\n- **Future Crop Length** - Number of future frames beyond generation horizon for constraint input (min: 0, max: FPS x 10s - generation horizon, step size: patch size, default: max)\n- **Replan Buffer Size** - How many frames of replan buffer to use. Ensure that the generation time is less than the (replan buffer + 1) frames playback time.\n- **Replan Trigger Threshold** - When remaining generated-but-unplayed frames \u003C= this threshold, trigger replan.\n\n#### Tab: IO\n\n- **Session Export\u002FLoad** - Export\u002Fload the full session (motion and\u002For constraints) to\u002Ffrom a file.\n- **Root Constraints** - Load\u002Fsave root constraints to a file.\n- **Scene** - Load a scene file and adjust mesh transform\u002Ftranslation.\n- **Capture Viewport Image** - Capture the current viewport image and save it to a file.\n\n\u003C\u002Fdetails>\n\n### Model Integration in Demo\n\nThe interactive demo is a good reference for how ARDY can be used autoregressively for realtime applications. The key entry points are:\n\n- **Loading:** [`load_model()`](ardy\u002Fmodel\u002Fload_model.py) resolves a checkpoint, builds the model, and attaches the LLM2Vec text encoder (built separately via `load_text_encoder()` so it can be reused across models). Models are referenced by nickname — a skeleton name (`\"core\"`, `\"g1\"`, `\"soma\"`, resolving to that skeleton's default horizon) or skeleton+horizon (`\"core8\"`, `\"g152\"`, `\"soma60\"`) — or by full name (`\"ARDY-Core-RP-20FPS-Horizon40\"`); see [`ardy\u002Fmodel\u002Fregistry.py`](ardy\u002Fmodel\u002Fregistry.py). The demo wires this up in [`ModelLoadingMixin.load_model`](scripts\u002Finteractive_demo\u002Floading.py).\n- **Generation:** the model runs autoregressively via [`Ardy.autoregressive_step()`](ardy\u002Fmodel\u002Fardy_model.py) — text embeddings and optional kinematic constraints (`motion_mask`\u002F`observed_motion`) in, motion frames out. See it called each step in [`GenerationMixin._generate_step`](scripts\u002Finteractive_demo\u002Fgeneration.py), which shows how to encode the text prompt, assemble history + constraint masks, and decode the output back to joint positions with `motion_rep.inverse(...)`.\n\n---\n\n## Command-Line Generation\n\nFor non-interactive \u002F batch generation, [`scripts\u002Fgenerate.py`](scripts\u002Fgenerate.py) generates motion from a text prompt and writes it to disk (`.npz`; plus a MuJoCo-qpos `.csv` for G1). Generated files are written to the `outputs\u002F` folder by default.\n\n```bash\n# Simplest: 5s of motion with the default model, written to outputs\u002Foutput.npz\npython scripts\u002Fgenerate.py \"A person walks in a circle.\"\n\n# Choose a model + length, and fix the seed for reproducible output (writes outputs\u002Fjump.npz)\npython scripts\u002Fgenerate.py \"A person jumps.\" --model core --duration 8.0 --seed 0 --output jump\n\n# Multiple samples (writes a folder outputs\u002Fwaves\u002F with waves_00.npz, waves_01.npz, ...)\npython scripts\u002Fgenerate.py \"A person waves.\" --model g1 --num_samples 4 --output waves\n```\n\nEach `.npz` contains the generated motion — `posed_joints` (world-space joint positions, `[T, J, 3]`), local\u002Fglobal joint rotations, root positions, and foot contacts — plus the `fps` and the prompt `text`.\n\nTo play back a generated motion in the browser (same viser viewport as the interactive demo), use [`scripts\u002Fvisualize.py`](scripts\u002Fvisualize.py):\n\n```bash\npython scripts\u002Fvisualize.py outputs\u002Foutput.npz   # single file\npython scripts\u002Fvisualize.py outputs\u002Fwaves        # folder written with --num_samples > 1\n```\n\nthen open `http:\u002F\u002Flocalhost:2334`. The skeleton (core \u002F g1 \u002F soma) is detected from the file; the viewer offers play\u002Fpause, a frame slider, mesh\u002Fskeleton toggles, and a sample dropdown for multi-sample folders.\n\nMost useful flags:\n\n| Flag | Meaning |\n|------|---------|\n| `--model` | Model nickname `core`\u002F`g1` (optionally with horizon, e.g. `core8`) or full folder name. Default: `core`. |\n| `--duration` | Length in seconds (default 5.0). |\n| `--num_samples` | Number of samples; when >1, writes a numbered folder. |\n| `--seed` | Fix the random seed for reproducible output. |\n| `--output` | Output stem (single sample → one file; multiple → a folder). Bare names go under `outputs\u002F`; pass a path (e.g. `results\u002Fjump`) to write elsewhere. |\n| `--history_frames` | History visible to each autoregressive step. Default: the longest history fitting the model's trained 10s window — keep it unless you want faster prompt adaptation. |\n| `--constraints` | Path to a saved kinematic-constraint list. |\n| `--no-postprocess` | Don't apply motion post-processing to reduce foot skating and hit constraints. |\n\n> Tip: if you generate repeatedly, start the standalone text-encoder service in the background first (`python scripts\u002Frun_text_encoder_server.py`) so each run connects to it instead of loading the LLM2Vec model in-process every time.\n\n---\n\n## Related Humanoid Work at NVIDIA\nARDY is closely related to several works from NVIDIA on humanoid motion and control:\n* [Kimodo](https:\u002F\u002Fgithub.com\u002Fnv-tlabs\u002Fkimodo) - human motion generation model specializing in controllable _offline_ motion authoring.\n* [MotionBricks](https:\u002F\u002Fnvlabs.github.io\u002Fmotionbricks\u002F) - real-time motion generation framework that specializes in fast and robust motion in-betweening.\n* [GEAR SONIC](https:\u002F\u002Fgithub.com\u002FNVlabs\u002FGR00T-WholeBodyControl) - humanoid behavior foundation model for physical robots. ARDY can be used as a planner to generate motions given to SONIC.\n\n---\n\n## Citation\n\nIf you use this code in your research, please cite:\n\n```bibtex\n@article{zhao2026ardy,\n  title   = {ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation},\n  author  = {Zhao, Kaifeng and Petrovich, Mathis and Zhang, Haotian and Wang, Tingwu and Tang, Siyu and Rempe, Davis},\n  journal = {ACM Transactions on Graphics (TOG)},\n  year    = {2026},\n  volume  = {45},\n  number  = {4},\n  articleno = {86},\n  doi     = {10.1145\u002F3811284}\n}\n```\n\n## License\n\nThis codebase is licensed under [Apache-2.0](LICENSE). Note that model checkpoints and data are licensed separately as indicated on the HuggingFace download pages.\n\nThis project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.\n\n## Acknowledgments\n\nThis project builds upon excellent open-source projects:\n- [Viser](https:\u002F\u002Fgithub.com\u002Fnerfstudio-project\u002Fviser) for interactive 3D motion generation demo\n- [LLM2Vec](https:\u002F\u002Fgithub.com\u002FMcGill-NLP\u002Fllm2vec) for text encoding\n\n## Contact\n\nFor questions or issues, please open an issue on this repository or reach out directly to the authors.\n\n---\n","ARDY 是一个面向交互式人体运动生成的自回归扩散模型，支持在线文本提示与多类型运动约束（如根轨迹、全身关键帧、稀疏关节点位\u002F旋转）的实时响应。其核心技术融合了扩散建模与混合表征（隐空间+显式运动参数），通过自回归机制实现长时序运动合成，并提供 TensorRT 加速与交互式可视化（Gradio\u002FViser）支持。适用于虚拟人驱动、游戏动画编辑、人机交互仿真等需要低延迟、高可控性的三维运动生成场景。","2026-07-14 02:30:05","CREATED_QUERY"]