[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72557":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":31,"readmeContent":32,"aiSummary":33,"trendingCount":16,"starSnapshotCount":16,"syncStatus":34,"lastSyncTime":35,"discoverSource":36},72557,"3dgrut","nv-tlabs\u002F3dgrut","nv-tlabs","Ray tracing and hybrid rasterization of Gaussian particles","",null,"Python",2248,257,36,58,0,7,25,75,21,29.23,"Apache License 2.0",false,"main",true,[27,28,29,30],"3d-reconstruction","computer-vision","gaussian-splatting","ray-tracing","2026-06-12 02:03:04","\u003Cp align=\"center\">\n  \u003Cimg height=\"100\" src=\"assets\u002F3dgrut_logo.png\">\n\u003C\u002Fp>\n\n---\n\u003Cp align=\"center\">\n  \u003Cimg width=\"100%\" src=\"assets\u002Fnvidia-hq-playground.gif\">\n\u003C\u002Fp>\n\nThis repository provides the official implementations of **3D Gaussian Ray Tracing (3DGRT)** and **3D Gaussian Unscented Transform (3DGUT)**. Unlike traditional methods that rely on splatting, 3DGRT performs ray tracing of volumetric Gaussian particles instead. This enables support for distorted cameras with complex, time-dependent effects such as rolling shutters, while also efficiently simulating secondary rays required for rendering phenomena like reflection, refraction, and shadows. However, 3DGRT requires dedicated ray-tracing hardware and remains slower than 3DGS.\n\nTo mitigate this limitation, we also propose 3DGUT, which enables support for distorted cameras with complex, time-dependent effects within a rasterization framework, maintaining the efficiency of rasterization methods. By aligning the rendering formulations of 3DGRT and 3DGUT, we introduce a hybrid approach called **3DGRUT**. This technique allows for rendering primary rays via rasterization and secondary rays via ray tracing, combining the strengths of both methods for improved performance and flexibility.\n\nFor projects that require a fast, modular, and production-ready Gaussian Splatting framework, we recommend using [gsplat](https:\u002F\u002Fgithub.com\u002Fnerfstudio-project\u002Fgsplat), which also provides support for 3DGUT.\n\n> __3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes__\n> [Nicolas Moenne-Loccoz*](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fnicolas-moënne-loccoz-71040512\u002F?original_referer=https%3A%2F%2Fwww%2Egoogle%2Ecom%2F&originalSubdomain=ca), [Ashkan Mirzaei*](https:\u002F\u002Fashmrz.github.io), [Or Perel](https:\u002F\u002Forperel.github.io\u002F), [Riccardo De Lutio](https:\u002F\u002Friccardodelutio.github.io\u002F), [Janick Martinez Esturo](https:\u002F\u002Fjme.pub\u002F),\n> [Gavriel State](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fgavstate\u002F?originalSubdomain=ca), [Sanja Fidler](https:\u002F\u002Fwww.cs.utoronto.ca\u002F~fidler\u002F), [Nicholas Sharp^](https:\u002F\u002Fnmwsharp.com\u002F), [Zan Gojcic^](https:\u002F\u002Fzgojcic.github.io\u002F) _(*,^ indicates equal contribution)_\n> _SIGGRAPH Asia 2024 (Journal Track)_\n> __[Project page](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGRT) \u002F [Paper](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGRT\u002Fres\u002F3dgrt_compressed.pdf) \u002F [Video](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGRT\u002Fres\u002F3dgrt_supplementary_video.mp4) \u002F [BibTeX](assets\u002F3dgrt2024.bib)__\n\n> __3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting__\n> [Qi Wu*](https:\u002F\u002Fwilsoncernwq.github.io\u002F), [Janick Martinez Esturo*](https:\u002F\u002Fjme.pub\u002F), [Ashkan Mirzaei](https:\u002F\u002Fashmrz.github.io),\n> [Nicolas Moenne-Loccoz](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fnicolas-moënne-loccoz-71040512\u002F?original_referer=https%3A%2F%2Fwww%2Egoogle%2Ecom%2F&originalSubdomain=ca), [Zan Gojcic](https:\u002F\u002Fzgojcic.github.io\u002F)  _(* indicates equal contribution)_\n> _CVPR 2025 (Oral)_\n> __[Project page](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGUT) \u002F [Paper](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGUT\u002Fres\u002F3DGUT_ready_main.pdf) \u002F [Video](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGUT\u002F#supp_video) \u002F [BibTeX](assets\u002F3dgut2025.bib)__\n\n\n## 🔥 News\n- ✅[2026\u002F03] **NCore v4:** Support for training from NCore v4 datasets ([NCore](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fncore), [commands](#training-on-ncore-v4-datasets)).\n- ✅[2026\u002F01] Physically-Plausible ISP support.\n- ✅[2025\u002F08] Support for the 3DGRT and 3DGS\u002F3DGRT pipelines is now available with the Vulkan API as part of the [Vulkan Gaussian Splatting Project](https:\u002F\u002Fgithub.com\u002Fnvpro-samples\u002Fvk_gaussian_splatting). 3DGUT will also be available soon.\n- ✅[2025\u002F07] Support for datasets with multiple sensors (only for COLMAP-style datasets).\n- ✅[2025\u002F07] Support for Windows has been added.\n- ✅[2025\u002F06] Playground supports PBR meshes and environment maps.\n- ✅[2025\u002F04] Support for image masks.\n- ✅[2025\u002F04] SparseAdam support.\n- ✅[2025\u002F04] MCMC densification strategy support.\n- ✅[2025\u002F04] Stable release [v1.0.0](https:\u002F\u002Fgithub.com\u002Fnv-tlabs\u002F3dgrut\u002Freleases\u002Ftag\u002Fv1.0.0) tagged.\n- ✅[2025\u002F03] Initial code release!\n- ✅[2025\u002F02] [3DGUT](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGUT\u002Fres\u002F3DGUT_ready_main.pdf) was accepted to CVPR 2025!\n- ✅[2024\u002F08] [3DGRT](https:\u002F\u002Fresearch.nvidia.com\u002Flabs\u002Ftoronto-ai\u002F3DGRT\u002Fres\u002F3dgrt_compressed.pdf) was accepted to SIGGRAPH Asia 2024!\n\n## Contents\n\n- [🔥 News](#-news)\n- [Contents](#contents)\n- [🔧 1 Dependencies and Installation](#-1-dependencies-and-installation)\n  - [Running with Docker](#running-with-docker)\n- [💻 2. Train 3DGRT or 3DGUT scenes](#-2-train-3dgrt-or-3dgut-scenes)\n  - [Training on NCore v4 datasets](#training-on-ncore-v4-datasets)\n  - [Using image masks](#using-image-masks)\n  - [Exporting USDZ for use in Omniverse and Isaac Sim](#exporting-usdz-for-use-in-omniverse-and-isaac-sim)\n- [🎥 3. Rendering from Checkpoints](#-3-rendering-from-checkpoints)\n  - [To visualize training progress interactively](#to-visualize-training-progress-interactively)\n  - [To visualize a pre-trained checkpoint](#to-visualize-a-pre-trained-checkpoint)\n- [📋 4. Evaluations](#-4-evaluations)\n- [🛝 5. Interactive Playground GUI](#-5-interactive-playground-gui)\n- [📄 6. Contributing](#-6-contributing)\n- [🎓 7. Citations](#-7-citations)\n- [🙏 8. Acknowledgements](#-8-acknowledgements)\n\n## 🔧 1 Dependencies and Installation\n- Supported CUDA versions: 11.8, 12.4, 12.6, 12.8 (default), 13.0 (experimental)\n- For good performance with 3DGRT, we recommend using an NVIDIA GPU with Ray Tracing (RT) cores.\n- Both Linux and Windows are supported via UV install scripts.\n\n### Option A: Using UV (Recommended)\n\n(Kindly contributed by [@MasahiroOgawa](https:\u002F\u002Fgithub.com\u002FMasahiroOgawa))\n\n[UV](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002F) provides faster installation and better dependency resolution.\n\n```bash\ngit clone --recursive https:\u002F\u002Fgithub.com\u002Fnv-tlabs\u002F3dgrut.git\ncd 3dgrut\n```\n\n\u003Cdetails open>\n\u003Csummary>\u003Cstrong>Linux\u003C\u002Fstrong>\u003C\u002Fsummary>\n\nThe install scripts automatically find or install a GCC version compatible with your chosen CUDA toolkit.\n\n**Prerequisites:**\n1. **uv** installed: `curl -LsSf https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.sh | sh`\n2. A CUDA toolkit — choose one of the sub-options below.\n3. **OpenGL headers** for playground: `sudo apt-get install libgl1-mesa-dev`\n\n**Sub-option A1 — System CUDA** (use an existing `nvcc` in `PATH` or `CUDA_HOME`):\n\n```bash\n.\u002Finstall_env_uv.sh          # venv name defaults to \"3dgrut\"\nsource .venv\u002Fbin\u002Factivate\n```\n\n**Sub-option A2 — conda-managed CUDA** (let conda install the CUDA toolkit):\n\n```bash\n# Step 1: create a conda environment with the CUDA toolkit\nCUDA_VERSION=12 .\u002Fscripts\u002Fcreate_conda.sh 3dgrut\nconda activate 3dgrut\n# Step 2: install Python dependencies\n.\u002Finstall_env_uv.sh  # or conda run -n 3dgrut .\u002Finstall_env_uv.sh\n```\n\n**Sub-option A3 — Local venv CUDA** (download CUDA into `.venv\u002F`, no system-wide install needed):\n\n```bash\n# Supported CUDA_VERSION values: 11.8 (or 11), 12.4, 12.6, 12.8 (or 12), 13.0 (or 13)\nFORCE_LOCAL_CUDA=1 CUDA_VERSION=12 .\u002Fscripts\u002Fcreate_venv_cuda.sh   # ~4 GB download on first run\nsource .venv\u002Fbin\u002Factivate\n.\u002Finstall_env_uv.sh\n```\n\n> [!NOTE]\n> Requires **wget**: `sudo apt-get install wget`\n> The CUDA toolkit runfile (~4GB) is cached at `\u002Ftmp\u002Fcuda_\u003Cversion>_linux.run` and reused on subsequent runs.\n> The downloaded CUDA toolkit is installed locally to `.venv\u002Fcuda-{version}\u002F`. You can force a local install\n> even with system CUDA available by setting `FORCE_LOCAL_CUDA=1` in the environment variables.\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>Windows\u003C\u002Fstrong>\u003C\u002Fsummary>\n\n**Prerequisites:**\n1. **uv** installed: `powershell -ExecutionPolicy ByPass -c \"irm https:\u002F\u002Fastral.sh\u002Fuv\u002Finstall.ps1 | iex\"`\n2. **CUDA Toolkit** installed from [NVIDIA CUDA Downloads](https:\u002F\u002Fdeveloper.nvidia.com\u002Fcuda-downloads)\n3. **Visual Studio Build Tools** (2019 or later) with the **Desktop development with C++** workload.\n   The script auto-detects `cl.exe`, `cmake`, and `ninja` from the VS installation.\n   If both a CUDA-compatible VS (2019–2022) and a newer one are installed, the script prefers the compatible version.\n   For VS 2025+ (not yet officially supported by CUDA), the script automatically adds `--allow-unsupported-compiler` to nvcc.\n\nFrom a PowerShell terminal in the project root:\n\n```powershell\n.\\install_env_uv.ps1                     # auto-detects CUDA, venv name defaults to \"3dgrut\"\n```\n\nTo override `CUDA_HOME`:\n\n```powershell\n$env:CUDA_HOME = \"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.8\"\n.\\install_env_uv.ps1\n```\n\nAfter installation, **activate the virtual environment** (required for every new terminal session):\n\n```powershell\n.venv\\Scripts\\Activate.ps1\n```\n\nThis also sets the build environment variables (`TORCH_CUDA_ARCH_LIST`, `CUDA_HOME`, VS Build Tools paths, etc.) that were persisted during installation.\n\n\u003C\u002Fdetails>\n\n### Option B: Using Legacy Conda Script\n\n\u003Cdetails>\n\u003Csummary>Legacy Conda Installation via \u003Ccode>install_env.sh\u003C\u002Fcode> (CUDA 11.8.0 \u002F 12.8.1 only)\u003C\u002Fsummary>\n\u003C\u002Fbr>\n\n> [!NOTE]\n> `install_env.sh` is a legacy script that only supports CUDA 11.8.0 and 12.8.1 and requires\n> manual GCC management. For new setups, prefer **Sub-option A1** above, which supports more\n> CUDA versions and handles GCC compatibility automatically.\n\n```bash\ngit clone --recursive https:\u002F\u002Fgithub.com\u002Fnv-tlabs\u002F3dgrut.git\ncd 3dgrut\nchmod +x install_env.sh\n.\u002Finstall_env.sh 3dgrut\nconda activate 3dgrut\n```\n\nIf your system GCC is newer than 11, install `gcc-11` first and pass the `WITH_GCC11` flag:\n```sh\nsudo apt-get install gcc-11 g++-11\n.\u002Finstall_env.sh 3dgrut WITH_GCC11\n```\n\u003C\u002Fdetails>\n\n### Blackwell \u002F RTX 50 series Support\n\nWe support CUDA 12.8 (Blackwell \u002F RTX 50 series) — kindly contributed by \u003Ca href=\"https:\u002F\u002Fwww.github.com\u002Fjohnnynunez\">@johnnynunez\u003C\u002Fa>:\n\nUsing the legacy script:\n```sh\nCUDA_VERSION=12.8.1 .\u002Finstall_env.sh 3dgrut_cuda12 WITH_GCC11\n```\n\nOr using the UV script:\n```sh\nFORCE_LOCAL_CUDA=1 CUDA_VERSION=12 .\u002Fscripts\u002Fcreate_venv_cuda.sh 3dgrut_cuda12\nsource .venv\u002Fbin\u002Factivate\n.\u002Finstall_env_uv.sh\n```\n\n### Building and Running with Docker\n\nTo build the Docker image:\n```sh\ndocker build --build-arg CUDA_VERSION=12.8.1 -t 3dgrut:cuda128 .\n```\n\nBuild the Docker image:\n```bash\ngit clone --recursive https:\u002F\u002Fgithub.com\u002Fnv-tlabs\u002F3dgrut.git\ncd 3dgrut\ndocker build . -t 3dgrut\n```\n\nRun it:\n```bash\nxhost +local:root\ndocker run -v --rm -it --gpus=all --net=host --ipc=host -v $PWD:\u002Fworkspace --runtime=nvidia -e DISPLAY 3dgrut\n```\n> [!NOTE]\n> Remember to set the DISPLAY environment variable if you are running on a remote server from the command line.\n\n## 💻 2. Train 3DGRT or 3DGUT scenes\n\nWe provide different configurations for training using 3DGRT and 3DGUT models on common benchmark datasets.\nFor example, you can download the [NeRF Synthetic dataset](https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fnguyenhung1903\u002Fnerf-synthetic-dataset),\nthe [MipNeRF360 dataset](https:\u002F\u002Fjonbarron.info\u002Fmipnerf360\u002F), or [ScanNet++](https:\u002F\u002Fkaldir.vc.in.tum.de\u002Fscannetpp\u002F),\nand then run one of the following commands:\n\n```bash\n# Train Lego with 3DGRT & 3DGUT\npython train.py --config-name apps\u002Fnerf_synthetic_3dgrt.yaml path=data\u002Fnerf_synthetic\u002Flego out_dir=runs experiment_name=lego_3dgrt\npython train.py --config-name apps\u002Fnerf_synthetic_3dgut.yaml path=data\u002Fnerf_synthetic\u002Flego out_dir=runs experiment_name=lego_3dgut\n\n# Train Bonsai\npython train.py --config-name apps\u002Fcolmap_3dgrt.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgrt dataset.downsample_factor=2\npython train.py --config-name apps\u002Fcolmap_3dgut.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgut dataset.downsample_factor=2\n\n# Train Scannet++\npython train.py --config-name apps\u002Fscannetpp_3dgrt.yaml path=data\u002Fscannetpp\u002F0a5c013435\u002Fdslr out_dir=runs experiment_name=0a5c013435_3dgrt\npython train.py --config-name apps\u002Fscannetpp_3dgut.yaml path=data\u002Fscannetpp\u002F0a5c013435\u002Fdslr out_dir=runs experiment_name=0a5c013435_3dgut\n```\n\n### Training on NCore v4 datasets\n\nSet `path` to your **NCore v4 sequence JSON**. Data layout and tooling are described in the open-source [**NCore**](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fncore) repository. Training defaults are in `configs\u002Fdataset\u002Fncore.yaml`.\n\n```bash\npython train.py --config-name apps\u002Fncore_3dgut.yaml      path=\u003Cpath>\u002F\u003Csequence-meta>.json out_dir=runs experiment_name=ncore_3dgut\npython train.py --config-name apps\u002Fncore_3dgut_mcmc.yaml path=\u003Cpath>\u002F\u003Csequence-meta>.json out_dir=runs experiment_name=ncore_3dgut_mcmc\npython train.py --config-name apps\u002Fncore_3dgrt.yaml      path=\u003Cpath>\u002F\u003Csequence-meta>.json out_dir=runs experiment_name=ncore_3dgrt\npython train.py --config-name apps\u002Fncore_3dgrt_mcmc.yaml path=\u003Cpath>\u002F\u003Csequence-meta>.json out_dir=runs experiment_name=ncore_3dgrt_mcmc\n# Example overrides: dataset.downsample=0.5 num_workers=8\n```\n\nWe also support the MCMC densification strategy and the selective Adam optimizer for 3DGRT and 3DGUT.\n\nTo enable MCMC, use:\n```bash\npython train.py --config-name apps\u002Fcolmap_3dgrt_mcmc.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgrt dataset.downsample_factor=2\npython train.py --config-name apps\u002Fcolmap_3dgut_mcmc.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgut dataset.downsample_factor=2\n```\n\nTo enable selective Adam, use:\n```bash\npython train.py --config-name apps\u002Fcolmap_3dgrt.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgrt dataset.downsample_factor=2 optimizer.type=selective_adam\npython train.py --config-name apps\u002Fcolmap_3dgut.yaml path=data\u002Fmipnerf360\u002Fbonsai out_dir=runs experiment_name=bonsai_3dgut dataset.downsample_factor=2 optimizer.type=selective_adam\n```\n\nIf you use MCMC and Selective Adam in your research, please cite [3dgs-mcmc](https:\u002F\u002Fgithub.com\u002Fubc-vision\u002F3dgs-mcmc), [taming-3dgs](https:\u002F\u002Fgithub.com\u002Fhumansensinglab\u002Ftaming-3dgs),\nand the [gSplat](https:\u002F\u002Fgithub.com\u002Fnerfstudio-project\u002Fgsplat\u002Ftree\u002Fmain) library from which the code was adopted (links to the code are provided in the source files).\n\n> [!Note]\n> For ScanNet++, we expect the dataset to be preprocessed using the method described in [FisheyeGS](https:\u002F\u002Fgithub.com\u002Fzmliao\u002FFisheye-GS?tab=readme-ov-file#prepare-training-data-on-scannet-dataset).\n\n> [!Note]\n> If you're running from the PyCharm IDE, enable the rich console as follows:\n> Run Configuration > Modify Options > Emulate terminal in output console*\n\n### Using image masks\nIn order to use image masks, you need to provide a mask for each image in the dataset. The mask is a grayscale image (0s and 1s) that masks out the parts of the image that should not be used during training, i.e. all the pixels with value 0 will be ignored in the loss computation.\n\nThe provided masks should have the same resolution as their corresponding images and be stored in the same folder with the same name but with `_mask.png` extension. For example, to mask out the parts of the image `path-to-image\u002Fimage.jpeg`, the mask should be stored at `path-to-image\u002Fimage_mask.png`.\n\n**NOTE**: The masks are only used for loss computation and not for computing the metrics.\n\n### Exporting USDZ for use in Omniverse and Isaac Sim\n\nAs a beta feature, Omniverse Kit 107.3 and Isaac Sim 5.0 are able to support rendering 3D Gaussians in a specific custom USDZ-based format that uses an extension of the UsdVolVolume Schema.\n\nThe 3DGRUT repository can output trained scenes to this format by enabling the `export_usd` flag:\n\n```bash\npython train.py --config-name apps\u002Fcolmap_3dgut.yaml path=data\u002Fmipnerf360\u002Fgarden\u002F out_dir=runs experiment_name=garden_3dgut dataset.downsample_factor=2 export_usd.enabled=true\n```\n\n> [!NOTE]\n> The USD output schema is currently compatible with Isaac Sim 5.0, but how USD and reconstruction workflows work together is highly likely to change in future versions. This is a beta feature.\n\n#### Converting PLY files to USDZ\n\nIf you have existing Gaussian data in PLY format, for example, from 3DGS, you can convert it to the USDZ format using the `ply_to_usd.py` script:\n\n```bash\npython -m threedgrut.export.scripts.ply_to_usd path\u002Fto\u002Fyour\u002Fmodel.ply --output_file path\u002Fto\u002Foutput.usdz\n```\n\nThis is useful for converting 3DGS models from other sources to the USDZ format. Note that the resulting USDZ does not include a mesh. If you need a mesh inside the USDZ (e.g. for collision geometry), follow the next step.\n\n#### Adding a Mesh to a USDZ File\n\nYou can add a mesh (PLY or USD) into an existing USDZ file using the `add_mesh_to_usdz.py` script. This is useful for producing USDZ assets with physics properties such as collision geometry.\n\n\n```bash\npython -m threedgrut.export.scripts.add_mesh_to_usdz --input_usdz path\u002Fto\u002Finput.usdz --output_usdz path\u002Fto\u002Foutput.usdz --mesh_ply path\u002Fto\u002Fmesh.ply --set_collision\n```\n\nOptional flags:\n- `--set_collision` — enable collision on mesh prims.\n- `--set_invisible` — make mesh prims invisible.\n- `--referencing_usd` — specify which USD file in the package to modify (default: auto-detect the one with a Volume prim).\n\n## 🎥 3. Rendering from Checkpoints\nEvaluate a checkpoint with splatting, the OptiX tracer, or PyTorch:\n```bash\npython render.py --checkpoint runs\u002Flego\u002Fckpt_last.pt --out-dir outputs\u002Feval\n```\n\n### To visualize training progress interactively\n```bash\npython train.py --config-name apps\u002Fnerf_synthetic_3dgut.yaml path=data\u002Fnerf_synthetic\u002Flego with_gui=True\n```\n> [!NOTE]\n> Remember to set the DISPLAY environment variable if you are running on a remote server from the command line.\n\nAlternatively, use the viser GUI contributed by the community (@tangkangqi):\n```bash\npython train.py --config-name apps\u002Fnerf_synthetic_3dgut.yaml path=data\u002Fnerf_synthetic\u002Flego with_viser_gui=True\n```\n> [!NOTE]\n> Remember to install viser first via `pip install viser` and forward the port 8080 to your local machine if you are running on a remote server.\n\n\n### To visualize a pre-trained checkpoint\n```bash\npython train.py --config-name apps\u002Fnerf_synthetic_3dgut.yaml path=data\u002Fnerf_synthetic\u002Flego with_gui=True test_last=False export_ingp.enabled=False resume=runs\u002Flego\u002Fckpt_last.pt\n```\n\nOn startup, you might see a black screen, but you can use the GUI to navigate to the correct camera views:\n\u003Cimg src=\"assets\u002Ftrain_gui_initial.jpg\" height=\"400\"\u002F>\n\u003Cimg src=\"assets\u002Frender_lego.jpg\" height=\"400\"\u002F>\n\nSimilarly, you can use the viser GUI by setting `with_viser_gui=True` instead of `with_gui=True`.\n\n\n## 📋 4. Evaluations\n\nWe provide scripts to reproduce results reported in publications.\n\n```bash\n# Training\nbash .\u002Fbenchmark\u002Fmipnerf360_3dgut.sh \u003Cconfig-yaml>\n# Rendering\nbash .\u002Fbenchmark\u002Fmipnerf360_3dgut_render.sh \u003Cresults-folder>\n```\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>\u003Ca name=\"grt-benchmark\">3DGRT Results Produced on RTX 5090\u003C\u002Fa>\u003C\u002Fstrong>\u003C\u002Fsummary>\n\u003Cbr\u002F>\n\n\n**NeRF Synthetic Dataset**\n\n```bash\nbash .\u002Fbenchmark\u002Fnerf_synthetic.sh apps\u002Fnerf_synthetic_3dgrt.yaml\nbash .\u002Fbenchmark\u002Fnerf_synthetic_render.sh results\u002Fnerf_synthetic\n```\n\n|            | PSNR\t  | SSIM\t| Train (s) |\tFPS |\n|------------|--------|-------|-------|------|\n| Chair      | 35.85\t| 0.988\t| 556.4\t| 299 |\n| Drums      | 25.87\t| 0.953\t| 462.8\t| 389 |\n| Ficus      | 36.57\t| 0.989\t| 331.0\t| 465 |\n| Hotdog     | 37.88\t| 0.986\t| 597.0\t| 270 |\n| Lego       | 36.70\t| 0.985\t| 469.8\t| 360 |\n| Materials  | 30.42\t| 0.962\t| 463.3\t| 347 |\n| Mic        | 35.90\t| 0.992\t| 443.4\t| 291 |\n| Ship       | 31.73\t| 0.909\t| 510.7\t| 360 |\n| *Average*  | 33.87\t| 0.971\t| 479.3\t| 347 |\n\n\n**MipNeRF360 Dataset**\n\n```bash\nbash .\u002Fbenchmark\u002Fmipnerf360.sh apps\u002Fcolmap_3dgrt.yaml\nbash .\u002Fbenchmark\u002Fmipnerf360_render.sh results\u002Fmipnerf360\n```\n|           | PSNR  | SSIM\t| Train (s) |\tFPS |\n|-----------|-------|-------|-------|------|\n| Bicycle   | 24.85\t| 0.748\t| 2335\t| 66 |\n| Bonsai    | 31.95\t| 0.942\t| 3383\t| 72 |\n| Counter   | 28.47\t| 0.905\t| 3247\t| 62 |\n| Flowers   | 21.42\t| 0.615\t| 2090\t| 86 |\n| Garden    | 26.97\t| 0.852\t| 2253\t| 70 |\n| Kitchen   | 30.13\t| 0.921\t| 4837\t| 39 |\n| Room      | 30.35\t| 0.911\t| 2734\t| 73 |\n| Stump     | 26.37\t| 0.770\t| 1995\t| 73 |\n| Treehill  | 22.08\t| 0.622\t| 2413\t| 68 |\n| *Average* | 27.22\t| 0.817\t| 2869\t| 68 |\n\n\u003C\u002Fdetails>\n\n\n\u003Cdetails>\n\u003Csummary>\u003Cstrong>\u003Ca name=\"gut-benchmark\">3DGUT Results Produced on RTX 5090\u003C\u002Fa>\u003C\u002Fstrong>\u003C\u002Fsummary>\n\u003Cbr\u002F>\n\n**NeRF Synthetic Dataset**\n\n```bash\nbash .\u002Fbenchmark\u002Fnerf_synthetic.sh paper\u002F3dgut\u002Funsorted_nerf_synthetic.yaml\nbash .\u002Fbenchmark\u002Fnerf_synthetic_render.sh results\u002Fnerf_synthetic\n```\n\n|            | PSNR\t  | SSIM\t| Train (s) |\tFPS |\n|------------|--------|-------|-------|------|\n| Chair      | 35.61\t| 0.988\t| 265.6\t| 599  |\n| Drums      | 25.99\t| 0.953\t| 254.1\t| 694  |\n| Ficus      | 36.43\t| 0.988\t| 183.5\t| 1053 |\n| Hotdog     | 38.11\t| 0.986\t| 184.8\t| 952  |\n| Lego       | 36.47\t| 0.984\t| 221.7\t| 826  |\n| Materials  | 30.39\t| 0.960\t| 194.3\t| 1000 |\n| Mic        | 36.32\t| 0.992\t| 204.7\t| 775  |\n| Ship       | 31.72\t| 0.908\t| 208.5\t| 870  |\n| *Average*  | 33.88\t| 0.970\t| 214.6\t| 846  |\n\n\n**MipNeRF360 Dataset**\n\nGS Strategy, Unsorted\n\n```bash\nbash .\u002Fbenchmark\u002Fmipnerf360.sh paper\u002F3dgut\u002Funsorted_colmap.yaml\nbash .\u002Fbenchmark\u002Fmipnerf360_render.sh results\u002Fmipnerf360\n```\n\n|           | PSNR  | SSIM\t| Train (s) |\tFPS |\n|-----------|-------|-------|-------|------|\n| Bicycle   | 25.01\t| 0.759\t| 949.8\t| 275 |\n| Bonsai    | 32.46\t| 0.945\t| 485.3\t| 362 |\n| Counter   | 29.14\t| 0.911\t| 484.5\t| 380 |\n| Flowers   | 21.45\t| 0.612\t| 782.0\t| 253 |\n| Garden    | 27.18\t| 0.856\t| 810.2\t| 316 |\n| Kitchen   | 31.16\t| 0.928\t| 664.8\t| 275 |\n| Room      | 31.63\t| 0.920\t| 448.8\t| 370 |\n| Stump     | 26.50\t| 0.773\t| 742.6\t| 319 |\n| Treehill  | 22.35\t| 0.627\t| 809.6\t| 299 |\n| *Average* | 27.43\t| 0.815\t| 686.4\t| 317 |\n\n\nMCMC Strategy, Unsorted\n\n```bash\nbash .\u002Fbenchmark\u002Fmipnerf360.sh paper\u002F3dgut\u002Funsorted_colmap_mcmc.yaml\nbash .\u002Fbenchmark\u002Fmipnerf360_render.sh results\u002Fmipnerf360\n```\n|           | PSNR  | SSIM\t| Train (s) |\tFPS |\n|-----------|-------|-------|-------|------|\n| Bicycle   | 25.31\t| 0.765\t| 502.3\t| 361 |\n| Bonsai    | 32.51\t| 0.947\t| 670.6\t| 274 |\n| Counter   | 29.40\t| 0.916\t| 752.7\t| 254 |\n| Flowers   | 21.86\t| 0.616\t| 553.3\t| 298 |\n| Garden    | 27.06\t| 0.852\t| 512.7\t| 360 |\n| Kitchen   | 31.71\t| 0.930\t| 739.6\t| 258 |\n| Room      | 32.04\t| 0.928\t| 643.7\t| 313 |\n| Stump     | 27.06\t| 0.795\t| 487.0\t| 339 |\n| Treehill  | 23.11\t| 0.650\t| 508.6\t| 365 |\n| *Average* | 27.78\t| 0.822\t| 596.7\t| 308 |\n\nGS Strategy, Unsorted, Sparse Adam\n\n|           | PSNR   | SSIM  | Train (s) | FPS |\n|-----------|--------|-------|-----------|-----|\n| Bicycle   | 25.04  | 0.759 | 835.2     | -   |\n| Bonsai    | 32.63  | 0.945 | 457.1     | -   |\n| Counter   | 29.12  | 0.911 | 468.8     | -   |\n| Flowers   | 21.55  | 0.614 | 741.7     | -   |\n| Garden    | 27.12  | 0.855 | 757.4     | -   |\n| Kitchen   | 31.37  | 0.929 | 639.3     | -   |\n| Room      | 31.72  | 0.921 | 415.2     | -   |\n| Stump     | 26.58  | 0.774 | 695.7     | -   |\n| Treehill  | 22.30  | 0.625 | 749.8     | -   |\n| *Average* | 27.49  | 0.815 | 640.0     | -   |\n\n\n**Scannet++ Dataset**\n\n```bash\nbash .\u002Fbenchmark\u002Fscannetpp.sh paper\u002F3dgut\u002Funsorted_scannetpp.yaml\nbash .\u002Fbenchmark\u002Fscannetpp_render.sh results\u002Fscannetpp\n```\n> [!Note]\n> We followed [FisheyeGS](https:\u002F\u002Fgithub.com\u002Fzmliao\u002FFisheye-GS?tab=readme-ov-file#prepare-training-data-on-scannet-dataset)'s convention to prepare the dataset for fair comparisons.\n\n|           | PSNR  | SSIM\t| Train (s) |\tFPS |\n|-----------|-------|-------|-------|------|\n| 0a5c013435 | 29.67\t| 0.930\t| 292.3\t| 389 |\n| 8d563fc2cc | 26.88\t| 0.912\t| 286.1\t| 439 |\n| bb87c292ad | 31.58\t| 0.941\t| 316.9\t| 448 |\n| d415cc449b | 28.12\t| 0.871\t| 394.6\t| 483 |\n| e8ea9b4da8 | 33.47\t| 0.954\t| 280.8\t| 394 |\n| fe1733741f | 25.60\t| 0.858\t| 355.8\t| 450 |\n| *Average*  | 29.22\t| 0.911\t| 321.1\t| 434 |\n\n\u003C\u002Fdetails>\n\n## 🛝 5. Interactive Playground GUI\n\nThe playground allows interactive exploration of pretrained scenes, with ray-tracing effects such as inserted objects,\nreflections, refractions, depth of field, and more.\n\nRun the playground UI to visualize a pretrained scene with:\n```\npython playground.py --gs_object \u003Cckpt_path>\n```\n\nSee [Playground README](threedgrut_playground\u002FREADME.md) for details.\n\n*Update (2025\u002F04): The playground engine is now exposed, and remote rendering is supported; see README for details.*\n\n## 📄 6. Contributing\n\nContributions are welcome! Please feel free to submit a pull request.\n\nFormatting uses `black` and `isort`. Please run\n`black . --target-version=py311 --line-length=120 --exclude=thirdparty\u002Ftiny-cuda-nn` and\n`isort . --skip=thirdparty\u002Ftiny-cuda-nn --profile=black` before submitting a pull request.\n\n## 🎓 7. Citations\n\n```\n@article{loccoz20243dgrt,\n    author = {Nicolas Moenne-Loccoz and Ashkan Mirzaei and Or Perel and Riccardo de Lutio and Janick Martinez Esturo and Gavriel State and Sanja Fidler and Nicholas Sharp and Zan Gojcic},\n    title = {3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes},\n    journal = {ACM Transactions on Graphics and SIGGRAPH Asia},\n    year = {2024},\n}\n```\n\n```\n@article{wu20253dgut,\n    title={3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting},\n    author={Wu, Qi and Martinez Esturo, Janick and Mirzaei, Ashkan and Moenne-Loccoz, Nicolas and Gojcic, Zan},\n    journal = {Conference on Computer Vision and Pattern Recognition (CVPR)},\n    year={2025}\n}\n```\n\n## 🙏 8. Acknowledgements\n\nWe sincerely thank our colleagues for their valuable contributions to this project.\n\nHassan Abu Alhaija, Ronnie Sharif, Beau Perschall and Lars Fabiunke for assistance with assets.\nGreg Muthler, Magnus Andersson, Maksim Eisenstein, Tanki Zhang, Nathan Morrical, Dietger van Antwerpen and John Burgess for performance feedback.\nThomas Müller, Merlin Nimier-David, and Carsten Kolve for inspiration and pointers.\nZiyu Chen, Clement Fuji-Tsang, Masha Shugrina, and George Kopanas for technical & experiment assistance,\nand to Ramana Kiran and Shailesh Mishra for typo fixes.\n","3DGRUT 是一个用于高斯粒子的光线追踪和混合光栅化的项目。它主要实现了3D Gaussian Ray Tracing (3DGRT) 和 3D Gaussian Unscented Transform (3DGUT) 技术，前者通过光线追踪体素高斯粒子来支持具有复杂时间依赖效应（如卷帘快门）的扭曲相机，并能有效模拟反射、折射和阴影等渲染现象；后者则在光栅化框架内提供对这些效果的支持，保持了光栅化方法的效率。此外，3DGRUT 还结合了两种技术的优势，允许使用光栅化处理主光线，光线追踪处理次级光线，从而提高性能和灵活性。该项目适合需要高质量3D重建与计算机视觉效果的应用场景，特别是在存在复杂成像条件的情况下。",2,"2026-06-11 03:42:34","high_star"]