[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-94864":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":20,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":29,"readmeContent":30,"aiSummary":31,"trendingCount":16,"starSnapshotCount":16,"syncStatus":32,"lastSyncTime":33,"discoverSource":34},94864,"SCoPE","TencentARC\u002FSCoPE","TencentARC","SCoPE: Sightline-Coordinate Positional Encoding for 3D-Aware Video Generation","https:\u002F\u002Fvisual-ai.github.io\u002Fscope\u002F",null,"Python",152,11,5,1,0,39,66.64,"Other",false,"main",[23,24,25,26,27,28],"3d","dit","novel-view-synthesis","video-diffusion","video-diffusion-model","world-models","2026-08-24 04:01:22","\u003Cdiv align=\"center\">\n\n# SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers\n\nMinghao Yin · Jiahao Lu · Wenbo Hu · Wang Zhao · Ying Shan · Kai Han\n\n[![Project Page](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject%20Page-visual--ai.github.io%2Fscope-1E90FF)](https:\u002F\u002Fvisual-ai.github.io\u002Fscope\u002F)\n[![arXiv](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FarXiv-2606.27345-B31B1B?logo=arxiv)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.27345)\n[![GitHub](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FGitHub-SCoPE-181717?logo=github)](https:\u002F\u002Fgithub.com\u002FTencentARC\u002FSCoPE)\n[![Demo](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗%20Demo-Spaces-FF9D00)](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FTencentARC\u002Fscope-camera-video-generation)\n[![Model](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗%20Model-TencentARC%2FSCoPE-FFD21E)](https:\u002F\u002Fhuggingface.co\u002FTencentARC\u002FSCoPE)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-green)](LICENSE.txt)\n\n\u003Cimg src=\"assets\u002Fteaser_part1.jpg\" alt=\"SCoPE method overview\" width=\"864\">\n\n\u003Cimg src=\"assets\u002Fteaser_grid.gif\" alt=\"Six camera-controlled SCoPE video results in a 3-by-2 grid\" width=\"864\">\n\n\u003C\u002Fdiv>\n\nSCoPE adds camera sightlines as positional coordinates to a pretrained video diffusion\ntransformer. Given a first frame, a text prompt, and a camera trajectory, it generates a video\nthat follows the requested camera motion while preserving the original image-to-video prior.\n\n## 🚀 Try It Online\nTry SCoPE in your browser — no setup required: **[🤗 Hugging Face Space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FTencentARC\u002Fscope-camera-video-generation)**\n\n## 📑 Table of Contents\n\n- [Overview](#overview)\n- [Repository structure](#repository-structure)\n- [Installation](#installation)\n- [Model download](#model-download)\n- [Inference](#inference)\n  - [Quick start — choose a trajectory](#quick-start--choose-a-trajectory)\n  - [Specify a trajectory directly](#specify-a-trajectory-directly)\n  - [All trajectories for one scene](#all-trajectories-for-one-scene)\n  - [Existing case with an external trajectory](#existing-case-with-an-external-trajectory)\n  - [Custom image and trajectory](#custom-image-and-trajectory)\n  - [Visualizing camera control](#visualizing-camera-control)\n  - [Inference options](#inference-options)\n- [Training](#training)\n- [Citation](#citation)\n- [Acknowledgements](#acknowledgements)\n- [License](#license)\n\n## Overview\n\nVideo diffusion transformers address their tokens by position on the pixel-time grid: an address\nin the tensor, not in the world. The world point a token depicts lies on a surface that has not\nbeen generated yet, but its camera ray is fixed the moment the user specifies a trajectory. SCoPE\ntherefore treats the ray as a **second positional coordinate**, so camera control becomes a\nproperty of the coordinate system rather than an added module.\n\nThe ray is added to the pretrained attention's queries and keys, and the score gains a term that\nreads the two rays alone; its canonical form — the reciprocal product of line geometry — measures\nhow nearly two lines of sight meet. A **Normalize-Gate-Inject** scheme makes a single encoding\ntrainable across both metric and up-to-scale pose sources. The retrofit keeps RoPE bit-exact,\nstarts from the unchanged pretrained DiT, and adds under 0.1% new parameters.\n\nIn practice this means:\n\n- **Camera motion as coordinates.** Each video token is tied to its camera ray via Plücker\n  coordinates; no separate control branch is trained.\n- **Robust to heterogeneous pose sources.** The per-clip near-depth normalization plus the learned\n  scale gate let SCoPE consume poses from different reconstruction pipelines and scene scales.\n- **Self-contained Wan2.2-I2V-A14B release.** The model repository contains everything required for\n  inference; users do not need to download a second Wan2.2 checkpoint.\n\n## Repository structure\n\n| Path | Description |\n| --- | --- |\n| `inference.py` | Inference entry: manifest, case + external pose, custom, or all-trajectory generation. |\n| `train.py` | Training entry for the RDPO high-only recipe. |\n| `scope\u002Fencoding.py` | Sightline-Coordinate positional encoding (Normalize-Gate-Inject). |\n| `scope\u002Fgeometry.py` | Camera-ray and Plücker-coordinate utilities. |\n| `scope\u002Fcamera.py` | Converts camera trajectories into SCoPE coordinates. |\n| `scope\u002Fmodeling.py` | Wan self-attention augmented with the SCoPE encoding. |\n| `scope\u002Fpatch.py` | Installs SCoPE attention into both Wan2.2-A14B experts. |\n| `scope\u002Fpipeline.py` | Wan2.2-A14B inference pipeline with camera conditioning. |\n| `scope\u002Fweights.py` | Loads the complete SCoPE model from sharded weights. |\n| `scope\u002Fconfig.py` | Inference defaults matching the released checkpoint. |\n| `scope\u002Ftraining.py` | Config-driven RDPO high-only training loop. |\n| `scope\u002Fdata\u002F` | Four native dataset loaders + the shared pose convention. |\n| `configs\u002F` | Training YAML and the default negative prompt. |\n| `scripts\u002F` | `overlay_camera.py`, `estimate_near_depth.py`, `build_omniworld_index.py`, `train.sh`, `inference.sh`. |\n| `examples\u002F` | `manifest.json`, example first frames, and camera trajectories. |\n| `assets\u002F` | Teaser image and demo GIFs. |\n| `diffsynth\u002F` | Vendored DiffSynth code required by inference. |\n| `tests\u002F` | Unit tests. |\n\n## Installation\n\nSCoPE requires Python 3.11 and a CUDA-capable GPU. The released weights were trained and\nevaluated with **PyTorch 2.9.1 (CUDA 12.8)**; because changing the PyTorch version can change the\nnumerical output, we recommend reproducing this exact environment.\n\nRecommended — [uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002F) (resolves the pinned CUDA 12.8 torch build):\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FTencentARC\u002FSCoPE.git\ncd SCoPE\nuv sync\nsource .venv\u002Fbin\u002Factivate\n```\n\nAlternative — pip with the PyTorch CUDA 12.8 wheel index:\n\n```bash\nconda create -n scope python=3.11 -y\nconda activate scope\npip install -e . --extra-index-url https:\u002F\u002Fdownload.pytorch.org\u002Fwhl\u002Fcu128\n```\n\nOptional extras:\n\n- `pip install -e .[viz]` installs matplotlib, needed by the camera-control overlay tool\n  `scripts\u002Foverlay_camera.py`.\n- FlashAttention is optional; the code falls back to PyTorch SDPA when it is not available.\n\n## Model download\n\nDownload the SCoPE model from Hugging Face:\n\n```bash\npip install -U huggingface_hub\nhf download TencentARC\u002FSCoPE --local-dir checkpoints\u002FSCoPE\n```\n\nThe checkpoint is approximately 67 GB. Keep both the checkpoint and the Hugging Face cache on\nlocal storage.\n\n## Inference\n\nAll inference is driven by `inference.py`. The repository ships example first frames, prompts, and\ncamera trajectories under `examples\u002F`, so every command below runs out of the box. See\n[Inference options](#inference-options) for the full flag list.\n\n### Quick start — choose a trajectory\n\n**Input:** one bundled case.\n\n**Output:** one 81-frame video using the camera trajectory selected in the terminal\n(480 × 832, seed 42).\n\nPass a case without `--trajectory` to see its available camera poses and choose one by number or\ntrajectory id:\n\n```bash\npython inference.py \\\n  --model_path checkpoints\u002FSCoPE \\\n  --case omni-misty-forest \\\n  --output_path outputs\u002Fomni-misty-forest__truck_right.mp4\n```\n\nThe terminal will prompt before loading the model:\n\n```text\nAvailable trajectories for 'omni-misty-forest':\n  [1] real_split0_000041  (poses\u002Fomni-misty-forest\u002Freal_split0_000041.npy)\n  [2] real_split0_000081  (poses\u002Fomni-misty-forest\u002Freal_split0_000081.npy)\n  [3] real_split0_000161  (poses\u002Fomni-misty-forest\u002Freal_split0_000161.npy)\n  [4] snake_fwd  (poses\u002Fomni-misty-forest\u002Fsnake_fwd.npy)\n  [5] truck_right  (poses\u002Fomni-misty-forest\u002Ftruck_right.npy)\nSelect a trajectory [1-5] or enter its id: 5\n```\n\n| Input first frame | Output (`truck_right`) |\n| :---: | :---: |\n| \u003Cimg src=\"assets\u002Fdemo\u002Fmisty_first_frame.jpg\" width=\"320\"> | \u003Cimg src=\"assets\u002Fdemo\u002Fforest_truck_right.gif\" width=\"320\"> |\n\nThe output is shown with the camera-control HUD overlay (see\n[Visualizing camera control](#visualizing-camera-control)); the bottom-right inset traces the\ndriving camera path.\n\n### Specify a trajectory directly\n\nThis is the non-interactive form of the same bundled-case inference. Pass the trajectory id\ndirectly for scripts, cluster jobs, and exactly reproducible commands:\n\n```bash\npython inference.py \\\n  --model_path checkpoints\u002FSCoPE \\\n  --case omni-misty-forest \\\n  --trajectory truck_right \\\n  --output_path outputs\u002Fomni-misty-forest__truck_right.mp4\n```\n\n`bash scripts\u002Finference.sh` wraps this explicit form. It generates the same result as choosing\n`truck_right` in the Quick Start selector.\n\n### All trajectories for one scene\n\n**Input:** one first frame + every trajectory bundled for that scene.\n**Output:** a series of videos (one per trajectory), generated with a single model load.\n\n```bash\npython inference.py \\\n  --model_path checkpoints\u002FSCoPE \\\n  --case omni-misty-forest \\\n  --all_trajectories \\\n  --output_dir outputs\u002Fomni-misty-forest\n```\n\nThe same first frame driven by all five bundled trajectories. `truck_right` and `snake_fwd` are\nsynthetic camera paths; **GT 1–3** are the scene's own recorded OmniWorld camera paths (use their\n`--trajectory` ids to reproduce them).\n\n| | | |\n| :---: | :---: | :---: |\n| **Input first frame** | **`truck_right`** | **`snake_fwd`** |\n| \u003Cimg src=\"assets\u002Fdemo\u002Fmisty_first_frame.jpg\" width=\"260\"> | ![truck_right](assets\u002Fdemo\u002Fforest_truck_right.gif) | ![snake_fwd](assets\u002Fdemo\u002Fforest_snake_fwd.gif) |\n| **GT 1** · `real_split0_000041` | **GT 2** · `real_split0_000081` | **GT 3** · `real_split0_000161` |\n| ![gt1](assets\u002Fdemo\u002Fforest_gt1.gif) | ![gt2](assets\u002Fdemo\u002Fforest_gt2.gif) | ![gt3](assets\u002Fdemo\u002Fforest_gt3.gif) |\n\nExisting MP4 files are skipped, so interrupted runs can be resumed with the same command.\n\n### Existing case with an external trajectory\n\nUse a bundled case's first frame, prompt, `x_fov`, and `xi`, but drive it with any compatible\nlocal pose file. This avoids copying case metadata or searching for a matching pose directory.\n\n```bash\npython inference.py \\\n  --model_path checkpoints\u002FSCoPE \\\n  --case omni-misty-forest \\\n  --camera_path path\u002Fto\u002Fexternal_camera_poses.npy \\\n  --output_path outputs\u002Fomni-misty-forest__external.mp4\n```\n\nThe pose must still use the same `[81,3,4]` or `[81,4,4]` OpenCV camera-to-world convention.\nSCoPE validates its shape and finite values before loading model weights. Use `--x_fov` or `--xi`\nonly when the external camera requires values different from the selected case. `--trajectory`\nand `--camera_path` are alternative pose sources and cannot be used together.\n\n### Custom image and trajectory\n\n**Input:** your own first frame, prompt, OpenCV camera-to-world poses, and horizontal FOV.\n**Output:** one video following your trajectory.\n\n```bash\npython inference.py \\\n  --model_path checkpoints\u002FSCoPE \\\n  --input_image path\u002Fto\u002Ffirst_frame.png \\\n  --prompt \"A person walks along a misty forest trail.\" \\\n  --camera_path path\u002Fto\u002Fcamera_poses.npy \\\n  --x_fov 1.11847 \\\n  --output_path outputs\u002Fcustom.mp4\n```\n\n`camera_poses.npy` must have shape `[81, 3, 4]` or `[81, 4, 4]` and use OpenCV camera-to-world\ncoordinates. `x_fov` is the horizontal field of view in radians. Pinhole cameras use the default\n`xi=0`; unified camera models can set `--xi` explicitly.\n\n### Visualizing camera control\n\n**Input:** a generated video + the camera trajectory used to produce it.\n**Output:** the same video with a camera-frustum HUD (and optional WASD keys) composited in.\n\n`scripts\u002Foverlay_camera.py` derives the overlay from the pose: it renders the accumulated camera\nfrustum in the bottom-right corner and, for keyboard-style trajectories, the active WASD keys in\nthe bottom-left corner. At runtime it also prints a lightweight WASD assessment; when horizontal\nmotion is too small or too few frames map to an active key, it recommends `--hide_wasd`. The\nassessment is advisory and does not change the output automatically. Every demo video in this\nREADME was produced with this tool.\n\n```bash\n# Camera-frustum HUD only (recommended for scenic \u002F dolly \u002F orbit motions):\npython scripts\u002Foverlay_camera.py outputs\u002Fomni-misty-forest__truck_right.mp4 \\\n  --pose examples\u002Fposes\u002Fomni-misty-forest\u002Ftruck_right.npy \\\n  --out_dir outputs\u002Foverlay --hide_wasd\n\n# Add the WASD key indicator (for keyboard-style \u002F drone trajectories):\npython scripts\u002Foverlay_camera.py outputs\u002Fclip.mp4 --pose path\u002Fto\u002Fpose.npy --out_dir outputs\u002Foverlay\n```\n\nRequires the `viz` extra (`pip install -e .[viz]`) and `ffmpeg` on `PATH`.\n\n### Inference options\n\n| Option | Default | Description |\n| --- | --- | --- |\n| `--model_path` | `TencentARC\u002FSCoPE` | Local path or Hugging Face id of the SCoPE model. |\n| `--manifest` | `examples\u002Fmanifest.json` | Example manifest for `--case` \u002F `--trajectory`. |\n| `--case` | first case | Example case id; when supplied alone, choose its trajectory interactively. |\n| `--trajectory` | – | Optional bundled trajectory id; bypasses the interactive selector and is required for non-interactive jobs using `--case`. |\n| `--all_trajectories` | off | Generate every trajectory of `--case` (uses `--output_dir`). |\n| `--input_image` | – | Custom first frame (with `--prompt`, `--camera_path`, `--x_fov`). |\n| `--prompt` | – | Custom text prompt. |\n| `--camera_path` | – | External pose `.npy`; use with `--case` or all custom-input flags. |\n| `--x_fov` | case value \u002F required for custom | Horizontal field of view in radians. |\n| `--xi` | case value \u002F `0.0` for custom | Unified-camera distortion parameter. |\n| `--output_path` | `outputs\u002Fsample.mp4` | Output file for a single generation. |\n| `--output_dir` | `outputs` | Output directory for `--all_trajectories`. |\n| `--overwrite` | off | Regenerate existing outputs (`--all_trajectories`). |\n| `--negative_prompt` | `configs\u002Fnegative_prompt.txt` | Negative prompt file. |\n| `--seed` | `42` | Random seed. |\n| `--cache_dir` | – | Hugging Face cache directory. |\n| `--vram_limit_gb` | – | VRAM budget for offloading (enables VRAM management). |\n\n## Training\n\nThe public training entry reproduces the RDPO high-noise recipe used for SCoPE: only the\nhigh-noise expert is optimized, and training timesteps are sampled from `[0.9, 1.0)`. The released\nmixture concatenates four datasets — RealEstate10K, DL3DV, PanShot, and OmniWorld — each read by\nits own native loader in `scope\u002Fdata\u002F`.\n\nA run is described by a single YAML config:\n\n```bash\npython train.py --config configs\u002Ftrain_rdpo_high_only.yaml --num_gpus 8\n```\n\n`scripts\u002Ftrain.sh` wraps this command. Multi-GPU training uses FSDP automatically; the model is\nlarge, so multi-GPU training is strongly recommended.\n\n### Training options\n\n| Option | Default | Description |\n| --- | --- | --- |\n| `--config` | `configs\u002Ftrain_rdpo_high_only.yaml` | Training YAML (data mixture, trainer, optimizer). |\n| `--model_path` | from config | Override the config `model_path`. |\n| `--output_dir` | from config | Override the checkpoint\u002Foutput directory. |\n| `--num_gpus` | from config (`8`) | Override GPU count; `>1` enables FSDP full-shard. |\n| `--max_steps` | from config (`10000`) | Override the number of training steps. |\n| `--resume_from_checkpoint` | – | Resume from an existing checkpoint. |\n\n### Camera convention\n\nEvery dataset is normalized with the same convention as RealEstate10K:\n\n1. Poses are OpenCV camera-to-world matrices, expressed **relative to the first camera** (the\n   first frame becomes the identity pose).\n2. Translation is preprocessed by `trajectory_scale \u002F near_depth`, where `near_depth` is a\n   per-clip near-distance depth estimate. This normalization only makes the translation magnitude\n   comparable across datasets — absolute scale is handled inside the model by the learned scale\n   gate, so `trajectory_scale` stays `1.0` for all datasets.\n\n### Data preparation\n\nPoint the config at your local dataset roots, then precompute the per-clip `near_depth` for each\ndataset with the shared estimator (RAFT optical flow + two-view triangulation):\n\n```bash\n# OmniWorld first needs a validity index over its training windows:\npython scripts\u002Fbuild_omniworld_index.py --data_root \u002Fpath\u002Fto\u002FOmniWorld \\\n  --output \u002Fpath\u002Fto\u002FOmniWorld\u002Fvalid_entries.json\n\n# Estimate per-clip near-depth for each dataset (repeat per dataset):\npython scripts\u002Festimate_near_depth.py --dataset realestate10k \\\n  --data_root \u002Fpath\u002Fto\u002FRealEstate10K --split train \\\n  --output \u002Fpath\u002Fto\u002FRealEstate10K\u002Fnear_depth_train.json\n```\n\nThe same `estimate_near_depth.py` handles all four datasets via `--dataset {realestate10k,dl3dv,\npanshot,omniworld}`. See `configs\u002Ftrain_rdpo_high_only.yaml` for the full list of per-dataset\npaths and options (`sample_stride` is random ≤4 for RealEstate10K and 1 for the others).\n\n## Citation\n\n```bibtex\n@article{yin2026scope,\n  title={SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers},\n  author={Yin, Minghao and Lu, Jiahao and Hu, Wenbo and Zhao, Wang and Shan, Ying and Han, Kai},\n  year={2026}\n}\n```\n\n## Acknowledgements\n\nSCoPE is built on [Wan2.2](https:\u002F\u002Fgithub.com\u002FWan-Video\u002FWan2.2) and\n[DiffSynth-Studio](https:\u002F\u002Fgithub.com\u002Fmodelscope\u002FDiffSynth-Studio). We thank the authors and\ncontributors of these projects.\n\n## License\n\nSCoPE is released under the [Apache-2.0 License](LICENSE.txt).\n","SCoPE 是一种面向3D感知视频生成的新型位置编码方法，通过将相机视线（sightline）建模为世界坐标系下的射线，并作为第二组 positional coordinate 注入视频扩散Transformer中，实现对相机运动轨迹的显式、几何一致的控制。其核心技术特点是无需微调预训练模型即可即插即用地支持文本+首帧+相机轨迹三输入驱动的视频生成，在保持原有图像到视频先验的同时，提升新视角合成与自由视角视频生成的几何合理性。适用于需要精确相机控制的3D内容创作场景，如虚拟拍摄、数字人运镜、建筑可视化漫游及AR\u002FVR素材生成。",2,"2026-08-17 02:30:08","CREATED_QUERY"]