[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92294":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":9,"rankLanguage":9,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":19,"hasPages":19,"topics":21,"createdAt":9,"pushedAt":9,"updatedAt":22,"readmeContent":23,"aiSummary":24,"trendingCount":15,"starSnapshotCount":15,"syncStatus":25,"lastSyncTime":26,"discoverSource":27},92294,"lingbot-vision","Robbyant\u002Flingbot-vision","Robbyant","Self-supervised learning for spatial perception",null,"Python",650,21,55,1,0,354,58.03,"Apache License 2.0",false,"main",[],"2026-07-22 04:02:05","\u003Cdiv align=\"center\">\n\n\u003Ch1>LingBot-Vision: Vision Pretraining for Dense Spatial Perception\u003C\u002Fh1>\n\n\u003C\u002Fdiv>\n\n\u003Cdiv align=\"center\">\n\n[![PDF](https:\u002F\u002Fimg.shields.io\u002Fstatic\u002Fv1?label=Paper&message=PDF&color=red&logo=adobeacrobatreader)](https:\u002F\u002Fgithub.com\u002FRobbyant\u002Flingbot-vision\u002Fblob\u002Fmain\u002Fpaper.pdf)\n[![arXiv](https:\u002F\u002Fimg.shields.io\u002Fstatic\u002Fv1?label=Paper&message=arXiv&color=lightgrey&logo=arxiv)](https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.05247)\n[![Project](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Website-blue)](https:\u002F\u002Ftechnology.robbyant.com\u002Flingbot-vision)\n[![HuggingFace](https:\u002F\u002Fimg.shields.io\u002Fstatic\u002Fv1?label=%F0%9F%A4%97%20Model&message=HuggingFace&color=orange)](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Frobbyant\u002Flingbot-vision)\n[![ModelScope](https:\u002F\u002Fimg.shields.io\u002Fstatic\u002Fv1?label=%F0%9F%A4%96%20Model&message=ModelScope&color=purple)](https:\u002F\u002Fwww.modelscope.cn\u002Fcollections\u002FRobbyant\u002FLingBot-Vision)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache--2.0-green)](LICENSE)\n\n\u003C\u002Fdiv>\n\n-----\n\n### 🔭 Meet LingBot-Vision! A family of self-supervised ViT backbones for dense spatial perception! 🖼️📐\n\n**LingBot-Vision** is a family of self-supervised Vision Transformer backbones for dense spatial perception, from ViT-S\u002F16 up to a 1.1B-parameter ViT-g\u002F16. The flagship model is pretrained with **masked boundary modeling** — a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations.\n\n![Boundary-centric masked modeling](.\u002Fassets\u002Fteaser_crop.png)\n\n\u003Csub>**Boundary-centric masked modeling.** Each row shows the input image, the PCA projection of frozen patch tokens, teacher-discovered boundary tokens, and cosine-similarity maps from selected boundary-token queries. The features capture semantic grouping and geometric structure at the same time.\u003C\u002Fsub>\n\nLingBot-Vision learns boundaries, shapes, and semantic regions all together, making it a drop-in visual encoder for dense downstream tasks:\n\n- 🎨 **Dense feature visualization** — PCA maps of frozen patch tokens reveal coherent object regions and crisp boundaries\n- 📏 **Depth estimation** — frozen patch tokens expose spatial structure to lightweight dense readouts\n- 🧩 **Semantic segmentation** — boundary-faithful features align region transitions with object contours\n- 🎬 **Video object segmentation** — training-free token matching and label propagation with frozen features\n- 🤖 **Depth completion** — LingBot-Vision is the visual encoder initialization for LingBot-Depth 2.0 (see below)\n\n## 🌊 Meet LingBot-Depth 2.0\n\nBy simply replacing the encoder with LingBot-Vision at the ViT-L\u002F16 and ViT-g\u002F16 scales, and scaling the curated RGB-D training corpus from 3M to 150M samples, LingBot-Depth 2.0 achieves substantial performance gains over the previous and other system, as detailed in the technical report.\n\n![LingBot-Depth 2.0 on mirror and glass scenes](.\u002Fassets\u002Flingbot_depth2_mirror_glass.jpg)\n\n\u003Csub>**LingBot-Depth 2.0 on mirror and glass scenes.** Each group shows input RGB, raw sensor depth, refined depth, and refined point clouds. Raw depth is missing on difficult surfaces such as window panes, glass balustrades, and reflective floors. LingBot-Depth 2.0 completes these regions as stable, contiguous surfaces across frames.\u003C\u002Fsub>\n\n---\n\n## 📦 Model Zoo\n\nWe train a ViT-g\u002F16 teacher with roughly 1.1B parameters and distill ViT-L, ViT-B, and ViT-S backbones from it for inference and downstream use. Full training and evaluation details are covered in the technical report.\n\nAll released weights are **backbone-only** `.pt` checkpoints, stored as `model.pt` in each model repository (see the full [Hugging Face collection](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Frobbyant\u002Flingbot-vision)):\n\n| Model | Backbone | Embed dim | Hugging Face Weights | ModelScope Weights |\n|-------|----------|----------:|----------------------|--------------------|\n| **LingBot-Vision-Giant**\u003Cbr>\u003Csub>highest-quality dense features\u003C\u002Fsub> | ViT-g\u002F16 · SwiGLU · fp32 RoPE · 4 register tokens | 1536 | [vit-giant](https:\u002F\u002Fhuggingface.co\u002Frobbyant\u002Flingbot-vision-vit-giant) | [vit-giant](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FRobbyant\u002Flingbot-vision-vit-giant) |\n| **LingBot-Vision-Large** ⭐\u003Cbr>\u003Csub>recommended: strong features, practical inference\u003C\u002Fsub> | ViT-L\u002F16, distilled from Giant | 1024 | [vit-large](https:\u002F\u002Fhuggingface.co\u002Frobbyant\u002Flingbot-vision-vit-large) | [vit-large](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FRobbyant\u002Flingbot-vision-vit-large) |\n| **LingBot-Vision-Base**\u003Cbr>\u003Csub>balanced inference cost\u003C\u002Fsub> | ViT-B\u002F16, distilled from Giant | 768 | [vit-base](https:\u002F\u002Fhuggingface.co\u002Frobbyant\u002Flingbot-vision-vit-base) | [vit-base](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FRobbyant\u002Flingbot-vision-vit-base) |\n| **LingBot-Vision-Small**\u003Cbr>\u003Csub>lightweight demos and downstream use\u003C\u002Fsub> | ViT-S\u002F16, distilled from Giant | 384 | [vit-small](https:\u002F\u002Fhuggingface.co\u002Frobbyant\u002Flingbot-vision-vit-small) | [vit-small](https:\u002F\u002Fwww.modelscope.cn\u002Fmodels\u002FRobbyant\u002Flingbot-vision-vit-small) |\n\nConfig files are packaged under `lingbot_vision\u002Fconfigs\u002F` and selected automatically by `load_pretrained_backbone`.\n\n## 🔧 Installation\n\n**Requirements**: Python ≥ 3.10 · PyTorch ≥ 2.0 · CUDA-capable GPU (recommended for large-model inference)\n\n**1. Clone the repository**\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Frobbyant\u002Flingbot-vision.git\ncd lingbot-vision\n```\n\n**2. Create a conda environment**\n\n```bash\nconda create -n lingbot-vision python=3.10 -y\nconda activate lingbot-vision\n```\n\n**3. Install lingbot-vision**\n\n```bash\npython -m pip install -r requirements.txt\npython -m pip install -e .\n```\n\n## 🚀 Quick Start\n\n### Load LingBot-Vision Pretrains with PyTorch\n\nThe model is automatically downloaded from Hugging Face on first use. This example uses the small model for a lightweight smoke run; `large` is the default variant, and `giant` is available as the largest backbone.\n\n```python\nimport torch\n\nfrom lingbot_vision import load_pretrained_backbone, extract_patch_tokens, load_image\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ndtype = torch.bfloat16 if device == \"cuda\" else torch.float32\n\n# Downloads model.pt from robbyant\u002Flingbot-vision-vit-small.\nbackbone, embed_dim = load_pretrained_backbone(\n    variant=\"small\",\n    device=device,\n    dtype=dtype,\n)\n\nimg_norm, _, _ = load_image(\n    \"examples\u002Fexample.png\",\n    size=512,\n    patch_size=backbone.patch_size,\n    mode=\"square\",\n)\npatch_tokens, patch_grid = extract_patch_tokens(backbone, img_norm, device, dtype)\n\nprint(patch_tokens.shape, patch_grid, embed_dim)\n# torch.Size([1, 1024, 384]) (32, 32) 384\n```\n\n`patch_tokens` has shape `[B, H * W, C]`, where `H` and `W` are the patch-grid dimensions. `variant` can be `giant`, `large`, `base`, or `small`; if omitted, it defaults to `large`. You can also pass a local directory or an explicit Hugging Face model repo to `load_pretrained_backbone`.\n\n### Run the PCA demo\n\nDownload a backbone checkpoint from Hugging Face (or ModelScope, once available), then run:\n\n```bash\n.\u002Fscripts\u002Frun_pca_demo.sh \\\n  --config-file lingbot_vision\u002Fconfigs\u002Flbot_vision_vitl.yaml \\\n  --ckpt \u002Fpath\u002Fto\u002Fmodel.pt \\\n  --input examples\u002Fexample.png \\\n  --out outputs\u002Fpca_demo \\\n  --size 512 \\\n  --mode square \\\n  --dtype bf16\n```\n\nImages are loaded as RGB, resized according to `--size` and `--mode`, aligned to the model patch size, and normalized with ImageNet statistics. The demo maps the top three PCA components of the patch tokens to RGB and writes both PCA-only and input\u002FPCA panel visualizations to the output directory. Use `--dtype fp32 --device cpu` for CPU-only inference.\n\n\u003Cdetails>\n\u003Csummary>\u003Cb>All demo options\u003C\u002Fb>\u003C\u002Fsummary>\n\n| Parameter | Description |\n| --- | --- |\n| `--config-file` | Model config file under `lingbot_vision\u002Fconfigs\u002F`. |\n| `--ckpt` | Local path to a pure backbone `.pt` checkpoint. |\n| `--input` | Image file or directory of images. |\n| `--out` | Output directory for PCA visualizations. |\n| `--size` | Target input size. For ViT-g\u002F16, 512 gives a 32 x 32 patch grid. |\n| `--mode` | `square` resizes to `size` x `size` (does not preserve aspect ratio); `shortest` resizes the shortest side to `size`, then center-crops a `size` x `size` square. |\n| `--dtype` | `bf16`, `fp16`, or `fp32`. |\n| `--device` | PyTorch device, for example `cuda` or `cpu`. |\n\n\u003C\u002Fdetails>\n\n### Checkpoint format\n\nReleased checkpoints are `.pt` files containing backbone weights only — no optimizer states, projection heads, or training-time boundary heads. The loader accepts a raw state dict or a dictionary with a `backbone` entry:\n\n```python\nstate_dict\n{\"backbone\": state_dict}\n```\n\nIf checkpoint keys are prefixed with `backbone.`, the loader strips the prefix automatically.\n\n## 📖 Citation\n\n```bibtex\n@article{lingbot-vision2026,\n  title={Vision Pretraining for Dense Spatial Perception},\n  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},\n  journal={arXiv preprint arXiv:2607.05247},\n  year={2026}\n}\n```\n\n## 📜 License\n\nThis project is released under the Apache License 2.0. See [LICENSE](LICENSE) for details.\n\n## 🙏 Acknowledgments\n\nLingBot-Vision is part of the LingBot spatial perception effort. We thank [DINOv2](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdinov2) and [DINOv3](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fdinov3) for their contributions to self-supervised learning.\n\n## 📮 Contact\n\nFor questions, discussions, or collaborations: \n\n- **Issues**: Open an [issue](https:\u002F\u002Fgithub.com\u002Frobbyant\u002Flingbot-vision\u002Fissues) on GitHub\n- **Email**: Contact [Zelin Fu](https:\u002F\u002Fgithub.com\u002FTakuLingFu) (fuzelin.fzl@antgroup.com) or [Nan Xue](https:\u002F\u002Fxuenan.net) (xuenan@ieee.org)\n","LingBot-Vision 是一个面向密集空间感知的自监督视觉预训练模型家族，基于 Vision Transformer 架构，核心采用边界中心化的掩码建模（masked boundary modeling）目标，使模型在学习语义表征的同时显式建模物体边界、几何结构与空间关系。支持从 ViT-S\u002F16 到 1.1B 参数 ViT-g\u002F16 多种规模，可作为即插即用的视觉编码器，适用于深度估计、语义分割、视频目标分割、深度补全等需像素级空间理解的下游任务。项目开源、轻量适配，已在 RGB-D 等多模态密集预测任务中验证有效性。",2,"2026-07-08 04:30:00","CREATED_QUERY"]