[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-2259":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":25,"hasPages":23,"topics":26,"createdAt":10,"pushedAt":10,"updatedAt":40,"readmeContent":41,"aiSummary":42,"trendingCount":16,"starSnapshotCount":16,"syncStatus":43,"lastSyncTime":44,"discoverSource":45},2259,"LLaVA","haotian-liu\u002FLLaVA","haotian-liu","[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.","https:\u002F\u002Fllava.hliu.cc",null,"Python",24867,2773,156,1098,0,3,14,89,10,45,"Apache License 2.0",false,"main",true,[27,28,29,30,31,32,33,34,35,36,37,38,39],"chatbot","chatgpt","foundation-models","gpt-4","instruction-tuning","llama","llama-2","llama2","llava","multi-modality","multimodal","vision-language-model","visual-language-learning","2026-06-12 02:00:39","# 🌋 LLaVA: Large Language and Vision Assistant\n\n*Visual instruction tuning towards large language and vision models with GPT-4 level capabilities.*\n\n[📢 [LLaVA-NeXT Blog](https:\u002F\u002Fllava-vl.github.io\u002Fblog\u002F2024-01-30-llava-next\u002F)] [[Project Page](https:\u002F\u002Fllava-vl.github.io\u002F)] [[Demo](https:\u002F\u002Fllava.hliu.cc\u002F)]  [[Data](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FData.md)] [[Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md)]\n\n🤝Community Contributions: [[llama.cpp](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp\u002Fpull\u002F3436)] [[Colab](https:\u002F\u002Fgithub.com\u002Fcamenduru\u002FLLaVA-colab)] [[🤗Space](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fbadayvedat\u002FLLaVA)] [[Replicate](https:\u002F\u002Freplicate.com\u002Fyorickvp\u002Fllava-13b)] [[AutoGen](https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fautogen\u002Fblob\u002Fmain\u002Fnotebook\u002Fagentchat_lmm_llava.ipynb)]  [[BakLLaVA](https:\u002F\u002Fgithub.com\u002FSkunkworksAI\u002FBakLLaVA)]\n\n**Improved Baselines with Visual Instruction Tuning** [[Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.03744)] [[HF](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2310.03744)] \u003Cbr>\n[Haotian Liu](https:\u002F\u002Fhliu.cc), [Chunyuan Li](https:\u002F\u002Fchunyuan.li\u002F), [Yuheng Li](https:\u002F\u002Fyuheng-li.github.io\u002F), [Yong Jae Lee](https:\u002F\u002Fpages.cs.wisc.edu\u002F~yongjaelee\u002F)\n\n**Visual Instruction Tuning** (NeurIPS 2023, **Oral**) [[Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2304.08485)] [[HF](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2304.08485)] \u003Cbr>\n[Haotian Liu*](https:\u002F\u002Fhliu.cc), [Chunyuan Li*](https:\u002F\u002Fchunyuan.li\u002F), [Qingyang Wu](https:\u002F\u002Fscholar.google.ca\u002Fcitations?user=HDiw-TsAAAAJ&hl=en\u002F), [Yong Jae Lee](https:\u002F\u002Fpages.cs.wisc.edu\u002F~yongjaelee\u002F) (*Equal Contribution)\n\n\u003C!--p align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fllava.hliu.cc\u002F\">\u003Cimg src=\"images\u002Fllava_logo.png\" width=\"50%\">\u003C\u002Fa> \u003Cbr>\n    Generated by \u003Ca href=\"https:\u002F\u002Fgligen.github.io\u002F\">GLIGEN\u003C\u002Fa> via \"a cute lava llama with glasses\" and box prompt\n\u003C\u002Fp-->\n\n\n## Release\n\n- [2024\u002F05\u002F10] 🔥 **LLaVA-NeXT** (Stronger) models are released, stronger LMM with support of LLama-3 (8B) and Qwen-1.5 (72B\u002F110B). [[Blog](https:\u002F\u002Fllava-vl.github.io\u002Fblog\u002F2024-05-10-llava-next-stronger-llms\u002F)] [[Checkpoints](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Flmms-lab\u002Fllava-next-6623288e2d61edba3ddbf5ff)] [[Demo](https:\u002F\u002Fllava-next.lmms-lab.com\u002F)] [[Code](https:\u002F\u002Fgithub.com\u002FLLaVA-VL\u002FLLaVA-NeXT\u002F)] \n- [2024\u002F05\u002F10] 🔥 **LLaVA-NeXT** (Video) is released. The image-only-trained LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer. DPO training with AI feedback on videos can yield significant improvement. [[Blog](https:\u002F\u002Fllava-vl.github.io\u002Fblog\u002F2024-04-30-llava-next-video\u002F)] [[Checkpoints](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002Flmms-lab\u002Fllava-next-video-661e86f5e8dabc3ff793c944)] [[Code](https:\u002F\u002Fgithub.com\u002FLLaVA-VL\u002FLLaVA-NeXT\u002F)]\n- [03\u002F10] Releasing **LMMs-Eval**, a highly efficient evaluation pipeline we used when developing LLaVA-NeXT. It supports the evaluation of LMMs on dozens of public datasets and allows new dataset onboarding, making the dev of new LMMs much faster. [[Blog](https:\u002F\u002Flmms-lab.github.io\u002Flmms-eval-blog\u002Flmms-eval-0.1\u002F)] [[Codebase](https:\u002F\u002Fgithub.com\u002FEvolvingLMMs-Lab\u002Flmms-eval)]\n- [1\u002F30] 🔥 **LLaVA-NeXT** (LLaVA-1.6) is out! With additional scaling to LLaVA-1.5, LLaVA-NeXT-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks\u002Fapplications than before. Check out the [blog post](https:\u002F\u002Fllava-vl.github.io\u002Fblog\u002F2024-01-30-llava-next\u002F), and explore the [demo](https:\u002F\u002Fllava.hliu.cc\u002F)! Models are available in [Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md). Training\u002Feval data and scripts coming soon.\n- [11\u002F10] [LLaVA-Plus](https:\u002F\u002Fllava-vl.github.io\u002Fllava-plus\u002F) is released: Learning to Use Tools for Creating Multimodal Agents, with LLaVA-Plus (LLaVA that Plug and Learn to Use Skills). [[Project Page](https:\u002F\u002Fllava-vl.github.io\u002Fllava-plus\u002F)] [[Demo](https:\u002F\u002Fllavaplus.ngrok.io\u002F)] [[Code](https:\u002F\u002Fgithub.com\u002FLLaVA-VL\u002FLLaVA-Plus-Codebase)] [[Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.05437)]\n- [11\u002F2] [LLaVA-Interactive](https:\u002F\u002Fllava-vl.github.io\u002Fllava-interactive\u002F) is released: Experience the future of human-AI multimodal interaction with an all-in-one demo for Image Chat, Segmentation, Generation and Editing. [[Project Page](https:\u002F\u002Fllava-vl.github.io\u002Fllava-interactive\u002F)] [[Demo](https:\u002F\u002Fllavainteractive.ngrok.io\u002F)] [[Code](https:\u002F\u002Fgithub.com\u002FLLaVA-VL\u002FLLaVA-Interactive-Demo)] [[Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.00571)]\n- [10\u002F26] 🔥 LLaVA-1.5 with LoRA achieves comparable performance as full-model finetuning, with a reduced GPU RAM requirement ([ckpts](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md#llava-v15), [script](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA#train)). We also provide a [doc](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FFinetune_Custom_Data.md) on how to finetune LLaVA-1.5 on your own dataset with LoRA.\n- [10\u002F12] Check out the Korean LLaVA (Ko-LLaVA), created by ETRI, who has generously supported our research! [[🤗 Demo](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fetri-vilab\u002FKo-LLaVA)]\n- [10\u002F5] 🔥 LLaVA-1.5 is out! Achieving SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods like Qwen-VL-Chat that use billion-scale data. Check out the [technical report](https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.03744), and explore the [demo](https:\u002F\u002Fllava.hliu.cc\u002F)! Models are available in [Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md). The training data and scripts of LLaVA-1.5 are released [here](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA#train), and evaluation scripts are released [here](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FEvaluation.md)!\n- [9\u002F26] LLaVA is improved with reinforcement learning from human feedback (RLHF) to improve fact grounding and reduce hallucination. Check out the new SFT and RLHF checkpoints at project [[LLavA-RLHF]](https:\u002F\u002Fllava-rlhf.github.io\u002F)\n- [9\u002F22] [LLaVA](https:\u002F\u002Farxiv.org\u002Fabs\u002F2304.08485) is accepted by NeurIPS 2023 as **oral presentation**, and [LLaVA-Med](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.00890) is accepted by NeurIPS 2023 Datasets and Benchmarks Track as **spotlight presentation**.\n\n\u003Cdetails>\n\u003Csummary>More\u003C\u002Fsummary>\n\n- [11\u002F6] Support **Intel** dGPU and CPU platforms. [More details here.](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Ftree\u002Fintel\u002Fdocs\u002Fintel)\n- [10\u002F12] LLaVA is now supported in [llama.cpp](https:\u002F\u002Fgithub.com\u002Fggerganov\u002Fllama.cpp\u002Fpull\u002F3436) with 4-bit \u002F 5-bit quantization support!\n- [10\u002F11] The training data and scripts of LLaVA-1.5 are released [here](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA#train), and evaluation scripts are released [here](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FEvaluation.md)!\n- [10\u002F10] [Roboflow Deep Dive](https:\u002F\u002Fblog.roboflow.com\u002Ffirst-impressions-with-llava-1-5\u002F): First Impressions with LLaVA-1.5.\n- [9\u002F20] We summarize our empirical study of training 33B and 65B LLaVA models in a [note](https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.09958). Further, if you are interested in the comprehensive review, evolution and trend of multimodal foundation models, please check out our recent survey paper [``Multimodal Foundation Models: From Specialists to General-Purpose Assistants''.](https:\u002F\u002Farxiv.org\u002Fabs\u002F2309.10020)\n\u003Cp align=\"center\">\n  \u003Cimg src=\"https:\u002F\u002Fgithub.com\u002FComputer-Vision-in-the-Wild\u002FCVinW_Readings\u002Fblob\u002Fmain\u002Fimages\u002Fmfm_evolution.jpeg?raw=true\" width=50%\u002F>\n\u003C\u002Fp>\n\n- [7\u002F19] 🔥 We release a major upgrade, including support for LLaMA-2, LoRA training, 4-\u002F8-bit inference, higher resolution (336x336), and a lot more. We release [LLaVA Bench](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FLLaVA_Bench.md) for benchmarking open-ended visual chat with results from Bard and Bing-Chat. We also support and verify training with RTX 3090 and RTX A6000. Check out [LLaVA-from-LLaMA-2](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FLLaVA_from_LLaMA2.md), and our [model zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md)!\n- [6\u002F26] [CVPR 2023 Tutorial](https:\u002F\u002Fvlp-tutorial.github.io\u002F) on **Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4**!  Please check out [[Slides](https:\u002F\u002Fdatarelease.blob.core.windows.net\u002Ftutorial\u002Fvision_foundation_models_2023\u002Fslides\u002FChunyuan_cvpr2023_tutorial_lmm.pdf)] [[Notes](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.14895)] [[YouTube](https:\u002F\u002Fyoutu.be\u002FmkI7EPD1vp8)] [[Bilibli](https:\u002F\u002Fwww.bilibili.com\u002Fvideo\u002FBV1Ng4y1T7v3\u002F)].\n- [6\u002F11] We released the preview for the most requested feature: DeepSpeed and LoRA support!  Please see documentations [here](.\u002Fdocs\u002FLoRA.md).\n- [6\u002F1] We released **LLaVA-Med: Large Language and Vision Assistant for Biomedicine**, a step towards building biomedical domain large language and vision models with GPT-4 level capabilities.  Checkout the [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.00890) and [page](https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLLaVA-Med).\n- [5\u002F6] We are releasing [LLaVA-Lighting-MPT-7B-preview](https:\u002F\u002Fhuggingface.co\u002Fliuhaotian\u002FLLaVA-Lightning-MPT-7B-preview), based on MPT-7B-Chat!  See [here](#LLaVA-MPT-7b) for more details.\n- [5\u002F2] 🔥 We are releasing LLaVA-Lighting!  Train a lite, multimodal GPT-4 with just $40 in 3 hours!  See [here](#train-llava-lightning) for more details.\n- [4\u002F27] Thanks to the community effort, LLaVA-13B with 4-bit quantization allows you to run on a GPU with as few as 12GB VRAM!  Try it out [here](https:\u002F\u002Fgithub.com\u002Foobabooga\u002Ftext-generation-webui\u002Ftree\u002Fmain\u002Fextensions\u002Fllava).\n- [4\u002F17] 🔥 We released **LLaVA: Large Language and Vision Assistant**. We propose visual instruction tuning, towards building large language and vision models with GPT-4 level capabilities.  Checkout the [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2304.08485) and [demo](https:\u002F\u002Fllava.hliu.cc\u002F).\n\n\u003C\u002Fdetails>\n\n\u003C!-- \u003Ca href=\"https:\u002F\u002Fllava.hliu.cc\u002F\">\u003Cimg src=\"assets\u002Fdemo.gif\" width=\"70%\">\u003C\u002Fa> -->\n\n[![Code License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode%20License-Apache_2.0-green.svg)](https:\u002F\u002Fgithub.com\u002Ftatsu-lab\u002Fstanford_alpaca\u002Fblob\u002Fmain\u002FLICENSE)\n**Usage and License Notices**: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the [OpenAI Terms of Use](https:\u002F\u002Fopenai.com\u002Fpolicies\u002Fterms-of-use) for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. [Llama community license](https:\u002F\u002Fai.meta.com\u002Fllama\u002Flicense\u002F) for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.\n\n\n## Contents\n- [Install](#install)\n- [LLaVA Weights](#llava-weights)\n- [Demo](#Demo)\n- [Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md)\n- [Dataset](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FData.md)\n- [Train](#train)\n- [Evaluation](#evaluation)\n\n## Install\n\nIf you are not using Linux, do *NOT* proceed, see instructions for [macOS](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FmacOS.md) and [Windows](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FWindows.md).\n\n1. Clone this repository and navigate to LLaVA folder\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA.git\ncd LLaVA\n```\n\n2. Install Package\n```Shell\nconda create -n llava python=3.10 -y\nconda activate llava\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n\n3. Install additional packages for training cases\n```\npip install -e \".[train]\"\npip install flash-attn --no-build-isolation\n```\n\n### Upgrade to latest code base\n\n```Shell\ngit pull\npip install -e .\n\n# if you see some import errors when you upgrade,\n# please try running the command below (without #)\n# pip install flash-attn --no-build-isolation --no-cache-dir\n```\n\n### Quick Start With HuggingFace\n\n\u003Cdetails>\n\u003Csummary>Example Code\u003C\u002Fsummary>\n\n```Python\nfrom llava.model.builder import load_pretrained_model\nfrom llava.mm_utils import get_model_name_from_path\nfrom llava.eval.run_llava import eval_model\n\nmodel_path = \"liuhaotian\u002Fllava-v1.5-7b\"\n\ntokenizer, model, image_processor, context_len = load_pretrained_model(\n    model_path=model_path,\n    model_base=None,\n    model_name=get_model_name_from_path(model_path)\n)\n```\n\nCheck out the details wth the `load_pretrained_model` function in `llava\u002Fmodel\u002Fbuilder.py`.\n\nYou can also use the `eval_model` function in `llava\u002Feval\u002Frun_llava.py` to get the output easily. By doing so, you can use this code on Colab directly after downloading this repository.\n\n``` python\nmodel_path = \"liuhaotian\u002Fllava-v1.5-7b\"\nprompt = \"What are the things I should be cautious about when I visit here?\"\nimage_file = \"https:\u002F\u002Fllava-vl.github.io\u002Fstatic\u002Fimages\u002Fview.jpg\"\n\nargs = type('Args', (), {\n    \"model_path\": model_path,\n    \"model_base\": None,\n    \"model_name\": get_model_name_from_path(model_path),\n    \"query\": prompt,\n    \"conv_mode\": None,\n    \"image_file\": image_file,\n    \"sep\": \",\",\n    \"temperature\": 0,\n    \"top_p\": None,\n    \"num_beams\": 1,\n    \"max_new_tokens\": 512\n})()\n\neval_model(args)\n```\n\u003C\u002Fdetails>\n\n## LLaVA Weights\nPlease check out our [Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md) for all public LLaVA checkpoints, and the instructions of how to use the weights.\n\n## Demo\n\n### Gradio Web UI\n\nTo launch a Gradio demo locally, please run the following commands one by one. If you plan to launch multiple model workers to compare between different checkpoints, you only need to launch the controller and the web server *ONCE*.\n\n```mermaid\nflowchart BT\n    %% Declare Nodes\n    gws(\"Gradio (UI Server)\")\n    c(\"Controller (API Server):\u003Cbr\u002F>PORT: 10000\")\n    mw7b(\"Model Worker:\u003Cbr\u002F>llava-v1.5-7b\u003Cbr\u002F>PORT: 40000\")\n    mw13b(\"Model Worker:\u003Cbr\u002F>llava-v1.5-13b\u003Cbr\u002F>PORT: 40001\")\n    sglw13b(\"SGLang Backend:\u003Cbr\u002F>llava-v1.6-34b\u003Cbr\u002F>http:\u002F\u002Flocalhost:30000\")\n    lsglw13b(\"SGLang Worker:\u003Cbr\u002F>llava-v1.6-34b\u003Cbr\u002F>PORT: 40002\")\n\n    %% Declare Styles\n    classDef data fill:#3af,stroke:#48a,stroke-width:2px,color:#444\n    classDef success fill:#8f8,stroke:#0a0,stroke-width:2px,color:#444\n    classDef failure fill:#f88,stroke:#f00,stroke-width:2px,color:#444\n\n    %% Assign Styles\n    class id,od data;\n    class cimg,cs_s,scsim_s success;\n    class ncimg,cs_f,scsim_f failure;\n\n    subgraph Demo Connections\n        direction BT\n        c\u003C-->gws\n        \n        mw7b\u003C-->c\n        mw13b\u003C-->c\n        lsglw13b\u003C-->c\n        sglw13b\u003C-->lsglw13b\n    end\n```\n\n#### Launch a controller\n```Shell\npython -m llava.serve.controller --host 0.0.0.0 --port 10000\n```\n\n#### Launch a gradio web server.\n```Shell\npython -m llava.serve.gradio_web_server --controller http:\u002F\u002Flocalhost:10000 --model-list-mode reload\n```\nYou just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen. You may notice that there is no model in the model list. Do not worry, as we have not launched any model worker yet. It will be automatically updated when you launch a model worker.\n\n#### Launch a SGLang worker\n\nThis is the recommended way to serve LLaVA model with high throughput, and you need to install SGLang first. Note that currently `4-bit` quantization is not supported yet on SGLang-LLaVA, and if you have limited GPU VRAM, please check out model worker with [quantization](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA?tab=readme-ov-file#launch-a-model-worker-4-bit-8-bit-inference-quantized).\n\n```Shell\npip install \"sglang[all]\"\n```\n\nYou'll first launch a SGLang backend worker which will execute the models on GPUs. Remember the `--port` you've set and you'll use that later.\n\n```Shell\n# Single GPU\nCUDA_VISIBLE_DEVICES=0 python3 -m sglang.launch_server --model-path liuhaotian\u002Fllava-v1.5-7b --tokenizer-path llava-hf\u002Fllava-1.5-7b-hf --port 30000\n\n# Multiple GPUs with tensor parallel\nCUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server --model-path liuhaotian\u002Fllava-v1.5-13b --tokenizer-path llava-hf\u002Fllava-1.5-13b-hf --port 30000 --tp 2\n```\n\nTokenizers (temporary): `llava-hf\u002Fllava-1.5-7b-hf`, `llava-hf\u002Fllava-1.5-13b-hf`, `liuhaotian\u002Fllava-v1.6-34b-tokenizer`.\n\nYou'll then launch a LLaVA-SGLang worker that will communicate between LLaVA controller and SGLang backend to route the requests. Set `--sgl-endpoint` to `http:\u002F\u002F127.0.0.1:port` where `port` is the one you just set (default: 30000).\n\n```Shell\npython -m llava.serve.sglang_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --sgl-endpoint http:\u002F\u002F127.0.0.1:30000\n```\n\n#### Launch a model worker\n\nThis is the actual *worker* that performs the inference on the GPU.  Each worker is responsible for a single model specified in `--model-path`.\n\n```Shell\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path liuhaotian\u002Fllava-v1.5-13b\n```\nWait until the process finishes loading the model and you see \"Uvicorn running on ...\".  Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.\n\nYou can launch as many workers as you want, and compare between different model checkpoints in the same Gradio interface. Please keep the `--controller` the same, and modify the `--port` and `--worker` to a different port number for each worker.\n```Shell\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port \u003Cdifferent from 40000, say 40001> --worker http:\u002F\u002Flocalhost:\u003Cchange accordingly, i.e. 40001> --model-path \u003Cckpt2>\n```\n\nIf you are using an Apple device with an M1 or M2 chip, you can specify the mps device by using the `--device` flag: `--device mps`.\n\n#### Launch a model worker (Multiple GPUs, when GPU VRAM \u003C= 24GB)\n\nIf the VRAM of your GPU is less than 24GB (e.g., RTX 3090, RTX 4090, etc.), you may try running it with multiple GPUs. Our latest code base will automatically try to use multiple GPUs if you have more than one GPU. You can specify which GPUs to use with `CUDA_VISIBLE_DEVICES`. Below is an example of running with the first two GPUs.\n\n```Shell\nCUDA_VISIBLE_DEVICES=0,1 python -m llava.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path liuhaotian\u002Fllava-v1.5-13b\n```\n\n#### Launch a model worker (4-bit, 8-bit inference, quantized)\n\nYou can launch the model worker with quantized bits (4-bit, 8-bit), which allows you to run the inference with reduced GPU memory footprint, potentially allowing you to run on a GPU with as few as 12GB VRAM. Note that inference with quantized bits may not be as accurate as the full-precision model. Simply append `--load-4bit` or `--load-8bit` to the **model worker** command that you are executing. Below is an example of running with 4-bit quantization.\n\n```Shell\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path liuhaotian\u002Fllava-v1.5-13b --load-4bit\n```\n\n#### Launch a model worker (LoRA weights, unmerged)\n\nYou can launch the model worker with LoRA weights, without merging them with the base checkpoint, to save disk space. There will be additional loading time, while the inference speed is the same as the merged checkpoints. Unmerged LoRA checkpoints do not have `lora-merge` in the model name, and are usually much smaller (less than 1GB) than the merged checkpoints (13G for 7B, and 25G for 13B).\n\nTo load unmerged LoRA weights, you simply need to pass an additional argument `--model-base`, which is the base LLM that is used to train the LoRA weights. You can check the base LLM of each LoRA weights in the [model zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md).\n\n```Shell\npython -m llava.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path liuhaotian\u002Fllava-v1-0719-336px-lora-vicuna-13b-v1.3 --model-base lmsys\u002Fvicuna-13b-v1.3\n```\n\n### CLI Inference\n\nChat about images using LLaVA without the need of Gradio interface. It also supports multiple GPUs, 4-bit and 8-bit quantized inference. With 4-bit quantization, for our LLaVA-1.5-7B, it uses less than 8GB VRAM on a single GPU.\n\n```Shell\npython -m llava.serve.cli \\\n    --model-path liuhaotian\u002Fllava-v1.5-7b \\\n    --image-file \"https:\u002F\u002Fllava-vl.github.io\u002Fstatic\u002Fimages\u002Fview.jpg\" \\\n    --load-4bit\n```\n\n\u003Cimg src=\"images\u002Fdemo_cli.gif\" width=\"70%\">\n\n## Train\n\n*Below is the latest training configuration for LLaVA v1.5. For legacy models, please refer to README of [this](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Ftree\u002Fv1.0.1) version for now. We'll add them in a separate doc later.*\n\nLLaVA training consists of two stages: (1) feature alignment stage: use our 558K subset of the LAION-CC-SBU dataset to connect a *frozen pretrained* vision encoder to a *frozen LLM*; (2) visual instruction tuning stage: use 150K GPT-generated multimodal instruction-following data, plus around 515K VQA data from academic-oriented tasks, to teach the model to follow multimodal instructions.\n\nLLaVA is trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the `per_device_train_batch_size` and increase the `gradient_accumulation_steps` accordingly. Always keep the global batch size the same: `per_device_train_batch_size` x `gradient_accumulation_steps` x `num_gpus`.\n\n### Hyperparameters\nWe use a similar set of hyperparameters as Vicuna in finetuning.  Both hyperparameters used in pretraining and finetuning are provided below.\n\n1. Pretraining\n\n| Hyperparameter | Global Batch Size | Learning rate | Epochs | Max length | Weight decay |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| LLaVA-v1.5-13B | 256 | 1e-3 | 1 | 2048 | 0 |\n\n2. Finetuning\n\n| Hyperparameter | Global Batch Size | Learning rate | Epochs | Max length | Weight decay |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| LLaVA-v1.5-13B | 128 | 2e-5 | 1 | 2048 | 0 |\n\n### Download Vicuna checkpoints (automatically)\n\nOur base model Vicuna v1.5, which is an instruction-tuned chatbot, will be downloaded automatically when you run our provided training scripts. No action is needed.\n\n### Pretrain (feature alignment)\n\nPlease download the 558K subset of the LAION-CC-SBU dataset with BLIP captions we use in the paper [here](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fliuhaotian\u002FLLaVA-Pretrain).\n\nPretrain takes around 5.5 hours for LLaVA-v1.5-13B on 8x A100 (80G), due to the increased resolution to 336px. It takes around 3.5 hours for LLaVA-v1.5-7B.\n\nTraining script with DeepSpeed ZeRO-2: [`pretrain.sh`](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fscripts\u002Fv1_5\u002Fpretrain.sh).\n\n- `--mm_projector_type mlp2x_gelu`: the two-layer MLP vision-language connector.\n- `--vision_tower openai\u002Fclip-vit-large-patch14-336`: CLIP ViT-L\u002F14 336px.\n\n\u003Cdetails>\n\u003Csummary>Pretrain takes around 20 hours for LLaVA-7B on 8x V100 (32G)\u003C\u002Fsummary>\n\n We provide training script with DeepSpeed [here](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fscripts\u002Fpretrain_xformers.sh).\nTips:\n- If you are using V100 which is not supported by FlashAttention, you can use the [memory-efficient attention](https:\u002F\u002Farxiv.org\u002Fabs\u002F2112.05682) implemented in [xFormers](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fxformers). Install xformers and replace `llava\u002Ftrain\u002Ftrain_mem.py` above with [llava\u002Ftrain\u002Ftrain_xformers.py](llava\u002Ftrain\u002Ftrain_xformers.py).\n\u003C\u002Fdetails>\n\n### Visual Instruction Tuning\n\n1. Prepare data\n\nPlease download the annotation of the final mixture our instruction tuning data [llava_v1_5_mix665k.json](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fliuhaotian\u002FLLaVA-Instruct-150K\u002Fblob\u002Fmain\u002Fllava_v1_5_mix665k.json), and download the images from constituting datasets:\n\n- COCO: [train2017](http:\u002F\u002Fimages.cocodataset.org\u002Fzips\u002Ftrain2017.zip)\n- GQA: [images](https:\u002F\u002Fdownloads.cs.stanford.edu\u002Fnlp\u002Fdata\u002Fgqa\u002Fimages.zip)\n- OCR-VQA: [download script](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing), **we save all files as `.jpg`**\n- TextVQA: [train_val_images](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Ftextvqa\u002Fimages\u002Ftrain_val_images.zip)\n- VisualGenome: [part1](https:\u002F\u002Fcs.stanford.edu\u002Fpeople\u002Frak248\u002FVG_100K_2\u002Fimages.zip), [part2](https:\u002F\u002Fcs.stanford.edu\u002Fpeople\u002Frak248\u002FVG_100K_2\u002Fimages2.zip)\n\nAfter downloading all of them, organize the data as follows in `.\u002Fplayground\u002Fdata`,\n\n```\n├── coco\n│   └── train2017\n├── gqa\n│   └── images\n├── ocr_vqa\n│   └── images\n├── textvqa\n│   └── train_images\n└── vg\n    ├── VG_100K\n    └── VG_100K_2\n```\n\n2. Start training!\n\nYou may download our pretrained projectors in [Model Zoo](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FMODEL_ZOO.md). It is not recommended to use legacy projectors, as they may be trained with a different version of the codebase, and if any option is off, the model will not function\u002Ftrain as we expected.\n\nVisual instruction tuning takes around 20 hours for LLaVA-v1.5-13B on 8x A100 (80G), due to the increased resolution to 336px. It takes around 10 hours for LLaVA-v1.5-7B on 8x A100 (40G).\n\nTraining script with DeepSpeed ZeRO-3: [`finetune.sh`](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fscripts\u002Fv1_5\u002Ffinetune.sh).\n\nIf you are do not have enough GPU memory:\n\n- Use LoRA: [`finetune_lora.sh`](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fscripts\u002Fv1_5\u002Ffinetune_lora.sh). We are able to fit 13B training in 8-A100-40G\u002F8-A6000, and 7B training in 8-RTX3090. Make sure `per_device_train_batch_size*gradient_accumulation_steps` is the same as the provided script for best reproducibility.\n- Replace `zero3.json` with `zero3_offload.json` which offloads some parameters to CPU RAM. This slows down the training speed.\n\nIf you are interested in finetuning LLaVA model to your own task\u002Fdata, please check out [`Finetune_Custom_Data.md`](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FFinetune_Custom_Data.md)。\n\nNew options to note:\n\n- `--mm_projector_type mlp2x_gelu`: the two-layer MLP vision-language connector.\n- `--vision_tower openai\u002Fclip-vit-large-patch14-336`: CLIP ViT-L\u002F14 336px.\n- `--image_aspect_ratio pad`: this pads the non-square images to square, instead of cropping them; it slightly reduces hallucination.\n- `--group_by_modality_length True`: this should only be used when your instruction tuning dataset contains both language (e.g. ShareGPT) and multimodal (e.g. LLaVA-Instruct). It makes the training sampler only sample a single modality (either image or language) during training, which we observe to speed up training by ~25%, and does not affect the final outcome.\n\n## Evaluation\n\nIn LLaVA-1.5, we evaluate models on a diverse set of 12 benchmarks. To ensure the reproducibility, we evaluate the models with greedy decoding. We do not evaluate using beam search to make the inference process consistent with the chat demo of real-time outputs.\n\nSee [Evaluation.md](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FEvaluation.md).\n\n### GPT-assisted Evaluation\n\nOur GPT-assisted evaluation pipeline for multimodal modeling is provided for a comprehensive understanding of the capabilities of vision-language models.  Please see our paper for more details.\n\n1. Generate LLaVA responses\n\n```Shell\npython model_vqa.py \\\n    --model-path .\u002Fcheckpoints\u002FLLaVA-13B-v0 \\\n    --question-file \\\n    playground\u002Fdata\u002Fcoco2014_val_qa_eval\u002Fqa90_questions.jsonl \\\n    --image-folder \\\n    \u002Fpath\u002Fto\u002Fcoco2014_val \\\n    --answers-file \\\n    \u002Fpath\u002Fto\u002Fanswer-file-our.jsonl\n```\n\n2. Evaluate the generated responses.  In our case, [`answer-file-ref.jsonl`](.\u002Fplayground\u002Fdata\u002Fcoco2014_val_qa_eval\u002Fqa90_gpt4_answer.jsonl) is the response generated by text-only GPT-4 (0314), with the context captions\u002Fboxes provided.\n\n```Shell\nOPENAI_API_KEY=\"sk-***********************************\" python llava\u002Feval\u002Feval_gpt_review_visual.py \\\n    --question playground\u002Fdata\u002Fcoco2014_val_qa_eval\u002Fqa90_questions.jsonl \\\n    --context llava\u002Feval\u002Ftable\u002Fcaps_boxes_coco2014_val_80.jsonl \\\n    --answer-list \\\n    \u002Fpath\u002Fto\u002Fanswer-file-ref.jsonl \\\n    \u002Fpath\u002Fto\u002Fanswer-file-our.jsonl \\\n    --rule llava\u002Feval\u002Ftable\u002Frule.json \\\n    --output \u002Fpath\u002Fto\u002Freview.json\n```\n\n3. Summarize the evaluation results\n\n```Shell\npython summarize_gpt_review.py\n```\n\n## Citation\n\nIf you find LLaVA useful for your research and applications, please cite using this BibTeX:\n```bibtex\n@misc{liu2024llavanext,\n    title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},\n    url={https:\u002F\u002Fllava-vl.github.io\u002Fblog\u002F2024-01-30-llava-next\u002F},\n    author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae},\n    month={January},\n    year={2024}\n}\n\n@misc{liu2023improvedllava,\n      title={Improved Baselines with Visual Instruction Tuning}, \n      author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},\n      publisher={arXiv:2310.03744},\n      year={2023},\n}\n\n@misc{liu2023llava,\n      title={Visual Instruction Tuning}, \n      author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},\n      publisher={NeurIPS},\n      year={2023},\n}\n```\n\n## Acknowledgement\n\n- [Vicuna](https:\u002F\u002Fgithub.com\u002Flm-sys\u002FFastChat): the codebase we built upon, and our base model Vicuna-13B that has the amazing language capabilities!\n\n## Related Projects\n\n- [Instruction Tuning with GPT-4](https:\u002F\u002Fgithub.com\u002FInstruction-Tuning-with-GPT-4\u002FGPT-4-LLM)\n- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLLaVA-Med)\n- [Otter: In-Context Multi-Modal Instruction Tuning](https:\u002F\u002Fgithub.com\u002FLuodian\u002FOtter)\n\nFor future project ideas, please check out:\n- [SEEM: Segment Everything Everywhere All at Once](https:\u002F\u002Fgithub.com\u002FUX-Decoder\u002FSegment-Everything-Everywhere-All-At-Once)\n- [Grounded-Segment-Anything](https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGrounded-Segment-Anything) to detect, segment, and generate anything by marrying [Grounding DINO](https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO) and [Segment-Anything](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything).\n","LLaVA 是一个视觉指令微调项目，旨在开发具备GPT-4级别能力的大型语言和视觉模型。其核心功能包括通过多模态学习来增强模型对图像和文本的理解与生成能力，支持基于LLaMA及其变体的预训练模型进行进一步优化。该项目特别适用于需要处理复杂视觉信息并结合自然语言理解的应用场景，如图像描述、视觉问答系统等。此外，LLaVA还提供了丰富的社区贡献资源，包括Colab笔记本、Hugging Face空间及多种部署方式，方便开发者快速上手和应用。",2,"2026-06-11 02:49:08","top_language"]