[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72306":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":16,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":18,"rankGlobal":10,"rankLanguage":10,"license":19,"archived":20,"fork":20,"defaultBranch":21,"hasWiki":20,"hasPages":20,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":16,"starSnapshotCount":16,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},72306,"MGM","JIA-Lab-research\u002FMGM","JIA-Lab-research","Official repo for \"Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models\"","",null,"Python",3325,275,26,60,0,1,59.42,"Apache License 2.0",false,"main",[23,24,25],"generation","large-language-models","vision-language-model","2026-06-12 04:01:04","# Official repo for \"Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models\"\n\n\u003Ca href='https:\u002F\u002Fmini-gemini.github.io\u002F'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-Green'>\u003C\u002Fa>\n\u003Ca href='http:\u002F\u002F103.170.5.190:7860\u002F'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Demo-violet'>\u003C\u002Fa>\n\u003Ca href='https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fwcy1122\u002FMGM'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗-Open%20In%20Spaces-blue.svg'>\u003C\u002Fa>\n\u003Ca href='https:\u002F\u002Farxiv.org\u002Fpdf\u002F2403.18814.pdf'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-Arxiv-red'>\u003C\u002Fa>\n\u003Ca href='https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FYanweiLi\u002Fmgm-6603c50b9b43d044171d0854'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Hugging%20Face-Models-blue'>\u003C\u002Fa>\n\u003Ca href='https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FYanweiLi\u002Fmgm-data-660463ea895a01d8f367624e'>\u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Hugging%20Face-Data-green'>\u003C\u002Fa>\n\n\nThe framework supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B with image understanding, reasoning, and generation simultaneously. We build this repo based on LLaVA.\n\n## Release\n- [05\u002F03] 🔥 We support LLaMA3-based models! Welcome to try them [here](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FYanweiLi\u002Fmgm-6603c50b9b43d044171d0854).\n- [04\u002F15] 🔥 The [Hugging Face demo](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fwcy1122\u002FMGM) is available. It's a 13B-HD version, welcome to watch and try.\n- [03\u002F28] 🔥 Mini-Gemini is coming! We release the [paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2403.18814.pdf), [demo](http:\u002F\u002F103.170.5.190:7860\u002F), [code](https:\u002F\u002Fgithub.com\u002Fdvlab-research\u002FMGM), [models](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FYanweiLi\u002Fmgm-6603c50b9b43d044171d0854'), and [data](https:\u002F\u002Fhuggingface.co\u002Fcollections\u002FYanweiLi\u002Fmgm-data-660463ea895a01d8f367624e)!\n\n## Contents\n- [Demo](#demo)\n- [Install](#install)\n- [Model](#model)\n- [Preparation](#preparation)\n- [Train](#train)\n- [Evaluation](#evaluation)\n- [Examples](#examples)\n- [Citation](#citation)\n- [Acknowledgement](#acknowledgement)\n- [License](#license)\n\n## Demo\nWe provide some selected examples in this section. More examples can be found in our [project page](https:\u002F\u002Fmini-gemini.github.io\u002F). Feel free to try our online [demo](http:\u002F\u002F103.170.5.190:7860\u002F)!\n\n\u003Cdiv align=center>\n\u003Cimg width=\"100%\" src=\"images\u002Fteaser.png\"\u002F>\n\u003C\u002Fdiv>\n\n## Install\nPlease follow the instructions below to install the required packages.\n\nNOTE: If you want to use the 2B version, please ensure to install the latest version Transformers (>=4.38.0).\n\n1. Clone this repository\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fdvlab-research\u002FMGM.git\n```\n\n2. Install Package\n```bash\nconda create -n mgm python=3.10 -y\nconda activate mgm\ncd MGM\npip install --upgrade pip  # enable PEP 660 support\npip install -e .\n```\n\n3. Install additional packages for training cases\n```bash\npip install ninja\npip install flash-attn --no-build-isolation\n```\n\n## Model\nThe framework is conceptually simple: dual vision encoders are utilized to provide low-resolution visual embedding and high-resolution candidates;\npatch info mining is proposed to conduct patch-level mining between high-resolution regions and low-resolution visual queries;\nLLM is utilized to marry text with images for both comprehension and generation at the same time.\n\n\u003Cdiv align=center>\n\u003Cimg width=\"98%\" src=\"images\u002Fpipeline.png\"\u002F>\n\u003C\u002Fdiv>\n\nWe provide all our fully finetuned models on Stage 1 and 2 data:\n\n| Model | LR | HR | Base LLM | Vision Encoder | Finetuning Data | Finetuning schedule | Download |\n|----------|----------|----------|----------|----------------|---------------|--------------------|------------------|\n| MGM-2B | 336 | 768 | Gemma-2B | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-2B) |\n| MGM-7B | 336 | 768 | Vicuna-7B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-7B) |\n| MGM-13B | 336 | 768 | Vicuna-13B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-13B) |\n| MGM-8B | 336 | 768 | LLaMA-3-8B-Instruct | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8B) |\n| MGM-8x7B | 336 | 768 | Mixtral-8x7B-Instruct-v0.1 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8x7B) |\n| MGM-34B | 336 | 768 | Nous-Hermes-2-Yi-34B | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-34B) |\n| MGM-7B-HD | 672 | 1536 | Vicuna-7B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-7B-HD) |\n| MGM-13B-HD | 672 | 1536 | Vicuna-13B-v1.5 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-13B-HD) |\n| MGM-8B-HD | 672 | 1536 | LLaMA-3-8B-Instruct | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8B-HD) |\n| MGM-8x7B-HD | 672 | 1536 | Mixtral-8x7B-Instruct-v0.1 | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8x7B-HD) |\n| MGM-34B-HD | 672 | 1536 | Nous-Hermes-2-Yi-34B | CLIP-L | MGM-Instruct | full_ft-1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-34B-HD) |\n\nHere are the pretrained weights on Stage 1 data only:\n| Model | LR | HR | Base LLM | Vision Encoder | Pretrain Data | Finetuning schedule | Download |\n|----------|----------|----------|----------|----------------|---------------|--------------------|------------------|\n| MGM-2B | 336 | 768 | Gemma-2B | CLIP-L | MGM-Pretrain | 1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-Pretrain\u002Ftree\u002Fmain\u002FMGM-2B) |\n| MGM-7B | 336 | 768 | Vicuna-7B-v1.5 | CLIP-L | MGM-Pretrain | 1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-Pretrain\u002Ftree\u002Fmain\u002FMGM-7B) |\n| MGM-13B | 336 | 768 | Vicuna-13B-v1.5 | CLIP-L | MGM-Pretrain | 1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-Pretrain\u002Ftree\u002Fmain\u002FMGM-13B) |\n| MGM-8x7B | 336 | 768 | Mixtral-8x7B-Instruct-v0.1 | CLIP-L | MGM-Pretrain | 1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-Pretrain\u002Ftree\u002Fmain\u002FMGM-8x7B) |\n| MGM-34B | 336 | 768 | Nous-Hermes-2-Yi-34B | CLIP-L | MGM-Pretrain | 1e | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-Pretrain\u002Ftree\u002Fmain\u002FMGM-34B) |\n\n## Preparation\n### Dataset\nWe provide the processed data for the model training. \nFor model pretraining, please download the following the training image-based data and organize them as:\n\n`->` means put the data in the local folder.\n- [LLaVA Images](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fliuhaotian\u002FLLaVA-Pretrain) -> `data\u002FMGM-Pretrain\u002Fimages`, `data\u002FMGM-Finetune\u002Fllava\u002FLLaVA-Pretrain\u002Fimages`\n- [ALLaVA Caption](https:\u002F\u002Fgithub.com\u002FFreedomIntelligence\u002FALLaVA) -> `data\u002FMGM-Pretrain\u002FALLaVA-4V`\n\nFor model finetuning, please download the following the instruction data and organize them as:\n\n`->` means put the data in the local folder.\n- [COCO train2017](http:\u002F\u002Fimages.cocodataset.org\u002Fzips\u002Ftrain2017.zip) -> `data\u002FMGM-Finetune\u002Fcoco`\n- [GQA](https:\u002F\u002Fdownloads.cs.stanford.edu\u002Fnlp\u002Fdata\u002Fgqa\u002Fimages.zip) -> `data\u002FMGM-Finetune\u002Fgqa`\n- [OCR-VQA](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing) (**we save all files as `.jpg`**) -> `data\u002FMGM-Finetune\u002Focr_vqa`\n- [TextVQA](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Ftextvqa\u002Fimages\u002Ftrain_val_images.zip) (not included for training) -> `data\u002FMGM-Finetune\u002Ftextvqa`\n- [VisualGenome part1](https:\u002F\u002Fcs.stanford.edu\u002Fpeople\u002Frak248\u002FVG_100K_2\u002Fimages.zip), [VisualGenome part2](https:\u002F\u002Fcs.stanford.edu\u002Fpeople\u002Frak248\u002FVG_100K_2\u002Fimages2.zip) -> `data\u002FMGM-Finetune\u002Fvg`\n- [ShareGPT4V-100K](https:\u002F\u002Fgithub.com\u002FInternLM\u002FInternLM-XComposer\u002Fblob\u002Fmain\u002Fprojects\u002FShareGPT4V\u002Fdocs\u002FData.md) -> `data\u002FMGM-Finetune\u002Fsam`, `share_textvqa`, `wikiart`, `web-celebrity`, `web-landmark`\n- [LAION GPT4V](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Flaion\u002Fgpt4v-dataset) -> `data\u002FMGM-Finetune\u002Fgpt4v-dataset`\n- [ALLaVA Instruction](https:\u002F\u002Fgithub.com\u002FFreedomIntelligence\u002FALLaVA) -> `data\u002FMGM-Pretrain\u002FALLaVA-4V`\n- [DocVQA](https:\u002F\u002Fwww.docvqa.org\u002Fdatasets\u002Fdocvqa) -> `data\u002FMGM-Finetune\u002Fdocvqa`\n- [ChartQA](https:\u002F\u002Fgithub.com\u002Fvis-nlp\u002FChartQA) -> `data\u002FMGM-Finetune\u002Fchartqa`\n- [DVQA](https:\u002F\u002Fgithub.com\u002Fkushalkafle\u002FDVQA_dataset) -> `data\u002FMGM-Finetune\u002Fdvqa`\n- [AI2D](https:\u002F\u002Fallenai.org\u002Fdata\u002Fdiagrams) -> `data\u002FMGM-Finetune\u002Fai2d`\n\nFor model evaluation, please follow this [link](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA\u002Fblob\u002Fmain\u002Fdocs\u002FEvaluation.md) for preparation. We use some extra benchmarks for evaluation. please download the following the training image-based data and organize them as:\n\n`->` means put the data in the local folder.\n- [MMMU](https:\u002F\u002Fmmmu-benchmark.github.io\u002F) -> `data\u002FMGM-Eval\u002FMMMU`\n- [MMB](https:\u002F\u002Fgithub.com\u002Fopen-compass\u002Fmmbench\u002F) -> `data\u002FMGM-Eval\u002FMMB`\n- [MathVista](https:\u002F\u002Fmathvista.github.io\u002F) -> `data\u002FMGM-Eval\u002FMathVista`\n\n\nPlease put the pretrained data, finetuned data, and eval data in  `MGM-Pretrain`, `MGM-Finetune`, and `MGM-Eval` subset following [Structure](#structure).\n\n\nFor meta info, please download the following files and organize them as in [Structure](#structure).\n\n| Data file name | Size |\n| --- | ---: |\n| [mgm_pretrain.json](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FYanweiLi\u002FMGM-Pretrain) | 1.68 G |\n| [mgm_instruction.json](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FYanweiLi\u002FMGM-Instruction) | 1.79 G |\n| [mgm_generation_pure_text.json](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FYanweiLi\u002FMGM-Instruction) | 0.04 G |\n\nIMPORTANT: `mgm_generation_pure_text.json` is a generation-related subset. **DO NOT** merge it with `mgm_instruction.json` as it is already included in it. You may merge this file with your customized LLM\u002FVLM SFT dataset to enable the reasoning generation ability.\n\n\n### Pretrained Weights\nWe recommend users to download the pretrained weights from the following link [CLIP-Vit-L-336](https:\u002F\u002Fhuggingface.co\u002Fopenai\u002Fclip-vit-large-patch14-336), [OpenCLIP-ConvNeXt-L](https:\u002F\u002Fhuggingface.co\u002Flaion\u002FCLIP-convnext_large_d_320.laion2B-s29B-b131K-ft-soup), [Gemma-2b-it](https:\u002F\u002Fhuggingface.co\u002Fgoogle\u002Fgemma-2b-it), [Vicuna-7b-v1.5](https:\u002F\u002Fhuggingface.co\u002Flmsys\u002Fvicuna-7b-v1.5), [Vicuna-13b-v1.5](https:\u002F\u002Fhuggingface.co\u002Flmsys\u002Fvicuna-13b-v1.5), [Mixtral-8x7B-Instruct-v0.1](https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMixtral-8x7B-Instruct-v0.1), and [Nous-Hermes-2-Yi-34B](https:\u002F\u002Fhuggingface.co\u002FNousResearch\u002FNous-Hermes-2-Yi-34B) , and put them in `model_zoo` following [Structure](#structure).\n\n\n### Structure\n\nThe folder structure should be organized as follows before training.\n\n```\nMGM\n├── mgm\n├── scripts\n├── work_dirs\n│   ├── MGM\n│   │   ├── MGM-2B\n│   │   ├── ...\n├── model_zoo\n│   ├── LLM\n│   │   ├── gemma\n│   │   │   ├── gemma-2b-it\n│   │   ├── vicuna\n│   │   │   ├── 7B-V1.5\n│   │   │   ├── 13B-V1.5\n│   │   ├── llama-3\n│   │   │   ├── Meta-Llama-3-8B-Instruct\n│   │   │   ├── Meta-Llama-3-70B-Instruct\n│   │   ├── mixtral\n│   │   │   ├── Mixtral-8x7B-Instruct-v0.1\n│   │   ├── Nous-Hermes-2-Yi-34B\n│   ├── OpenAI\n│   │   ├── clip-vit-large-patch14-336\n│   │   ├── openclip-convnext-large-d-320-laion2B-s29B-b131K-ft-soup\n├── data\n│   ├── MGM-Pretrain\n│   │   ├── mgm_pretrain.json\n│   │   ├── images\n│   │   ├── ALLaVA-4V\n│   ├── MGM-Finetune\n│   │   ├── mgm_instruction.json\n│   │   ├── llava\n│   │   ├── coco\n│   │   ├── gqa\n│   │   ├── ocr_vqa\n│   │   ├── textvqa\n│   │   ├── vg\n│   │   ├── gpt4v-dataset\n│   │   ├── sam\n│   │   ├── share_textvqa\n│   │   ├── wikiart\n│   │   ├── web-celebrity\n│   │   ├── web-landmark\n│   │   ├── ALLaVA-4V\n│   │   ├── docvqa\n│   │   ├── chartqa\n│   │   ├── dvqa\n│   │   ├── ai2d\n│   ├── MGM-Eval\n│   │   ├── MMMU\n│   │   ├── MMB\n│   │   ├── MathVista\n│   │   ├── ...\n```\n\n## Train\n\nThe training process consists of two stages: (1) feature alignment stage: bridge the vision and language tokens; (2) instruction tuning stage: teach the model to follow multimodal instructions.\n\nOur models are trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the `per_device_train_batch_size` and increase the `gradient_accumulation_steps` accordingly. Always keep the global batch size the same: `per_device_train_batch_size` x `gradient_accumulation_steps` x `num_gpus`.\n\nPlease make sure you download and organize the data following [Preparation](#preparation) before training.\n\nNOTE: Please set `hostfile` for 2 machine training and `hostfile_4` for 4 machine training.\n\nIf you want to train and finetune the framework, please run the following command for MGM-7B with image size 336:\n\n```bash\nbash scripts\u002Fllama\u002Ftrain\u002Fstage_1_2_full_v7b_336_hr_768.sh\n```\nor for MGM-13B with image size 336:\n```bash\nbash scripts\u002Fllama\u002Ftrain\u002Fstage_1_2_full_v13b_336_hr_768.sh\n```\nBecause we reuse the pre-trained projecter weights from the MGM-7B, you can directly use the MGM-7B-HD with image size 672 for stage-2 instruction tuning:\n```bash\nbash scripts\u002Fllama\u002Ftrain\u002Fstage_2_full_v7b_672_hr_1536.sh\n```\nPlease find more training scripts of `gemma`, `llama`, `mixtral`, and `yi` in `scripts\u002F`.\n\n\n## Evaluation\nWe perform evaluation on several image-based benchmarks. Please download the evaluation data following [Preparation](#preparation) and organize them as in [Structure](#structure).\n\n| Model | LLM | Res. | Link | TextVQA | MMB | MME | MM-Vet | MMMU_val | MMMU_test | MathVista |\n|----------|----------|----------|-----------|---|---|---|---|---|---|---|\nMGM-2B | Gemma-2B | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-2B) | 56.2 | 59.8 | 1341\u002F312 | 31.1 | 31.7 | 29.1 | 29.4\nMGM-7B | Vicuna-7B-v1.5 | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-7B) | 65.2 | 69.3 | 1523\u002F316 | 40.8 | 36.1 | 32.8 | 31.4 \nMGM-13B | Vicuna-13B-v1.5 | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-13B) | 65.9 | 68.5 | 1565\u002F322 | 46.0 | 38.1 | 33.5 | 37.0\nMGM-8B | LLaMA-3-8B-Instruct | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8B) | 67.6 | 72.7 | 1606\u002F341 | 47.3 | 38.2 | 36.3 | --\nMGM-8x7B | Mixtral-8x7B-Instruct-v0.1 | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8x7B) | 69.2 | 75.6 | 1639\u002F379 | 45.8 | 41.8 | 37.1 | 41.8\nMGM-34B | Nous-Hermes-2-Yi-34B | 336 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-34B) | 70.1 | 79.6 | 1666\u002F439 | 53.0 | 48.7 | 43.6 | 38.9\nMGM-7B-HD | Vicuna-7B-v1.5 | 672 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-7B-HD) | 68.4 | 65.8 | 1546\u002F319 | 41.3 | 36.8 | 32.9 | 32.2\nMGM-13B-HD | Vicuna-13B-v1.5 | 672 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-13B-HD) | 70.2 | 68.6 | 1597\u002F320 | 50.5 | 37.3 | 35.1 | 37.0\nMGM-8B-HD | LLaMA-3-8B-Instruct | 672 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8B-HD) | 71.6 | -- | 1532\u002F357 | -- | 37.0 | -- | --\nMGM-8x7B-HD | Mixtral-8x7B-Instruct-v0.1 | 672 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-8x7B-HD) | 71.9 | 74.7 | 1633\u002F356 | 53.5 | 40.0 | 37.0 | 43.1\nMGM-34B-HD | Nous-Hermes-2-Yi-34B | 672 | [ckpt](https:\u002F\u002Fhuggingface.co\u002FYanweiLi\u002FMGM-34B-HD) | 74.1 | 80.6 | 1659\u002F482 | 59.3 | 48.0 | 44.9 | 43.3\n\n\n\nIf you want to evaluate the model on image-based benchmarks, please use the scripts in `scripts\u002FMODEL_PATH\u002Feval`. \nFor example, run the following command for TextVQA evaluation with MGM-7B-HD:\n```bash\nbash scripts\u002Fllama\u002Feval\u002Ftextvqa.sh\n```\nPlease find more evaluation scripts in `scripts\u002FMODEL_PATH`.\n\n\n### CLI Inference\nChat with images without the need of Gradio interface. It also supports multiple GPUs, 4-bit and 8-bit quantized inference. With 4-bit quantization.\nPlease make sure you have installed [diffusers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers) and [PaddleOCR](https:\u002F\u002Fgithub.com\u002FPaddlePaddle\u002FPaddleOCR\u002Fblob\u002Frelease\u002F2.7\u002FREADME_en.md) (only for better experience with OCR), and try this for image and generation inference:\n\n```bash\npython -m mgm.serve.cli \\\n    --model-path work_dirs\u002FMGM\u002FMGM-13B-HD \\\n    --image-file \u003Cpath to your image>\n```\n\nor try this better experience with OCR (make sure you have installed [PaddleOCR](https:\u002F\u002Fgithub.com\u002FPaddlePaddle\u002FPaddleOCR\u002Fblob\u002Frelease\u002F2.7\u002FREADME_en.md)):\n```bash\npython -m mgm.serve.cli \\\n    --model-path work_dirs\u002FMGM\u002FMGM-13B-HD \\\n    --image-file \u003Cpath to your image> \\\n    --ocr\n```\n\nor try this for inference with generation (make sure you have installed [diffusers](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fdiffusers)):\n```bash\npython -m mgm.serve.cli \\\n    --model-path work_dirs\u002FMGM\u002FMGM-13B-HD \\\n    --image-file \u003Cpath to your image> \\\n    --gen\n```\n\nYou can also try 8bit or even 4bit for efficient inference \n```bash\npython -m mgm.serve.cli \\\n    --model-path work_dirs\u002FMGM\u002FMGM-13B-HD \\\n    --image-file \u003Cpath to your image> \\\n    --gen\n    --load-8bit\n```\n\n### Gradio Web UI\n\nHere, we adopt the Gradio UI similar to that in LLaVA to provide a user-friendly interface for our models.\nTo launch a Gradio demo locally, please run the following commands one by one. If you plan to launch multiple model workers to compare between different checkpoints, you only need to launch the controller and the web server *ONCE*.\n\n#### Launch a controller\n```Shell\npython -m mgm.serve.controller --host 0.0.0.0 --port 10000\n```\n\n#### Launch a gradio web server.\n```Shell\npython -m mgm.serve.gradio_web_server --controller http:\u002F\u002Flocalhost:10000 --model-list-mode reload\n```\nYou just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen. You may notice that there is no model in the model list. Do not worry, as we have not launched any model worker yet. It will be automatically updated when you launch a model worker.\n\n#### Launch a model worker\nThis is the actual *worker* that performs the inference on the GPU.  Each worker is responsible for a single model specified in `--model-path`.\n\n```Shell\npython -m mgm.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path work_dirs\u002FMGM\u002FMGM-13B-HD\n```\nWait until the process finishes loading the model and you see \"Uvicorn running on ...\".  Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.\n\nYou can launch as many workers as you want, and compare between different models in the same Gradio interface. Please keep the `--controller` the same, and modify the `--port` and `--worker` to a different port number for each worker.\n```Shell\npython -m mgm.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port \u003Cdifferent from 40000, say 40001> --worker http:\u002F\u002Flocalhost:\u003Cchange accordingly, i.e. 40001> --model-path work_dirs\u002FMGM\u002FMGM-34B-HD\n```\n\nIf you are using an Apple device with an M1 or M2 chip, you can specify the mps device by using the `--device` flag: `--device mps`.\n\n#### Launch a model worker (Multiple GPUs, when GPU VRAM \u003C= 24GB)\n\nIf the VRAM of your GPU is less than 24GB (e.g., RTX 3090, RTX 4090, etc.), you may try running it with multiple GPUs. Our latest code base will automatically try to use multiple GPUs if you have more than one GPU. You can specify which GPUs to use with `CUDA_VISIBLE_DEVICES`. Below is an example of running with the first two GPUs.\n\n```Shell\nCUDA_VISIBLE_DEVICES=0,1 python -m mgm.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path work_dirs\u002FMGM\u002FMGM-13B-HD\n```\n\n#### Launch a model worker (4-bit, 8-bit inference, quantized)\n\nYou can launch the model worker with quantized bits (4-bit, 8-bit), which allows you to run the inference with reduced GPU memory footprint. Note that inference with quantized bits may not be as accurate as the full-precision model. Simply append `--load-4bit` or `--load-8bit` to the **model worker** command that you are executing. Below is an example of running with 4-bit quantization.\n\n```Shell\npython -m mgm.serve.model_worker --host 0.0.0.0 --controller http:\u002F\u002Flocalhost:10000 --port 40000 --worker http:\u002F\u002Flocalhost:40000 --model-path work_dirs\u002FMGM\u002FMGM-13B-HD --load-4bit\n```\n\n## Examples\nWe provide some examples in this section. More examples can be found in our [project page](https:\u002F\u002Fmini-gemini.github.io\u002F).\n\n### Hi-Resolution Understanding\n\u003Cdiv align=center>\n\u003Cimg width=\"98%\" src=\"images\u002Fdemo_und.png\"\u002F>\n\u003C\u002Fdiv>\n\n### Generation with Reasoning\n\u003Cdiv align=center>\n\u003Cimg width=\"98%\" src=\"images\u002Fdemo_gen.png\"\u002F>\n\u003C\u002Fdiv>\n\n## Citation\nIf you find this repo useful for your research, please consider citing the paper\n```\n@article{li2024mgm,\n  title={Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models},\n  author={Li, Yanwei and Zhang, Yuechen and Wang, Chengyao and Zhong, Zhisheng and Chen, Yixin and Chu, Ruihang and Liu, Shaoteng and Jia, Jiaya},\n  journal={arXiv:2403.18814},\n  year={2023}\n}\n```\n\n## Acknowledgement\nThis project is not affiliated with Google LLC.\n\nWe would like to thank the following repos for their great work:\n\n- This work is built upon the [LLaVA](https:\u002F\u002Fgithub.com\u002Fhaotian-liu\u002FLLaVA).\n- This work utilizes LLMs from [Gemma](https:\u002F\u002Fhuggingface.co\u002Fgoogle\u002Fgemma-2b-it), [Vicuna](https:\u002F\u002Fgithub.com\u002Flm-sys\u002FFastChat), [Mixtral](https:\u002F\u002Fhuggingface.co\u002Fmistralai\u002FMixtral-8x7B-Instruct-v0.1), and [Nous-Hermes](https:\u002F\u002Fhuggingface.co\u002FNousResearch\u002FNous-Hermes-2-Yi-34B).\n\n## License\n[![Code License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode%20License-Apache_2.0-yellow.svg)](https:\u002F\u002Fgithub.com\u002Fdvlab-research\u002FMGM\u002Fblob\u002Fmain\u002FLICENSE)\n[![Data License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FData%20License-CC%20By%20NC%204.0-orange.svg)](https:\u002F\u002Fgithub.com\u002Fdvlab-research\u002FMGM\u002Fblob\u002Fmain\u002FDATA_LICENSE)\n[![Weight License](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FWeight%20License-CC%20By%20NC%204.0-red)](https:\u002F\u002Fgithub.com\u002Fdvlab-research\u002FMGM\u002Fblob\u002Fmain\u002FWEIGHT_LICENSE)\n\nThe data and checkpoint is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaVA, LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.\n","Mini-Gemini是一个基于多模态视觉语言模型的框架，旨在挖掘大规模语言模型在图像理解、推理和生成方面的潜力。该项目支持从2B到34B参数规模的密集型和混合专家系统（MoE）大型语言模型，采用双视觉编码器提供低分辨率视觉嵌入和高分辨率候选区域，以增强模型的多模态处理能力。Mini-Gemini基于LLaVA构建，并已发布包括论文、代码、预训练模型及数据集在内的完整资源。此项目适用于需要结合文本与图像信息进行复杂任务处理的应用场景，如跨模态搜索、内容生成或智能问答等。",2,"2026-06-11 03:41:18","high_star"]