[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-2624":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":19,"compositeScore":20,"rankGlobal":10,"rankLanguage":10,"license":21,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":31,"readmeContent":32,"aiSummary":33,"trendingCount":16,"starSnapshotCount":16,"syncStatus":34,"lastSyncTime":35,"discoverSource":36},2624,"Janus","deepseek-ai\u002FJanus","deepseek-ai","Janus-Series: Unified Multimodal Understanding and Generation Models","",null,"Python",17748,2231,147,159,0,7,25,1,45,"MIT License",false,"main",[25,26,27,28,29,30],"any-to-any","foundation-models","llm","multimodal","unified-model","vision-language-pretraining","2026-06-12 02:00:42","\u003C!-- markdownlint-disable first-line-h1 -->\n\u003C!-- markdownlint-disable html -->\n\u003C!-- markdownlint-disable no-duplicate-header -->\n\n\u003Cdiv align=\"center\">\n  \u003Cimg src=\"images\u002Flogo.svg\" width=\"60%\" alt=\"DeepSeek LLM\" \u002F>\n\u003C\u002Fdiv>\n\u003Chr>\n\n\u003Cdiv align=\"center\">\n\u003Ch1>🚀 Janus-Series: Unified Multimodal Understanding and Generation Models\u003C\u002Fh1>\n\n\u003C\u002Fdiv>\n\n\u003Cdiv align=\"center\">\n\n  \u003Ca href=\"https:\u002F\u002Fwww.deepseek.com\u002F\" target=\"_blank\">\n    \u003Cimg alt=\"Homepage\" src=\"images\u002Fbadge.svg\" \u002F>\n  \u003C\u002Fa>\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\" target=\"_blank\">\n    \u003Cimg alt=\"Hugging Face\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white\" \u002F>\n  \u003C\u002Fa>\n\n\u003C\u002Fdiv>\n\n\n\u003Cdiv align=\"center\">\n\n  \u003C!-- \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FTc7c45Zzu5\" target=\"_blank\">\n    \u003Cimg alt=\"Discord\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDiscord-DeepSeek%20AI-7289da?logo=discord&logoColor=white&color=7289da\" \u002F>\n  \u003C\u002Fa> -->\n  \u003C!-- \u003Ca href=\"images\u002Fqr.jpeg\" target=\"_blank\">\n    \u003Cimg alt=\"Wechat\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FWeChat-DeepSeek%20AI-brightgreen?logo=wechat&logoColor=white\" \u002F>\n  \u003C\u002Fa> -->\n  \u003C!-- \u003Ca href=\"https:\u002F\u002Ftwitter.com\u002Fdeepseek_ai\" target=\"_blank\">\n    \u003Cimg alt=\"Twitter Follow\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FTwitter-deepseek_ai-white?logo=x&logoColor=white\" \u002F>\n  \u003C\u002Fa> -->\n\n\u003C\u002Fdiv>\n\n\u003Cdiv align=\"center\">\n\n  \u003Ca href=\"LICENSE-CODE\">\n    \u003Cimg alt=\"Code License\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FCode_License-MIT-f5de53?&color=f5de53\">\n  \u003C\u002Fa>\n  \u003Ca href=\"LICENSE-MODEL\">\n    \u003Cimg alt=\"Model License\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FModel_License-Model_Agreement-f5de53?&color=f5de53\">\n  \u003C\u002Fa>\n\u003C\u002Fdiv>\n\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"#2-model-download\">\u003Cb>📥 Model Download\u003C\u002Fb>\u003C\u002Fa> |\n  \u003Ca href=\"#3-quick-start\">\u003Cb>⚡ Quick Start\u003C\u002Fb>\u003C\u002Fa> |\n  \u003Ca href=\"#4-license\">\u003Cb>📜 License\u003C\u002Fb>\u003C\u002Fa> |\n  \u003Ca href=\"#5-citation\">\u003Cb>📖 Citation\u003C\u002Fb>\u003C\u002Fa> \u003Cbr>\n  \u003C!-- 📄 Paper Link (\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13848\">\u003Cb>Janus\u003C\u002Fb>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13848\">\u003Cb>JanusFlow\u003C\u002Fb>\u003C\u002Fa>) | -->\n  🤗 Online Demo (\u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanus-Pro-7B\">\u003Cb>Janus-Pro-7B\u003C\u002Fb>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanus-1.3B\">\u003Cb>Janus\u003C\u002Fb>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanusFlow-1.3B\">\u003Cb>JanusFlow\u003C\u002Fb>\u003C\u002Fa>)\n\u003C\u002Fp>\n\n\n## News\n\n**2025.01.27**: Janus-Pro is released, an advanced version of Janus, improving both multimodal understanding and visual generation significantly. See [paper](.\u002Fjanus_pro_tech_report.pdf)\n\n**2024.11.13**: JanusFlow is released, a new unified model with rectified flow for image generation. See [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2411.07975), [demo](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanusFlow-1.3B) and [usage](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002FJanus?tab=readme-ov-file#janusflow).\n\n**2024.10.23**: Evaluation code for reproducing the multimodal understanding results from the paper has been added to VLMEvalKit. Please refer to [this link]( https:\u002F\u002Fgithub.com\u002Fopen-compass\u002FVLMEvalKit\u002Fpull\u002F541).\n\n**2024.10.20**: (1) Fix a bug in [tokenizer_config.json](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FJanus-1.3B\u002Fblob\u002Fmain\u002Ftokenizer_config.json). The previous version caused classifier-free guidance to not function properly, resulting in relatively poor visual generation quality. (2) Release Gradio demo ([online demo](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanus-1.3B) and  [local](#gradio-demo)).\n\n\n## 1. Introduction\n\n\u003Ca href=\".\u002Fjanus_pro_tech_report.pdf\">\u003Cb>Janus-Pro: Unified Multimodal Understanding and\nGeneration with Data and Model Scaling\u003C\u002Fb>\u003C\u002Fa>\n\n**Janus-Pro** is an advanced version of the previous work Janus. Specifically, Janus-Pro incorporates (1) an optimized training strategy, (2) expanded training data, and (3) scaling to larger model size. With these improvements, Janus-Pro achieves significant advancements in both multimodal understanding and text-to-image instruction-following capabilities, while also enhancing the stability of text-to-image generation.\n\n\u003Cdiv align=\"center\">\n\u003Cimg alt=\"image\" src=\"images\u002Fteaser_januspro.png\" style=\"width:90%;\">\n\u003C\u002Fdiv>\n\n\n\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13848\">\u003Cb>Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation\u003C\u002Fb>\u003C\u002Fa>\n\n**Janus** is a novel autoregressive framework that unifies multimodal understanding and generation. It addresses the limitations of previous approaches by decoupling visual encoding into separate pathways, while still utilizing a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder’s roles in understanding and generation, but also enhances the framework’s flexibility. Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.\n\n\u003Cdiv align=\"center\">\n\u003Cimg alt=\"image\" src=\"images\u002Fteaser.png\" style=\"width:90%;\">\n\u003C\u002Fdiv>\n\n\u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2411.07975\">\u003Cb>JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation\u003C\u002Fb>\u003C\u002Fa>\n\n**JanusFlow** introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.\n\n\u003Cdiv align=\"center\">\n\u003Cimg alt=\"image\" src=\"images\u002Fteaser_janusflow.png\" style=\"width:90%;\">\n\u003C\u002Fdiv>\n \n\n## 2. Model Download\n\nWe release Janus to the public to support a broader and more diverse range of research within both academic and commercial communities.\nPlease note that the use of this model is subject to the terms outlined in [License section](#5-license). Commercial usage is\npermitted under these terms.\n\n### Huggingface\n\n| Model                 | Sequence Length | Download                                                                    |\n|-----------------------|-----------------|-----------------------------------------------------------------------------|\n| Janus-1.3B | 4096            | [🤗 Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FJanus-1.3B) |\n| JanusFlow-1.3B | 4096        | [🤗 Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FJanusFlow-1.3B) |\n| Janus-Pro-1B | 4096            | [🤗 Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FJanus-Pro-1B) |\n| Janus-Pro-7B | 4096        | [🤗 Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fdeepseek-ai\u002FJanus-Pro-7B) |\n\n\n\n## 3. Quick Start\n\u003Cdetails>\n\u003Csummary>\u003Ch3>Janus-Pro\u003C\u002Fh3>\u003C\u002Fsummary>\n\n### Installation\n\nOn the basis of `Python >= 3.8` environment, install the necessary dependencies by running the following command:\n\n```shell\npip install -e .\n```\n\n\n### Simple Inference Example\n\n#### Multimodal Understanding\n```python\n\nimport torch\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\nfrom janus.utils.io import load_pil_images\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanus-Pro-7B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"\u003C|User|>\",\n        \"content\": f\"\u003Cimage_placeholder>\\n{question}\",\n        \"images\": [image],\n    },\n    {\"role\": \"\u003C|Assistant|>\", \"content\": \"\"},\n]\n\n# load images and prepare for inputs\npil_images = load_pil_images(conversation)\nprepare_inputs = vl_chat_processor(\n    conversations=conversation, images=pil_images, force_batchify=True\n).to(vl_gpt.device)\n\n# # run image encoder to get the image embeddings\ninputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)\n\n# # run the model to get the response\noutputs = vl_gpt.language_model.generate(\n    inputs_embeds=inputs_embeds,\n    attention_mask=prepare_inputs.attention_mask,\n    pad_token_id=tokenizer.eos_token_id,\n    bos_token_id=tokenizer.bos_token_id,\n    eos_token_id=tokenizer.eos_token_id,\n    max_new_tokens=512,\n    do_sample=False,\n    use_cache=True,\n)\n\nanswer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)\nprint(f\"{prepare_inputs['sft_format'][0]}\", answer)\n\n```\n\n#### Text-to-Image Generation\n```python\nimport os\nimport PIL.Image\nimport torch\nimport numpy as np\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\n\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanus-Pro-7B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"\u003C|User|>\",\n        \"content\": \"A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair\",\n    },\n    {\"role\": \"\u003C|Assistant|>\", \"content\": \"\"},\n]\n\nsft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(\n    conversations=conversation,\n    sft_format=vl_chat_processor.sft_format,\n    system_prompt=\"\",\n)\nprompt = sft_format + vl_chat_processor.image_start_tag\n\n\n@torch.inference_mode()\ndef generate(\n    mmgpt: MultiModalityCausalLM,\n    vl_chat_processor: VLChatProcessor,\n    prompt: str,\n    temperature: float = 1,\n    parallel_size: int = 16,\n    cfg_weight: float = 5,\n    image_token_num_per_image: int = 576,\n    img_size: int = 384,\n    patch_size: int = 16,\n):\n    input_ids = vl_chat_processor.tokenizer.encode(prompt)\n    input_ids = torch.LongTensor(input_ids)\n\n    tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()\n    for i in range(parallel_size*2):\n        tokens[i, :] = input_ids\n        if i % 2 != 0:\n            tokens[i, 1:-1] = vl_chat_processor.pad_id\n\n    inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)\n\n    generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()\n\n    for i in range(image_token_num_per_image):\n        outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)\n        hidden_states = outputs.last_hidden_state\n        \n        logits = mmgpt.gen_head(hidden_states[:, -1, :])\n        logit_cond = logits[0::2, :]\n        logit_uncond = logits[1::2, :]\n        \n        logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)\n        probs = torch.softmax(logits \u002F temperature, dim=-1)\n\n        next_token = torch.multinomial(probs, num_samples=1)\n        generated_tokens[:, i] = next_token.squeeze(dim=-1)\n\n        next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)\n        img_embeds = mmgpt.prepare_gen_img_embeds(next_token)\n        inputs_embeds = img_embeds.unsqueeze(dim=1)\n\n\n    dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size\u002F\u002Fpatch_size, img_size\u002F\u002Fpatch_size])\n    dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)\n\n    dec = np.clip((dec + 1) \u002F 2 * 255, 0, 255)\n\n    visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)\n    visual_img[:, :, :] = dec\n\n    os.makedirs('generated_samples', exist_ok=True)\n    for i in range(parallel_size):\n        save_path = os.path.join('generated_samples', \"img_{}.jpg\".format(i))\n        PIL.Image.fromarray(visual_img[i]).save(save_path)\n\n\ngenerate(\n    vl_gpt,\n    vl_chat_processor,\n    prompt,\n)\n```\n\n### Gradio Demo\nWe have deployed online demo in [Huggingface](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanus-Pro-7B).\n\n\nFor the local gradio demo, you can run with the following command:\n\n```\npip install -e .[gradio]\n\npython demo\u002Fapp_januspro.py\n```\n\nHave Fun!\n\n\u003C\u002Fdetails>\n\n\n\n\u003Cdetails>\n\u003Csummary>\u003Ch3>Janus\u003C\u002Fh3>\u003C\u002Fsummary>\n\n### Installation\n\nOn the basis of `Python >= 3.8` environment, install the necessary dependencies by running the following command:\n\n```shell\npip install -e .\n```\n\n\n### Simple Inference Example\n\n#### Multimodal Understanding\n```python\n\nimport torch\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\nfrom janus.utils.io import load_pil_images\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanus-1.3B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"User\",\n        \"content\": \"\u003Cimage_placeholder>\\nConvert the formula into latex code.\",\n        \"images\": [\"images\u002Fequation.png\"],\n    },\n    {\"role\": \"Assistant\", \"content\": \"\"},\n]\n\n# load images and prepare for inputs\npil_images = load_pil_images(conversation)\nprepare_inputs = vl_chat_processor(\n    conversations=conversation, images=pil_images, force_batchify=True\n).to(vl_gpt.device)\n\n# # run image encoder to get the image embeddings\ninputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)\n\n# # run the model to get the response\noutputs = vl_gpt.language_model.generate(\n    inputs_embeds=inputs_embeds,\n    attention_mask=prepare_inputs.attention_mask,\n    pad_token_id=tokenizer.eos_token_id,\n    bos_token_id=tokenizer.bos_token_id,\n    eos_token_id=tokenizer.eos_token_id,\n    max_new_tokens=512,\n    do_sample=False,\n    use_cache=True,\n)\n\nanswer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)\nprint(f\"{prepare_inputs['sft_format'][0]}\", answer)\n\n```\n\n#### Text-to-Image Generation\n```python\nimport os\nimport PIL.Image\nimport torch\nimport numpy as np\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\n\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanus-1.3B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"User\",\n        \"content\": \"A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair\",\n    },\n    {\"role\": \"Assistant\", \"content\": \"\"},\n]\n\nsft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(\n    conversations=conversation,\n    sft_format=vl_chat_processor.sft_format,\n    system_prompt=\"\",\n)\nprompt = sft_format + vl_chat_processor.image_start_tag\n\n\n@torch.inference_mode()\ndef generate(\n    mmgpt: MultiModalityCausalLM,\n    vl_chat_processor: VLChatProcessor,\n    prompt: str,\n    temperature: float = 1,\n    parallel_size: int = 16,\n    cfg_weight: float = 5,\n    image_token_num_per_image: int = 576,\n    img_size: int = 384,\n    patch_size: int = 16,\n):\n    input_ids = vl_chat_processor.tokenizer.encode(prompt)\n    input_ids = torch.LongTensor(input_ids)\n\n    tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()\n    for i in range(parallel_size*2):\n        tokens[i, :] = input_ids\n        if i % 2 != 0:\n            tokens[i, 1:-1] = vl_chat_processor.pad_id\n\n    inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)\n\n    generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()\n\n    for i in range(image_token_num_per_image):\n        outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)\n        hidden_states = outputs.last_hidden_state\n        \n        logits = mmgpt.gen_head(hidden_states[:, -1, :])\n        logit_cond = logits[0::2, :]\n        logit_uncond = logits[1::2, :]\n        \n        logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)\n        probs = torch.softmax(logits \u002F temperature, dim=-1)\n\n        next_token = torch.multinomial(probs, num_samples=1)\n        generated_tokens[:, i] = next_token.squeeze(dim=-1)\n\n        next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)\n        img_embeds = mmgpt.prepare_gen_img_embeds(next_token)\n        inputs_embeds = img_embeds.unsqueeze(dim=1)\n\n\n    dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size\u002F\u002Fpatch_size, img_size\u002F\u002Fpatch_size])\n    dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)\n\n    dec = np.clip((dec + 1) \u002F 2 * 255, 0, 255)\n\n    visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)\n    visual_img[:, :, :] = dec\n\n    os.makedirs('generated_samples', exist_ok=True)\n    for i in range(parallel_size):\n        save_path = os.path.join('generated_samples', \"img_{}.jpg\".format(i))\n        PIL.Image.fromarray(visual_img[i]).save(save_path)\n\n\ngenerate(\n    vl_gpt,\n    vl_chat_processor,\n    prompt,\n)\n```\n\n### Gradio Demo\nWe have deployed online demo in [Huggingface](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanus-1.3B).\n\n\nFor the local gradio demo, you can run with the following command:\n\n```\npip install -e .[gradio]\n\npython demo\u002Fapp.py\n```\n\nHave Fun!\n\n### FastAPI Demo\nIt's easy to run a FastAPI server to host an API server running the same functions as gradio.\n\nTo start FastAPI server, run the following command:\n\n```\npython demo\u002Ffastapi_app.py\n```\n\nTo test the server, you can open another terminal and run:\n\n```\npython demo\u002Ffastapi_client.py\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>\u003Ch3>JanusFlow\u003C\u002Fh3>\u003C\u002Fsummary>\n    \n### Installation\n\nOn the basis of `Python >= 3.8` environment, install the necessary dependencies by running the following command:\n\n```shell\npip install -e .\npip install diffusers[torch]\n```\n\n### 🤗 Huggingface Online Demo\nCheck out the demo in [this link](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fdeepseek-ai\u002FJanusFlow-1.3B).\n\n### Simple Inference Example\n\n#### Multimodal Understanding\n```python\n\nimport torch\nfrom janus.janusflow.models import MultiModalityCausalLM, VLChatProcessor\nfrom janus.utils.io import load_pil_images\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanusFlow-1.3B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt = MultiModalityCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"User\",\n        \"content\": \"\u003Cimage_placeholder>\\nConvert the formula into latex code.\",\n        \"images\": [\"images\u002Fequation.png\"],\n    },\n    {\"role\": \"Assistant\", \"content\": \"\"},\n]\n\n# load images and prepare for inputs\npil_images = load_pil_images(conversation)\nprepare_inputs = vl_chat_processor(\n    conversations=conversation, images=pil_images, force_batchify=True\n).to(vl_gpt.device)\n\n# # run image encoder to get the image embeddings\ninputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)\n\n# # run the model to get the response\noutputs = vl_gpt.language_model.generate(\n    inputs_embeds=inputs_embeds,\n    attention_mask=prepare_inputs.attention_mask,\n    pad_token_id=tokenizer.eos_token_id,\n    bos_token_id=tokenizer.bos_token_id,\n    eos_token_id=tokenizer.eos_token_id,\n    max_new_tokens=512,\n    do_sample=False,\n    use_cache=True,\n)\n\nanswer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)\nprint(f\"{prepare_inputs['sft_format'][0]}\", answer)\n\n```\n\n#### Text-to-Image Generation\n```python\nimport os\nimport PIL.Image\nimport torch\nimport numpy as np\nfrom janus.janusflow.models import MultiModalityCausalLM, VLChatProcessor\nimport torchvision\n\n\n# specify the path to the model\nmodel_path = \"deepseek-ai\u002FJanusFlow-1.3B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\n\nvl_gpt = MultiModalityCausalLM.from_pretrained(\n    model_path, trust_remote_code=True\n)\nvl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()\n\nfrom diffusers.models import AutoencoderKL\n# remember to use bfloat16 dtype, this vae doesn't work with fp16\nvae = AutoencoderKL.from_pretrained(\"stabilityai\u002Fsdxl-vae\")\nvae = vae.to(torch.bfloat16).cuda().eval()\n\nconversation = [\n    {\n        \"role\": \"User\",\n        \"content\": \"A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair\",\n    },\n    {\"role\": \"Assistant\", \"content\": \"\"},\n]\n\nsft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(\n    conversations=conversation,\n    sft_format=vl_chat_processor.sft_format,\n    system_prompt=\"\",\n)\nprompt = sft_format + vl_chat_processor.image_gen_tag\n\n\n@torch.inference_mode()\ndef generate(\n    mmgpt: MultiModalityCausalLM,\n    vl_chat_processor: VLChatProcessor,\n    prompt: str,\n    cfg_weight: float = 5.0,\n    num_inference_steps: int = 30,\n    batchsize: int = 5\n):\n    input_ids = vl_chat_processor.tokenizer.encode(prompt)\n    input_ids = torch.LongTensor(input_ids)\n    \n    tokens = torch.stack([input_ids] * 2 * batchsize).cuda()\n    tokens[batchsize:, 1:] = vl_chat_processor.pad_id\n    inputs_embeds = vl_gpt.language_model.get_input_embeddings()(tokens)\n\n    # we remove the last \u003Cbog> token and replace it with t_emb later\n    inputs_embeds = inputs_embeds[:, :-1, :] \n    \n    # generate with rectified flow ode\n    # step 1: encode with vision_gen_enc\n    z = torch.randn((batchsize, 4, 48, 48), dtype=torch.bfloat16).cuda()\n    \n    dt = 1.0 \u002F num_inference_steps\n    dt = torch.zeros_like(z).cuda().to(torch.bfloat16) + dt\n    \n    # step 2: run ode\n    attention_mask = torch.ones((2*batchsize, inputs_embeds.shape[1]+577)).to(vl_gpt.device)\n    attention_mask[batchsize:, 1:inputs_embeds.shape[1]] = 0\n    attention_mask = attention_mask.int()\n    for step in range(num_inference_steps):\n        # prepare inputs for the llm\n        z_input = torch.cat([z, z], dim=0) # for cfg\n        t = step \u002F num_inference_steps * 1000.\n        t = torch.tensor([t] * z_input.shape[0]).to(dt)\n        z_enc = vl_gpt.vision_gen_enc_model(z_input, t)\n        z_emb, t_emb, hs = z_enc[0], z_enc[1], z_enc[2]\n        z_emb = z_emb.view(z_emb.shape[0], z_emb.shape[1], -1).permute(0, 2, 1)\n        z_emb = vl_gpt.vision_gen_enc_aligner(z_emb)\n        llm_emb = torch.cat([inputs_embeds, t_emb.unsqueeze(1), z_emb], dim=1)\n\n        # input to the llm\n        # we apply attention mask for CFG: 1 for tokens that are not masked, 0 for tokens that are masked.\n        if step == 0:\n            outputs = vl_gpt.language_model.model(inputs_embeds=llm_emb, \n                                             use_cache=True, \n                                             attention_mask=attention_mask,\n                                             past_key_values=None)\n            past_key_values = []\n            for kv_cache in past_key_values:\n                k, v = kv_cache[0], kv_cache[1]\n                past_key_values.append((k[:, :, :inputs_embeds.shape[1], :], v[:, :, :inputs_embeds.shape[1], :]))\n            past_key_values = tuple(past_key_values)\n        else:\n            outputs = vl_gpt.language_model.model(inputs_embeds=llm_emb, \n                                             use_cache=True, \n                                             attention_mask=attention_mask,\n                                             past_key_values=past_key_values)\n        hidden_states = outputs.last_hidden_state\n        \n        # transform hidden_states back to v\n        hidden_states = vl_gpt.vision_gen_dec_aligner(vl_gpt.vision_gen_dec_aligner_norm(hidden_states[:, -576:, :]))\n        hidden_states = hidden_states.reshape(z_emb.shape[0], 24, 24, 768).permute(0, 3, 1, 2)\n        v = vl_gpt.vision_gen_dec_model(hidden_states, hs, t_emb)\n        v_cond, v_uncond = torch.chunk(v, 2)\n        v = cfg_weight * v_cond - (cfg_weight-1.) * v_uncond\n        z = z + dt * v\n        \n    # step 3: decode with vision_gen_dec and sdxl vae\n    decoded_image = vae.decode(z \u002F vae.config.scaling_factor).sample\n    \n    os.makedirs('generated_samples', exist_ok=True)\n    save_path = os.path.join('generated_samples', \"img.jpg\")\n    torchvision.utils.save_image(decoded_image.clip_(-1.0, 1.0)*0.5+0.5, save_path)\n\ngenerate(\n    vl_gpt,\n    vl_chat_processor,\n    prompt,\n    cfg_weight=2.0,\n    num_inference_steps=30,\n    batchsize=5\n)\n```\n\n### Gradio Demo\nFor the local gradio demo, you can run with the following command:\n\n```\npip install -e .[gradio]\n\npython demo\u002Fapp_janusflow.py\n```\n\nHave Fun!\n    \n\u003C\u002Fdetails>\n\n## 4. License\n\nThis code repository is licensed under [the MIT License](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002FDeepSeek-LLM\u002Fblob\u002FHEAD\u002FLICENSE-CODE). The use of Janus models is subject to [DeepSeek Model License](https:\u002F\u002Fgithub.com\u002Fdeepseek-ai\u002FDeepSeek-LLM\u002Fblob\u002FHEAD\u002FLICENSE-MODEL).\n\n## 5. Citation\n\n```bibtex\n@article{chen2025janus,\n  title={Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling},\n  author={Chen, Xiaokang and Wu, Zhiyu and Liu, Xingchao and Pan, Zizheng and Liu, Wen and Xie, Zhenda and Yu, Xingkai and Ruan, Chong},\n  journal={arXiv preprint arXiv:2501.17811},\n  year={2025}\n}\n\n@article{wu2024janus,\n  title={Janus: Decoupling visual encoding for unified multimodal understanding and generation},\n  author={Wu, Chengyue and Chen, Xiaokang and Wu, Zhiyu and Ma, Yiyang and Liu, Xingchao and Pan, Zizheng and Liu, Wen and Xie, Zhenda and Yu, Xingkai and Ruan, Chong and others},\n  journal={arXiv preprint arXiv:2410.13848},\n  year={2024}\n}\n\n@misc{ma2024janusflow,\n      title={JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation}, \n      author={Yiyang Ma and Xingchao Liu and Xiaokang Chen and Wen Liu and Chengyue Wu and Zhiyu Wu and Zizheng Pan and Zhenda Xie and Haowei Zhang and Xingkai yu and Liang Zhao and Yisong Wang and Jiaying Liu and Chong Ruan},\n      journal={arXiv preprint arXiv:2411.07975},\n      year={2024}\n}\n```\n\n## 6. Contact\n\nIf you have any questions, please raise an issue or contact us at [service@deepseek.com](mailto:service@deepseek.com).\n","Janus-Series 是一个统一的多模态理解和生成模型。它支持任意到任意（any-to-any）的多模态转换，包括文本、图像等多种数据类型之间的相互转换和理解。该项目基于 Python 开发，利用了最新的基础模型技术，如大语言模型（LLM）和视觉-语言预训练（VLP），以实现高效且高质量的多模态处理能力。适用于需要跨模态信息处理的应用场景，比如多媒体内容生成、智能对话系统以及跨模态搜索等。其开源特性允许研究者和开发者根据自身需求进行定制化开发或实验。",2,"2026-06-11 02:50:34","top_language"]