[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72596":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":25,"readmeContent":26,"aiSummary":27,"trendingCount":16,"starSnapshotCount":16,"syncStatus":28,"lastSyncTime":29,"discoverSource":30},72596,"Qwen2-Audio","QwenLM\u002FQwen2-Audio","QwenLM","The official repo of Qwen2-Audio chat & pretrained large audio language model proposed by Alibaba Cloud.","",null,"Python",2078,165,32,107,0,4,6,11,12,28.66,false,"main",[],"2026-06-12 02:03:05","\u003Cp align=\"left\">\n        \u003Ca href=\"README_CN.md\">中文\u003C\u002Fa> &nbsp｜ &nbsp English&nbsp&nbsp\n\u003C\u002Fp>\n\u003Cbr>\u003Cbr>\n\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002Fassets\u002Fblog\u002Fqwenaudio\u002Fqwen2audio_logo.png\" width=\"400\"\u002F>\n\u003Cp>\n\n\u003Cp align=\"center\">\nQwen2-Audio-7B \u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002Fqwen\u002FQwen2-Audio-7B\">🤖 \u003C\u002Fa> | \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen2-Audio-7B\">🤗\u003C\u002Fa>&nbsp ｜ Qwen-Audio-7B-Instruct \u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fmodels\u002Fqwen\u002FQwen2-Audio-7B-Instruct\">🤖 \u003C\u002Fa>| \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen2-Audio-7B-Instruct\">🤗\u003C\u002Fa>&nbsp ｜ Demo\u003Ca href=\"https:\u002F\u002Fmodelscope.cn\u002Fstudios\u002Fqwen\u002FQwen2-Audio-Instruct-Demo\"> 🤖\u003C\u002Fa> | \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002FQwen\u002FQwen2-Audio-Instruct-Demo\">🤗\u003C\u002Fa>&nbsp\n\u003Cbr>\n📑 \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2407.10759\">Paper\u003C\u002Fa> &nbsp&nbsp | &nbsp&nbsp 📑 \u003Ca href=\"https:\u002F\u002Fqwenlm.github.io\u002Fblog\u002Fqwen2-audio\">Blog\u003C\u002Fa> &nbsp&nbsp | &nbsp&nbsp 💬 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002FQwenLM\u002FQwen\u002Fblob\u002Fmain\u002Fassets\u002Fwechat.png\">WeChat (微信)\u003C\u002Fa>&nbsp&nbsp | &nbsp&nbsp \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FCV4E9rpNSD\">Discord\u003C\u002Fa>&nbsp&nbsp\n\u003C\u002Fp>\n\n\nWe introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. We introduce two distinct audio interaction modes:\n\n* voice chat: users can freely engage in voice interactions with Qwen2-Audio without text input;\n* audio analysis: users could provide audio and text instructions for analysis during the interaction;\n\n**We've released two models of the Qwen2-Audio series: Qwen2-Audio-7B and Qwen2-Audio-7B-Instruct.**\n\n## Architecture\n\nThe overview of three-stage training process of Qwen2-Audio.\n\n\u003Cp align=\"center\">\n    \u003Cimg src=\"assets\u002Fframework.png\" width=\"80%\"\u002F>\n\u003Cp>\n\n## News and Updates\n* 2024.8.9 🎉 We released the checkpoints of both `Qwen2-Audio-7B` and `Qwen2-Audio-7B-Instruct` on ModelScope and Hugging Face.\n* 2024.7.15 🎉 We released the paper of **Qwen2-Audio**, introducing the relevant model structure, training methods, and model performance. Check our [report](https:\u002F\u002Farxiv.org\u002Fabs\u002F2407.10759) for details!\n* 2023.11.30 🔥  We released the **Qwen-Audio** series.\n\n\u003Cbr>\n\n## Evaluation\nWe evaluated the Qwen2-Audio's abilities on 13 standard benchmarks as follows:\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth>Task\u003C\u002Fth>\u003Cth>Description\u003C\u002Fth>\u003Cth>Dataset\u003C\u002Fth>\u003Cth>Split\u003C\u002Fth>\u003Cth>Metric\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd rowspan=\"4\">ASR\u003C\u002Ftd>\u003Ctd rowspan=\"4\">Automatic Speech Recognition\u003C\u002Ftd>\u003Ctd>Fleurs\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003Ctd rowspan=\"4\">WER\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Aishell2\u003C\u002Ftd>\u003Ctd>test\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Librispeech\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Common Voice\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>S2TT\u003C\u002Ftd>\u003Ctd>Speech-to-Text Translation\u003C\u002Ftd>\u003Ctd>CoVoST2\u003C\u002Ftd>\u003Ctd>test\u003C\u002Ftd>\u003Ctd>BLEU \u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SER\u003C\u002Ftd>\u003Ctd>Speech Emotion Recognition\u003C\u002Ftd>\u003Ctd>Meld\u003C\u002Ftd>\u003Ctd>test\u003C\u002Ftd>\u003Ctd>ACC\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>VSC\u003C\u002Ftd>\u003Ctd>Vocal Sound Classification\u003C\u002Ftd>\u003Ctd>VocalSound\u003C\u002Ftd>\u003Ctd>test\u003C\u002Ftd>\u003Ctd>ACC\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"4\">\u003Ca href=\"https:\u002F\u002Fgithub.com\u002FOFA-Sys\u002FAIR-Bench\">AIR-Bench\u003C\u002Fa>\u003Cbr>\u003C\u002Ftd>\u003Ctd>Chat-Benchmark-Speech\u003C\u002Ftd>\u003Ctd>Fisher\u003Cbr>SpokenWOZ\u003Cbr>IEMOCAP\u003Cbr>Common voice\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003Ctd>GPT-4 Eval\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Chat-Benchmark-Sound\u003C\u002Ftd>\u003Ctd>Clotho\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003Ctd>GPT-4 Eval\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>Chat-Benchmark-Music\u003C\u002Ftd>\u003Ctd>MusicCaps\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003Ctd>GPT-4 Eval\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Chat-Benchmark-Mixed-Audio\u003C\u002Ftd>\u003Ctd>Common voice\u003Cbr>AudioCaps\u003Cbr>MusicCaps\u003C\u002Ftd>\u003Ctd>dev | test\u003C\u002Ftd>\u003Ctd>GPT-4 Eval\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\n\n\nThe below is the overal performance:\n\u003Cp align=\"center\">\n    \u003Cimg src=\"assets\u002Fradar_compare_qwen_audio.png\" width=\"70%\"\u002F>\n\u003Cp>\n\nThe details of evaluation are as follows:\n\u003Cbr>\n\u003Cb>(Note: The evaluation results we present are based on the initial model of the original training framework. However, the scores showed some fluctuations after converting the framework to Huggingface. Here, we present our complete evaluation results, starting with the initial model results from the paper.)\u003C\u002Fb>\n\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth rowspan=\"2\">Task\u003C\u002Fth>\u003Cth rowspan=\"2\">Dataset\u003C\u002Fth>\u003Cth rowspan=\"2\">Model\u003C\u002Fth>\u003Cth colspan=\"2\">Performance\u003C\u002Fth>\u003C\u002Ftr>\u003Ctr>\u003Cth>Metrics\u003C\u002Fth>\u003Cth>Results\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd rowspan=\"15\">ASR\u003C\u002Ftd>\u003Ctd rowspan=\"7\">\u003Cb>Librispeech\u003C\u002Fb>\u003Cbr>dev-clean | dev-other | \u003Cbr>test-clean | test-other\u003C\u002Ftd>\u003Ctd>SpeechT5\u003C\u002Ftd>\u003Ctd rowspan=\"7\">WER \u003C\u002Ftd>\u003Ctd>2.1 | 5.5 | 2.4 | 5.8\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechNet\u003C\u002Ftd>\u003Ctd>- | - | 30.7 | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SLM-FT\u003C\u002Ftd>\u003Ctd>- | - | 2.6 | 5.0\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SALMONN\u003C\u002Ftd>\u003Ctd>- | - | 2.1 | 4.9\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechVerse\u003C\u002Ftd>\u003Ctd>- | - | 2.1 | 4.4\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>1.8 | 4.0 | 2.0 | 4.2\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>1.3 | 3.4 | 1.6 | 3.6\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"2\">\u003Cb>Common Voice 15\u003C\u002Fb> \u003Cbr>en | zh | yue | fr\u003C\u002Ftd>\u003Ctd>Whisper-large-v3\u003C\u002Ftd>\u003Ctd rowspan=\"2\">WER \u003C\u002Ftd>\u003Ctd>9.3 | 12.8 | 10.9 | 10.8\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>8.6 | 6.9 | 5.9 | 9.6\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd rowspan=\"2\">\u003Cb>Fleurs\u003C\u002Fb> \u003Cbr>zh\u003C\u002Ftd>\u003Ctd>Whisper-large-v3\u003C\u002Ftd>\u003Ctd rowspan=\"2\">WER \u003C\u002Ftd>\u003Ctd>7.7\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>7.5\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"4\">\u003Cb>Aishell2\u003C\u002Fb> \u003Cbr>Mic | iOS | Android\u003C\u002Ftd>\u003Ctd>MMSpeech-base\u003C\u002Ftd>\u003Ctd rowspan=\"4\">WER \u003C\u002Ftd>\u003Ctd>4.5 | 3.9 | 4.0\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Paraformer-large\u003C\u002Ftd>\u003Ctd>- | \u003Cb>2.9\u003C\u002Fb> | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>3.3 | 3.1 | 3.3\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>3.0\u003C\u002Fb> | 3.0 | \u003Cb>2.9\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"8\">S2TT\u003C\u002Ftd>\u003Ctd rowspan=\"5\">\u003Cb>CoVoST2\u003C\u002Fb> \u003Cbr>en-de | de-en | \u003Cbr>en-zh | zh-en\u003C\u002Ftd>\u003Ctd>SALMONN\u003C\u002Ftd>\u003Ctd rowspan=\"5\">BLEU \u003C\u002Ftd>\u003Ctd>18.6 | - | 33.1 | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechLLaMA\u003C\u002Ftd>\u003Ctd>- | 27.1 | - | 12.3\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>BLSP\u003C\u002Ftd>\u003Ctd>14.1 | - | - | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>25.1 | 33.9 | 41.5 | 15.7\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>29.9 | 35.2 | 45.2 | 24.4\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd rowspan=\"3\">\u003Cb>CoVoST2\u003C\u002Fb> \u003Cbr>es-en | fr-en | it-en |\u003C\u002Ftd>\u003Ctd>SpeechLLaMA\u003C\u002Ftd>\u003Ctd rowspan=\"3\">BLEU \u003C\u002Ftd>\u003Ctd>27.9 | 25.2 | 25.9\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>39.7 | \u003Cb>38.5\u003C\u002Fb> | 36.0\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>40.0 | 38.5 | 36.3\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"3\">SER\u003C\u002Ftd>\u003Ctd rowspan=\"3\">\u003Cb>Meld\u003C\u002Fb>\u003C\u002Ftd>\u003Ctd>WavLM-large\u003C\u002Ftd>\u003Ctd rowspan=\"3\">ACC \u003C\u002Ftd>\u003Ctd>0.542\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>0.557\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>0.553\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"4\">VSC\u003C\u002Ftd>\u003Ctd rowspan=\"4\">\u003Cb>VocalSound\u003C\u002Fb>\u003C\u002Ftd>\u003Ctd>CLAP\u003C\u002Ftd>\u003Ctd rowspan=\"4\">ACC \u003C\u002Ftd>\u003Ctd>0.4945\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Pengi\u003C\u002Ftd>\u003Ctd>0.6035\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>0.9289\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>0.9392\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>AIR-Bench \u003Cbr>\u003C\u002Ftd>\u003Ctd>\u003Cb>Chat Benchmark\u003C\u002Fb>\u003Cbr>Speech | Sound |\u003Cbr> Music | Mixed-Audio\u003C\u002Ftd>\u003Ctd>SALMONN\u003Cbr>BLSP\u003Cbr>Pandagpt\u003Cbr>Macaw-LLM\u003Cbr>SpeechGPT\u003Cbr>Next-gpt\u003Cbr>Qwen-Audio\u003Cbr>Gemini-1.5-pro\u003Cbr>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>GPT-4 \u003C\u002Ftd>\u003Ctd>6.16 | 6.28 | 5.95 | 6.08\u003Cbr>6.17 | 5.55 | 5.08 | 5.33\u003Cbr>3.58 | 5.46 | 5.06 | 4.25\u003Cbr>0.97 | 1.01 | 0.91 | 1.01\u003Cbr>1.57 | 0.95 | 0.95 | 4.13\u003Cbr>3.86 | 4.76 | 4.18 | 4.13\u003Cbr>6.47 | 6.95 | 5.52 | 6.08\u003Cbr>6.97 | 5.49 | 5.06 | 5.27\u003Cbr>\u003Cb>7.18 | 6.99 | 6.79 | 6.77\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\n\n\u003Cb>(Second is after converting huggingface)\u003C\u002Fb>\n\n\u003Ctable>\u003Cthead>\u003Ctr>\u003Cth rowspan=\"2\">Task\u003C\u002Fth>\u003Cth rowspan=\"2\">Dataset\u003C\u002Fth>\u003Cth rowspan=\"2\">Model\u003C\u002Fth>\u003Cth colspan=\"2\">Performance\u003C\u002Fth>\u003C\u002Ftr>\u003Ctr>\u003Cth>Metrics\u003C\u002Fth>\u003Cth>Results\u003C\u002Fth>\u003C\u002Ftr>\u003C\u002Fthead>\u003Ctbody>\u003Ctr>\u003Ctd rowspan=\"15\">ASR\u003C\u002Ftd>\u003Ctd rowspan=\"7\">\u003Cb>Librispeech\u003C\u002Fb>\u003Cbr>dev-clean | dev-other | \u003Cbr>test-clean | test-other\u003C\u002Ftd>\u003Ctd>SpeechT5\u003C\u002Ftd>\u003Ctd rowspan=\"7\">WER \u003C\u002Ftd>\u003Ctd>2.1 | 5.5 | 2.4 | 5.8\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechNet\u003C\u002Ftd>\u003Ctd>- | - | 30.7 | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SLM-FT\u003C\u002Ftd>\u003Ctd>- | - | 2.6 | 5.0\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SALMONN\u003C\u002Ftd>\u003Ctd>- | - | 2.1 | 4.9\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechVerse\u003C\u002Ftd>\u003Ctd>- | - | 2.1 | 4.4\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>1.8 | 4.0 | 2.0 | 4.2\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>1.7 | 3.6 | 1.7 | 4.0\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"2\">\u003Cb>Common Voice 15\u003C\u002Fb> \u003Cbr>en | zh | yue | fr\u003C\u002Ftd>\u003Ctd>Whisper-large-v3\u003C\u002Ftd>\u003Ctd rowspan=\"2\">WER \u003C\u002Ftd>\u003Ctd>9.3 | 12.8 | 10.9 | 10.8\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>8.7 | 6.5 | 5.9 | 9.6\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd rowspan=\"2\">\u003Cb>Fleurs\u003C\u002Fb> \u003Cbr>zh\u003C\u002Ftd>\u003Ctd>Whisper-large-v3\u003C\u002Ftd>\u003Ctd rowspan=\"2\">WER \u003C\u002Ftd>\u003Ctd>7.7\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>7.0\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"4\">\u003Cb>Aishell2\u003C\u002Fb> \u003Cbr>Mic | iOS | Android\u003C\u002Ftd>\u003Ctd>MMSpeech-base\u003C\u002Ftd>\u003Ctd rowspan=\"4\">WER \u003C\u002Ftd>\u003Ctd>4.5 | 3.9 | 4.0\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Paraformer-large\u003C\u002Ftd>\u003Ctd>- | \u003Cb>2.9\u003C\u002Fb> | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>3.3 | 3.1 | 3.3\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>3.2\u003C\u002Fb> | 3.1 | \u003Cb>2.9\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"8\">S2TT\u003C\u002Ftd>\u003Ctd rowspan=\"5\">\u003Cb>CoVoST2\u003C\u002Fb> \u003Cbr>en-de | de-en | \u003Cbr>en-zh | zh-en\u003C\u002Ftd>\u003Ctd>SALMONN\u003C\u002Ftd>\u003Ctd rowspan=\"5\">BLEU \u003C\u002Ftd>\u003Ctd>18.6 | - | 33.1 | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>SpeechLLaMA\u003C\u002Ftd>\u003Ctd>- | 27.1 | - | 12.3\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>BLSP\u003C\u002Ftd>\u003Ctd>14.1 | - | - | -\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>25.1 | \u003Cb>33.9\u003C\u002Fb> | 41.5 | 15.7\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>29.6\u003C\u002Fb> | 33.6 | \u003Cb>45.6\u003C\u002Fb> | \u003Cb>24.0\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd rowspan=\"3\">\u003Cb>CoVoST2\u003C\u002Fb> \u003Cbr>es-en | fr-en | it-en |\u003C\u002Ftd>\u003Ctd>SpeechLLaMA\u003C\u002Ftd>\u003Ctd rowspan=\"3\">BLEU \u003C\u002Ftd>\u003Ctd>27.9 | 25.2 | 25.9\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>39.7 | 38.5 | 36.0\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>38.7 | 37.2 | 35.2\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"3\">SER\u003C\u002Ftd>\u003Ctd rowspan=\"3\">\u003Cb>Meld\u003C\u002Fb>\u003C\u002Ftd>\u003Ctd>WavLM-large\u003C\u002Ftd>\u003Ctd rowspan=\"3\">ACC \u003C\u002Ftd>\u003Ctd>0.542\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>0.557\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>0.535\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd rowspan=\"4\">VSC\u003C\u002Ftd>\u003Ctd rowspan=\"4\">\u003Cb>VocalSound\u003C\u002Fb>\u003C\u002Ftd>\u003Ctd>CLAP\u003C\u002Ftd>\u003Ctd rowspan=\"4\">ACC \u003C\u002Ftd>\u003Ctd>0.4945\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Pengi\u003C\u002Ftd>\u003Ctd>0.6035\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen-Audio\u003C\u002Ftd>\u003Ctd>0.9289\u003C\u002Ftd>\u003C\u002Ftr>\u003Ctr>\u003Ctd>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>\u003Cb>0.9395\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\n\u003Ctr>\u003Ctd>AIR-Bench \u003Cbr>\u003C\u002Ftd>\u003Ctd>\u003Cb>Chat Benchmark\u003C\u002Fb>\u003Cbr>Speech | Sound |\u003Cbr> Music | Mixed-Audio\u003C\u002Ftd>\u003Ctd>SALMONN\u003Cbr>BLSP\u003Cbr>Pandagpt\u003Cbr>Macaw-LLM\u003Cbr>SpeechGPT\u003Cbr>Next-gpt\u003Cbr>Qwen-Audio\u003Cbr>Gemini-1.5-pro\u003Cbr>Qwen2-Audio\u003C\u002Ftd>\u003Ctd>GPT-4 \u003C\u002Ftd>\u003Ctd>6.16 | 6.28 | 5.95 | 6.08\u003Cbr>6.17 | 5.55 | 5.08 | 5.33\u003Cbr>3.58 | 5.46 | 5.06 | 4.25\u003Cbr>0.97 | 1.01 | 0.91 | 1.01\u003Cbr>1.57 | 0.95 | 0.95 | 4.13\u003Cbr>3.86 | 4.76 | 4.18 | 4.13\u003Cbr>6.47 | \u003Cb>6.95\u003C\u002Fb> | 5.52 | 6.08\u003Cbr>6.97 | 5.49 | 5.06 | 5.27\u003Cbr>\u003Cb>7.24\u003C\u002Fb> | 6.83 | \u003Cb>6.73\u003C\u002Fb> | \u003Cb>6.42\u003C\u002Fb>\u003C\u002Ftd>\u003C\u002Ftr>\u003C\u002Ftbody>\u003C\u002Ftable>\n\n\nWe have provided **all** evaluation scripts to reproduce our results. Please refer to [eval_audio\u002FEVALUATION.md](eval_audio\u002FEVALUATION.md) for details.\n\n## Requirements\nThe code of Qwen2-Audio has been in the latest Hugging face transformers and we advise you to build from source with command `pip install git+https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers`, or you might encounter the following error:\n```\nKeyError: 'qwen2-audio'\n```\n\n## Quickstart\nBelow, we provide simple examples to show how to use Qwen2-Audio and Qwen2-Audio-Instruct with 🤗 Transformers.\nBefore running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.\nNow you can start with ModelScope or Transformers. Qwen2-Audio models currently perform best with audio clips under 30 seconds.\n#### 🤗 Transformers\nIn the following, we demonstrate how to use `Qwen2-Audio-7B-Instruct` for the inference, supporting both voice chat and audio analysis modes. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose.\n\n##### Voice Chat Inference\nIn the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002Fguess_age_gender.wav\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"Yes, the speaker is female and in her twenties.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002Ftranslate_to_chinese.wav\"},\n    ]},\n]\ntext = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)\naudios = []\nfor message in conversation:\n    if isinstance(message[\"content\"], list):\n        for ele in message[\"content\"]:\n            if ele[\"type\"] == \"audio\":\n                audios.append(librosa.load(\n                    BytesIO(urlopen(ele['audio_url']).read()), \n                    sr=processor.feature_extractor.sampling_rate)[0]\n                )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n```\n\n##### Audio Analysis Inference\nIn the audio analysis, users could provide both audio and text instructions for analysis:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation = [\n    {'role': 'system', 'content': 'You are a helpful assistant.'}, \n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002Fglass-breaking-151256.mp3\"},\n        {\"type\": \"text\", \"text\": \"What's that sound?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"text\", \"text\": \"What can you do when you hear that?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"Stay alert and cautious, and check if anyone is hurt or if there is any damage to property.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002F1272-128104-0000.flac\"},\n        {\"type\": \"text\", \"text\": \"What does the person say?\"},\n    ]},\n]\ntext = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)\naudios = []\nfor message in conversation:\n    if isinstance(message[\"content\"], list):\n        for ele in message[\"content\"]:\n            if ele[\"type\"] == \"audio\":\n                audios.append(\n                    librosa.load(\n                        BytesIO(urlopen(ele['audio_url']).read()), \n                        sr=processor.feature_extractor.sampling_rate)[0]\n                )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n```\n\n##### Batch Inference\nWe also support batch inference:\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import Qwen2AudioForConditionalGeneration, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\")\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen\u002FQwen2-Audio-7B-Instruct\", device_map=\"auto\")\n\nconversation1 = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002Fglass-breaking-151256.mp3\"},\n        {\"type\": \"text\", \"text\": \"What's that sound?\"},\n    ]},\n    {\"role\": \"assistant\", \"content\": \"It is the sound of glass shattering.\"},\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002Ff2641_0_throatclearing.wav\"},\n        {\"type\": \"text\", \"text\": \"What can you hear?\"},\n    ]}\n]\n\nconversation2 = [\n    {\"role\": \"user\", \"content\": [\n        {\"type\": \"audio\", \"audio_url\": \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen2-Audio\u002Faudio\u002F1272-128104-0000.flac\"},\n        {\"type\": \"text\", \"text\": \"What does the person say?\"},\n    ]},\n]\n\nconversations = [conversation1, conversation2]\n\ntext = [processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) for conversation in conversations]\n\naudios = []\nfor conversation in conversations:\n    for message in conversation:\n        if isinstance(message[\"content\"], list):\n            for ele in message[\"content\"]:\n                if ele[\"type\"] == \"audio\":\n                    audios.append(\n                        librosa.load(\n                            BytesIO(urlopen(ele['audio_url']).read()), \n                            sr=processor.feature_extractor.sampling_rate)[0]\n                    )\n\ninputs = processor(text=text, audios=audios, return_tensors=\"pt\", padding=True)\ninputs['input_ids'] = inputs['input_ids'].to(\"cuda\")\ninputs.input_ids = inputs.input_ids.to(\"cuda\")\n\ngenerate_ids = model.generate(**inputs, max_length=256)\ngenerate_ids = generate_ids[:, inputs.input_ids.size(1):]\n\nresponse = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)\n```\nRunning Qwen2-Audio pretrained base model is also simple.\n```python\nfrom io import BytesIO\nfrom urllib.request import urlopen\nimport librosa\nfrom transformers import AutoProcessor, Qwen2AudioForConditionalGeneration\n\nmodel = Qwen2AudioForConditionalGeneration.from_pretrained(\"Qwen\u002FQwen2-Audio-7B\" ,trust_remote_code=True)\nprocessor = AutoProcessor.from_pretrained(\"Qwen\u002FQwen2-Audio-7B\" ,trust_remote_code=True)\n\nprompt = \"\u003C|audio_bos|>\u003C|AUDIO|>\u003C|audio_eos|>Generate the caption in English:\"\nurl = \"https:\u002F\u002Fqianwen-res.oss-cn-beijing.aliyuncs.com\u002FQwen-Audio\u002Fglass-breaking-151256.mp3\"\naudio, sr = librosa.load(BytesIO(urlopen(url).read()), sr=processor.feature_extractor.sampling_rate)\ninputs = processor(text=prompt, audios=audio, return_tensors=\"pt\")\n\ngenerated_ids = model.generate(**inputs, max_length=256)\ngenerated_ids = generated_ids[:, inputs.input_ids.size(1):]\nresponse = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n```\n#### 🤖 ModelScope\nWe strongly advise users especially those in mainland China to use ModelScope. `snapshot_download` can help you solve issues concerning downloading checkpoints.\n## Demo\n### Web UI\nWe provide code for users to build a web UI demo. Before you start, make sure you install the following packages:\n```\npip install -r requirements_web_demo.txt\n```\nThen run the command below and click on the generated link:\n```\npython demo\u002Fweb_demo_audio.py\n```\n\u003Cbr>\n\n## demos \nMore impressive cases will be updated on our blog at [Qwen's blog](https:\u002F\u002Fqwenlm.github.io\u002Fblog\u002Fqwen2-audio).\n\n## We Are Hiring\n\nIf you are interested in joining us as full-time or intern, please contact us at `qwen_audio@list.alibaba-inc.com`.\n\u003Cbr>\n\n## License Agreement\n\nCheck the license of each model inside its HF repo. It is NOT necessary for you to submit a request for commercial usage.\n\u003Cbr>\n\n## Citation\n\nIf you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :)\n\n```BibTeX\n@article{Qwen-Audio,\n  title={Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models},\n  author={Chu, Yunfei and Xu, Jin and Zhou, Xiaohuan and Yang, Qian and Zhang, Shiliang and Yan, Zhijie  and Zhou, Chang and Zhou, Jingren},\n  journal={arXiv preprint arXiv:2311.07919},\n  year={2023}\n}\n```\n\n```BibTeX\n@article{Qwen2-Audio,\n  title={Qwen2-Audio Technical Report},\n  author={Chu, Yunfei and Xu, Jin and Yang, Qian and Wei, Haojie and Wei, Xipin and Guo,  Zhifang and Leng, Yichong and Lv, Yuanjun and He, Jinzheng and Lin, Junyang and Zhou, Chang and Zhou, Jingren},\n  journal={arXiv preprint arXiv:2407.10759},\n  year={2024}\n}\n```\n\u003Cbr>\n\n## Contact Us\n\nIf you are interested to leave a message to either our research team or product team, feel free to send an email to `qianwen_opensource@alibabacloud.com`.\n\n","Qwen2-Audio 是由阿里云提出的大型音频语言模型，能够处理多种音频输入并进行语音聊天或音频分析。其核心功能包括两种音频交互模式：语音聊天和音频分析。在语音聊天模式下，用户可以直接与模型进行语音对话；在音频分析模式下，用户可以提供音频及文本指令以获取分析结果。该项目采用Python语言开发，具备强大的多模态处理能力。Qwen2-Audio适用于需要高质量语音识别、情感分析以及语音到文本转换的应用场景，如智能客服、语音助手等。",2,"2026-06-11 03:42:44","high_star"]