[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-9847":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":17,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":23,"hasPages":21,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":30,"readmeContent":31,"aiSummary":32,"trendingCount":16,"starSnapshotCount":16,"syncStatus":33,"lastSyncTime":34,"discoverSource":35},9847,"musiclm-pytorch","lucidrains\u002Fmusiclm-pytorch","lucidrains","Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch","",null,"Python",3291,264,93,25,0,1,3,60.07,"MIT License",false,"main",true,[25,26,27,28,29],"artificial-intelligence","attention-mechanisms","deep-learning","music-synthesis","transformers","2026-06-12 04:00:47","\u003Cimg src=\".\u002Fmusiclm.png\" width=\"450px\">\u003C\u002Fimg>\n\n## MusicLM - Pytorch\n\nImplementation of \u003Ca href=\"https:\u002F\u002Fgoogle-research.github.io\u002Fseanet\u002Fmusiclm\u002Fexamples\u002F\">MusicLM\u003C\u002Fa>, Google's new SOTA model for music generation using attention networks, in Pytorch.\n\nThey are basically using text-conditioned \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Flucidrains\u002Faudiolm-pytorch\">AudioLM\u003C\u002Fa>, but surprisingly with the embeddings from a text-audio contrastive learned model named \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fabs\u002F2208.12415\">MuLan\u003C\u002Fa>. MuLan is what will be built out in this repository, with AudioLM modified from the other repository to support the music generation needs here.\n\nPlease join \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FxBPBXfcFHd\">\u003Cimg alt=\"Join us on Discord\" src=\"https:\u002F\u002Fimg.shields.io\u002Fdiscord\u002F823813159592001537?color=5865F2&logo=discord&logoColor=white\">\u003C\u002Fa> if you are interested in helping out with the replication with the \u003Ca href=\"https:\u002F\u002Flaion.ai\u002F\">LAION\u003C\u002Fa> community\n\n\u003Ca href=\"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=jTrYIGxOuKQ\">What's AI by Louis Bouchard\u003C\u002Fa>\n\n## Appreciation\n\n- \u003Ca href=\"https:\u002F\u002Fstability.ai\u002F\">Stability.ai\u003C\u002Fa> for the generous sponsorship to work and open source cutting edge artificial intelligence research\n\n- \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002F\">🤗 Huggingface\u003C\u002Fa> for their \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Faccelerate\u002Findex\">accelerate\u003C\u002Fa> training library\n\n## Usage\n\n```install\n$ pip install musiclm-pytorch\n```\n\n## Usage\n\n`MuLaN` first needs to be trained\n\n```python\nimport torch\nfrom musiclm_pytorch import MuLaN, AudioSpectrogramTransformer, TextTransformer\n\naudio_transformer = AudioSpectrogramTransformer(\n    dim = 512,\n    depth = 6,\n    heads = 8,\n    dim_head = 64,\n    spec_n_fft = 128,\n    spec_win_length = 24,\n    spec_aug_stretch_factor = 0.8\n)\n\ntext_transformer = TextTransformer(\n    dim = 512,\n    depth = 6,\n    heads = 8,\n    dim_head = 64\n)\n\nmulan = MuLaN(\n    audio_transformer = audio_transformer,\n    text_transformer = text_transformer\n)\n\n# get a ton of \u003Csound, text> pairs and train\n\nwavs = torch.randn(2, 1024)\ntexts = torch.randint(0, 20000, (2, 256))\n\nloss = mulan(wavs, texts)\nloss.backward()\n\n# after much training, you can embed sounds and text into a joint embedding space\n# for conditioning the audio LM\n\nembeds = mulan.get_audio_latents(wavs)  # during training\n\nembeds = mulan.get_text_latents(texts)  # during inference\n```\n\nTo obtain the conditioning embeddings for the three transformers that are a part of `AudioLM`, you must use the `MuLaNEmbedQuantizer` as so\n\n```python\nfrom musiclm_pytorch import MuLaNEmbedQuantizer\n\n# setup the quantizer with the namespaced conditioning embeddings, unique per quantizer as well as namespace (per transformer)\n\nquantizer = MuLaNEmbedQuantizer(\n    mulan = mulan,                          # pass in trained mulan from above\n    conditioning_dims = (1024, 1024, 1024), # say all three transformers have model dimensions of 1024\n    namespaces = ('semantic', 'coarse', 'fine')\n)\n\n# now say you want the conditioning embeddings for semantic transformer\n\nwavs = torch.randn(2, 1024)\nconds = quantizer(wavs = wavs, namespace = 'semantic') # (2, 8, 1024) - 8 is number of quantizers\n```\n\nTo train (or finetune) the three transformers that are a part of `AudioLM`, you simply follow the instructions over at `audiolm-pytorch` for training, but pass in the `MulanEmbedQuantizer` instance to the training classes under the keyword `audio_conditioner`\n\nex. `SemanticTransformerTrainer`\n\n```python\nimport torch\nfrom audiolm_pytorch import HubertWithKmeans, SemanticTransformer, SemanticTransformerTrainer\n\nwav2vec = HubertWithKmeans(\n    checkpoint_path = '.\u002Fhubert\u002Fhubert_base_ls960.pt',\n    kmeans_path = '.\u002Fhubert\u002Fhubert_base_ls960_L9_km500.bin'\n)\n\nsemantic_transformer = SemanticTransformer(\n    num_semantic_tokens = wav2vec.codebook_size,\n    dim = 1024,\n    depth = 6,\n    audio_text_condition = True      # this must be set to True (same for CoarseTransformer and FineTransformers)\n).cuda()\n\ntrainer = SemanticTransformerTrainer(\n    transformer = semantic_transformer,\n    wav2vec = wav2vec,\n    audio_conditioner = quantizer,   # pass in the MulanEmbedQuantizer instance above\n    folder ='\u002Fpath\u002Fto\u002Faudio\u002Ffiles',\n    batch_size = 1,\n    data_max_length = 320 * 32,\n    num_train_steps = 1\n)\n\ntrainer.train()\n```\n\nAfter much training on all three transformers (semantic, coarse, fine), you will pass your finetuned or trained-from-scratch `AudioLM` and `MuLaN` wrapped in `MuLaNEmbedQuantizer` to the `MusicLM`\n\n```python\n# you need the trained AudioLM (audio_lm) from above\n# with the MulanEmbedQuantizer (mulan_embed_quantizer)\n\nfrom musiclm_pytorch import MusicLM\n\nmusiclm = MusicLM(\n    audio_lm = audio_lm,                 # `AudioLM` from https:\u002F\u002Fgithub.com\u002Flucidrains\u002Faudiolm-pytorch\n    mulan_embed_quantizer = quantizer    # the `MuLaNEmbedQuantizer` from above\n)\n\nmusic = musiclm('the crystalline sounds of the piano in a ballroom', num_samples = 4) # sample 4 and pick the top match with mulan\n```\n\n## Todo\n\n- [x] mulan seems to be using decoupled contrastive learning, offer that as an option\n- [x] wrap mulan with mulan wrapper and quantize the output, project to audiolm dimensions\n- [x] modify audiolm to accept conditioning embeddings, optionally take care of different dimensions through a separate projection\n- [x] audiolm and mulan goes into musiclm and generate, filter with mulan\n- [x] give dynamic positional bias to self attention in AST\n- [x] implement MusicLM generating multiple samples and selecting top match with MuLaN\n\n- [ ] support variable lengthed audio with masking in audio transformer\n- [ ] add a version of mulan to \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fmlfoundations\u002Fopen_clip\">open clip\u003C\u002Fa>\n- [ ] set all the proper spectrogram hyperparameters\n\n## Citations\n\n```bibtex\n@inproceedings{Agostinelli2023MusicLMGM,\n    title     = {MusicLM: Generating Music From Text},\n    author    = {Andrea Agostinelli and Timo I. Denk and Zal{\\'a}n Borsos and Jesse Engel and Mauro Verzetti and Antoine Caillon and Qingqing Huang and Aren Jansen and Adam Roberts and Marco Tagliasacchi and Matthew Sharifi and Neil Zeghidour and C. Frank},\n    year      = {2023}\n}\n```\n\n```bibtex\n@article{Huang2022MuLanAJ,\n    title   = {MuLan: A Joint Embedding of Music Audio and Natural Language},\n    author  = {Qingqing Huang and Aren Jansen and Joonseok Lee and Ravi Ganti and Judith Yue Li and Daniel P. W. Ellis},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs\u002F2208.12415}\n}\n```\n\n```bibtex\n@misc{https:\u002F\u002Fdoi.org\u002F10.48550\u002Farxiv.2302.01327,\n    doi     = {10.48550\u002FARXIV.2302.01327},\n    url     = {https:\u002F\u002Farxiv.org\u002Fabs\u002F2302.01327},\n    author  = {Kumar, Manoj and Dehghani, Mostafa and Houlsby, Neil},\n    title   = {Dual PatchNorm},\n    publisher = {arXiv},\n    year    = {2023},\n    copyright = {Creative Commons Attribution 4.0 International}\n}\n```\n\n```bibtex\n@article{Liu2022PatchDropoutEV,\n    title   = {PatchDropout: Economizing Vision Transformers Using Patch Dropout},\n    author  = {Yue Liu and Christos Matsoukas and Fredrik Strand and Hossein Azizpour and Kevin Smith},\n    journal = {ArXiv},\n    year    = {2022},\n    volume  = {abs\u002F2208.07220}\n}\n```\n\n```bibtex\n@misc{liu2021swin,\n    title   = {Swin Transformer V2: Scaling Up Capacity and Resolution},\n    author  = {Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},\n    year    = {2021},\n    eprint  = {2111.09883},\n    archivePrefix = {arXiv},\n    primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@misc{gilmer2023intriguing\n    title  = {Intriguing Properties of Transformer Training Instabilities},\n    author = {Justin Gilmer, Andrea Schioppa, and Jeremy Cohen},\n    year   = {2023},\n    status = {to be published - one attention stabilization technique is circulating within Google Brain, being used by multiple teams}\n}\n```\n\n```bibtex\n@inproceedings{Shukor2022EfficientVP,\n    title   = {Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment},\n    author  = {Mustafa Shukor and Guillaume Couairon and Matthieu Cord},\n    booktitle = {British Machine Vision Conference},\n    year    = {2022}\n}\n```\n\n```bibtex\n@inproceedings{Zhai2023SigmoidLF,\n    title   = {Sigmoid Loss for Language Image Pre-Training},\n    author  = {Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},\n    year    = {2023}\n}\n```\n\n*The only truth is music.* - Jack Kerouac\n\n*Music is the universal language of mankind.* - Henry Wadsworth Longfellow\n","lucidrains\u002Fmusiclm-pytorch 是一个基于 PyTorch 实现的音乐生成模型，该模型复现了 Google 的最新 SOTA 模型 MusicLM。它利用注意力机制和深度学习技术，特别是通过文本-音频对比学习模型 MuLan 生成的嵌入向量来指导音乐合成过程。项目支持使用条件化AudioLM进行音乐创作，适合需要高质量、多样化音乐内容生成的应用场景，如影视配乐、游戏音效设计或个性化音乐推荐系统等。此开源项目遵循 MIT 许可证，并得到了 Stability.ai 和 Huggingface 社区的支持与贡献。",2,"2026-06-11 03:24:59","top_topic"]