[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-70754":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":10,"archived":22,"fork":22,"defaultBranch":23,"hasWiki":24,"hasPages":22,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":26,"readmeContent":27,"aiSummary":28,"trendingCount":16,"starSnapshotCount":16,"syncStatus":29,"lastSyncTime":30,"discoverSource":31},70754,"Wav2Lip","Rudrabha\u002FWav2Lip","Rudrabha","This repository contains the codes of \"A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild\", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs ","https:\u002F\u002Fsync.so",null,"Python",13034,2831,182,340,0,4,16,54,12,45,false,"master",true,[],"2026-06-12 02:02:42","# **Wav2Lip**: *Accurately Lip-syncing Videos In The Wild* \n\n# Commercial Version\n\nCreate your first lipsync generation in minutes. Please note, the commercial version is of a much higher quality than the old open source model!\n\n## Create your API Key\n\nCreate your API key from the [Dashboard](https:\u002F\u002Fsync.so\u002Fkeys). You will use this key to securely access the Sync API.\n\n## Make your first generation\n\nThe following example shows how to make a lipsync generation using the Sync API.\n\n### Python\n\n#### Step 1: Install Sync SDK\n\n```bash\npip install syncsdk\n```\n\n#### Step 2: Make your first generation\n\nCopy the following code into a file `quickstart.py` and replace `YOUR_API_KEY_HERE` with your generated API key.\n\n```python\n# quickstart.py\nimport time\nfrom sync import Sync\nfrom sync.common import Audio, GenerationOptions, Video\nfrom sync.core.api_error import ApiError\n\n# ---------- UPDATE API KEY ----------\n# Replace with your Sync.so API key\napi_key = \"YOUR_API_KEY_HERE\" \n\n# ----------[OPTIONAL] UPDATE INPUT VIDEO AND AUDIO URL ----------\n# URL to your source video\nvideo_url = \"https:\u002F\u002Fassets.sync.so\u002Fdocs\u002Fexample-video.mp4\"\n# URL to your audio file\naudio_url = \"https:\u002F\u002Fassets.sync.so\u002Fdocs\u002Fexample-audio.wav\"\n# ----------------------------------------\n\nclient = Sync(\n    base_url=\"https:\u002F\u002Fapi.sync.so\", \n    api_key=api_key\n).generations\n\nprint(\"Starting lip sync generation job...\")\n\ntry:\n    response = client.create(\n        input=[Video(url=video_url),Audio(url=audio_url)],\n        model=\"lipsync-2\",\n        options=GenerationOptions(sync_mode=\"cut_off\"),\n        outputFileName=\"quickstart\"\n    )\nexcept ApiError as e:\n    print(f'create generation request failed with status code {e.status_code} and error {e.body}')\n    exit()\n\njob_id = response.id\nprint(f\"Generation submitted successfully, job id: {job_id}\")\n\ngeneration = client.get(job_id)\nstatus = generation.status\nwhile status not in ['COMPLETED', 'FAILED']:\n    print('polling status for generation', job_id)\n    time.sleep(10)\n    generation = client.get(job_id)\n    status = generation.status\n\nif status == 'COMPLETED':\n    print('generation', job_id, 'completed successfully, output url:', generation.output_url)\nelse:\n    print('generation', job_id, 'failed')\n```\n\nRun the script:\n\n```bash\npython quickstart.py\n```\n\n#### Step 3: Done!\n\nIt may take a few minutes for the generation to complete. You should see the generated video URL in the terminal post completion.\n\n---\n\n### TypeScript\n\n#### Step 1: Install dependencies\n\n```bash\nnpm i @sync.so\u002Fsdk\n```\n\n#### Step 2: Make your first generation\n\nCopy the following code into a file `quickstart.ts` and replace `YOUR_API_KEY_HERE` with your generated API key.\n\n```typescript\n\u002F\u002F quickstart.ts\nimport { SyncClient, SyncError } from \"@sync.so\u002Fsdk\";\n\n\u002F\u002F ---------- UPDATE API KEY ----------\n\u002F\u002F Replace with your Sync.so API key\nconst apiKey = \"YOUR_API_KEY_HERE\";\n\n\u002F\u002F ----------[OPTIONAL] UPDATE INPUT VIDEO AND AUDIO URL ----------\n\u002F\u002F URL to your source video\nconst videoUrl = \"https:\u002F\u002Fassets.sync.so\u002Fdocs\u002Fexample-video.mp4\";\n\u002F\u002F URL to your audio file\nconst audioUrl = \"https:\u002F\u002Fassets.sync.so\u002Fdocs\u002Fexample-audio.wav\";\n\u002F\u002F ----------------------------------------\n\nconst client = new SyncClient({ apiKey });\n\nasync function main() {\n    console.log(\"Starting lip sync generation job...\");\n\n    let jobId: string;\n    try {\n        const response = await client.generations.create({\n            input: [\n                {\n                    type: \"video\",\n                    url: videoUrl,\n                },\n                {\n                    type: \"audio\",\n                    url: audioUrl,\n                },\n            ],\n            model: \"lipsync-2\",\n            options: {\n                sync_mode: \"cut_off\",\n            },\n            outputFileName: \"quickstart\"\n        });\n        jobId = response.id;\n        console.log(`Generation submitted successfully, job id: ${jobId}`);\n    } catch (err) {\n        if (err instanceof SyncError) {\n            console.error(`create generation request failed with status code ${err.statusCode} and error ${JSON.stringify(err.body)}`);\n        } else {\n            console.error('An unexpected error occurred:', err);\n        }\n        return;\n    }\n\n    let generation;\n    let status;\n    while (status !== 'COMPLETED' && status !== 'FAILED') {\n        console.log(`polling status for generation ${jobId}...`);\n        try {\n            await new Promise(resolve => setTimeout(resolve, 10000));\n            generation = await client.generations.get(jobId);\n            status = generation.status;\n        } catch (err) {\n            if (err instanceof SyncError) {\n                console.error(`polling failed with status code ${err.statusCode} and error ${JSON.stringify(err.body)}`);\n            } else {\n                console.error('An unexpected error occurred during polling:', err);\n            }\n            status = 'FAILED';\n        }\n    }\n\n    if (status === 'COMPLETED') {\n        console.log(`generation ${jobId} completed successfully, output url: ${generation?.outputUrl}`);\n    } else {\n        console.log(`generation ${jobId} failed`);\n    }\n}\n\nmain();\n```\n\nRun the script:\n\n```bash\nnpx tsx quickstart.ts -y\n```\n\n#### Step 3: Done!\n\nYou should see the generated video URL in the terminal.\n\n---\n\n## Next Steps\n\nWell done! You've just made your first lipsync generation with sync.so!\n\nReady to unlock the full potential of lipsync? Dive into our interactive [Studio](https:\u002F\u002Fsync.so\u002Flogin) to experiment with all available models, or explore our [API Documentation](\u002Fapi-reference) to take your lip-sync generations to the next level!\n\n## Contact\n- prady@sync.so\n- pavan@sync.so\n- sanjit@sync.so\n\n\n\n# Non Commercial Open-source Version\n\nThis code is part of the paper: _A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild_ published at ACM Multimedia 2020. \n[![PWC](https:\u002F\u002Fimg.shields.io\u002Fendpoint.svg?url=https:\u002F\u002Fpaperswithcode.com\u002Fbadge\u002Fa-lip-sync-expert-is-all-you-need-for-speech\u002Flip-sync-on-lrs2)](https:\u002F\u002Fpaperswithcode.com\u002Fsota\u002Flip-sync-on-lrs2?p=a-lip-sync-expert-is-all-you-need-for-speech)\n[![PWC](https:\u002F\u002Fimg.shields.io\u002Fendpoint.svg?url=https:\u002F\u002Fpaperswithcode.com\u002Fbadge\u002Fa-lip-sync-expert-is-all-you-need-for-speech\u002Flip-sync-on-lrs3)](https:\u002F\u002Fpaperswithcode.com\u002Fsota\u002Flip-sync-on-lrs3?p=a-lip-sync-expert-is-all-you-need-for-speech)\n[![PWC](https:\u002F\u002Fimg.shields.io\u002Fendpoint.svg?url=https:\u002F\u002Fpaperswithcode.com\u002Fbadge\u002Fa-lip-sync-expert-is-all-you-need-for-speech\u002Flip-sync-on-lrw)](https:\u002F\u002Fpaperswithcode.com\u002Fsota\u002Flip-sync-on-lrw?p=a-lip-sync-expert-is-all-you-need-for-speech)\n|📑 Original Paper|📰 Project Page|🌀 Demo|⚡ Live Testing|📔 Colab Notebook\n|:-:|:-:|:-:|:-:|:-:|\n[Paper](http:\u002F\u002Farxiv.org\u002Fabs\u002F2008.10010) | [Project Page](http:\u002F\u002Fcvit.iiit.ac.in\u002Fresearch\u002Fprojects\u002Fcvit-projects\u002Fa-lip-sync-expert-is-all-you-need-for-speech-to-lip-generation-in-the-wild\u002F) | [Demo Video](https:\u002F\u002Fyoutu.be\u002F0fXaDCZNOJc) | [Interactive Demo](https:\u002F\u002Fsynclabs.so\u002F) | [Colab Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1tZpDWXz49W6wDcTprANRGLo2D_EbD5J8?usp=sharing) \u002F[Updated Collab Notebook](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1IjFW1cLevs6Ouyu4Yht4mnR4yeuMqO7Y#scrollTo=MH1m608OymLH)\n \n![Logo](https:\u002F\u002Fdrive.google.com\u002Fuc?export=view&id=1Wn0hPmpo4GRbCIJR8Tf20Akzdi1qjjG9)\n----------\n**Highlights**\n----------\n - Weights of the visual quality disc has been updated in readme!\n - Lip-sync videos to any target speech with high accuracy :100:. Try our [interactive demo](https:\u002F\u002Fsync.so\u002F).\n - :sparkles: Works for any identity, voice, and language. Also works for CGI faces and synthetic voices.\n - Complete training code, inference code, and pretrained models are available :boom:\n - Or, quick-start with the Google Colab Notebook: [Link](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1tZpDWXz49W6wDcTprANRGLo2D_EbD5J8?usp=sharing). Checkpoints and samples are available in a Google Drive [folder](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1I-0dNLfFOSFwrfqjNa-SXuwaURHE5K4k?usp=sharing) as well. There is also a [tutorial video](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=Ic0TBhfuOrA) on this, courtesy of [What Make Art](https:\u002F\u002Fwww.youtube.com\u002Fchannel\u002FUCmGXH-jy0o2CuhqtpxbaQgA). Also, thanks to [Eyal Gruss](https:\u002F\u002Feyalgruss.com), there is a more accessible [Google Colab notebook](https:\u002F\u002Fj.mp\u002Fwav2lip) with more useful features. A tutorial collab notebook is present at this [link](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1IjFW1cLevs6Ouyu4Yht4mnR4yeuMqO7Y#scrollTo=MH1m608OymLH).  \n - :fire: :fire: Several new, reliable evaluation benchmarks and metrics [[`evaluation\u002F` folder of this repo]](https:\u002F\u002Fgithub.com\u002FRudrabha\u002FWav2Lip\u002Ftree\u002Fmaster\u002Fevaluation) released. Instructions to calculate the metrics reported in the paper are also present.\n--------\n**Disclaimer**\n--------\nAll results from this open-source code or our [demo website](https:\u002F\u002Fbhaasha.iiit.ac.in\u002Flipsync) should only be used for research\u002Facademic\u002Fpersonal purposes only. As the models are trained on the \u003Ca href=\"http:\u002F\u002Fwww.robots.ox.ac.uk\u002F~vgg\u002Fdata\u002Flip_reading\u002Flrs2.html\">LRS2 dataset\u003C\u002Fa>, any form of commercial use is strictly prohibited. For commercial requests please contact us directly!\nPrerequisites\n-------------\n- `Python 3.6` \n- ffmpeg: `sudo apt-get install ffmpeg`\n- Install necessary packages using `pip install -r requirements.txt`. Alternatively, instructions for using a docker image is provided [here](https:\u002F\u002Fgist.github.com\u002Fxenogenesi\u002Fe62d3d13dadbc164124c830e9c453668). Have a look at [this comment](https:\u002F\u002Fgithub.com\u002FRudrabha\u002FWav2Lip\u002Fissues\u002F131#issuecomment-725478562) and comment on [the gist](https:\u002F\u002Fgist.github.com\u002Fxenogenesi\u002Fe62d3d13dadbc164124c830e9c453668) if you encounter any issues. \n- Face detection [pre-trained model](https:\u002F\u002Fwww.adrianbulat.com\u002Fdownloads\u002Fpython-fan\u002Fs3fd-619a316812.pth) should be downloaded to `face_detection\u002Fdetection\u002Fsfd\u002Fs3fd.pth`. Alternative [link](https:\u002F\u002Fiiitaphyd-my.sharepoint.com\u002F:u:\u002Fg\u002Fpersonal\u002Fprajwal_k_research_iiit_ac_in\u002FEZsy6qWuivtDnANIG73iHjIBjMSoojcIV0NULXV-yiuiIg?e=qTasa8) if the above does not work.\nGetting the weights\n----------\n| Model  | Description |  Link to the model | \n| :-------------: | :---------------: | :---------------: |\n| Wav2Lip  | Highly accurate lip-sync | [Link](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F153HLrqlBNxzZcHi17PEvP09kkAfzRshM?usp=share_link)  |\n| Wav2Lip + GAN  | Slightly inferior lip-sync, but better visual quality | [Link](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F15G3U08c8xsCkOqQxE38Z2XXDnPcOptNk\u002Fview?usp=share_link) |\n\n\nLip-syncing videos using the pre-trained models (Inference)\n-------\nYou can lip-sync any video to any audio:\n```bash\npython inference.py --checkpoint_path \u003Cckpt> --face \u003Cvideo.mp4> --audio \u003Can-audio-source> \n```\nThe result is saved (by default) in `results\u002Fresult_voice.mp4`. You can specify it as an argument,  similar to several other available options. The audio source can be any file supported by `FFMPEG` containing audio data: `*.wav`, `*.mp3` or even a video file, from which the code will automatically extract the audio.\n##### Tips for better results:\n- Experiment with the `--pads` argument to adjust the detected face bounding box. Often leads to improved results. You might need to increase the bottom padding to include the chin region. E.g. `--pads 0 20 0 0`.\n- If you see the mouth position dislocated or some weird artifacts such as two mouths, then it can be because of over-smoothing the face detections. Use the `--nosmooth` argument and give it another try. \n- Experiment with the `--resize_factor` argument, to get a lower-resolution video. Why? The models are trained on faces that were at a lower resolution. You might get better, visually pleasing results for 720p videos than for 1080p videos (in many cases, the latter works well too). \n- The Wav2Lip model without GAN usually needs more experimenting with the above two to get the most ideal results, and sometimes, can give you a better result as well.\nPreparing LRS2 for training\n----------\nOur models are trained on LRS2. See [here](#training-on-datasets-other-than-lrs2) for a few suggestions regarding training on other datasets.\n##### LRS2 dataset folder structure\n```\ndata_root (mvlrs_v1)\n├── main, pretrain (we use only main folder in this work)\n|\t├── list of folders\n|\t│   ├── five-digit numbered video IDs ending with (.mp4)\n```\nPlace the LRS2 filelists (train, val, test) `.txt` files in the `filelists\u002F` folder.\n##### Preprocess the dataset for fast training\n```bash\npython preprocess.py --data_root data_root\u002Fmain --preprocessed_root lrs2_preprocessed\u002F\n```\nAdditional options like `batch_size` and the number of GPUs to use in parallel to use can also be set.\n##### Preprocessed LRS2 folder structure\n```\npreprocessed_root (lrs2_preprocessed)\n├── list of folders\n|\t├── Folders with five-digit numbered video IDs\n|\t│   ├── *.jpg\n|\t│   ├── audio.wav\n```\nTrain!\n----------\nThere are two major steps: (i) Train the expert lip-sync discriminator, (ii) Train the Wav2Lip model(s).\n##### Training the expert discriminator\nYou can download [the pre-trained weights](#getting-the-weights) if you want to skip this step. To train it:\n```bash\npython color_syncnet_train.py --data_root lrs2_preprocessed\u002F --checkpoint_dir \u003Cfolder_to_save_checkpoints>\n```\n##### Training the Wav2Lip models\nYou can either train the model without the additional visual quality discriminator (\u003C 1 day of training) or use the discriminator (~2 days). For the former, run: \n```bash\npython wav2lip_train.py --data_root lrs2_preprocessed\u002F --checkpoint_dir \u003Cfolder_to_save_checkpoints> --syncnet_checkpoint_path \u003Cpath_to_expert_disc_checkpoint>\n```\nTo train with the visual quality discriminator, you should run `hq_wav2lip_train.py` instead. The arguments for both files are similar. In both cases, you can resume training as well. Look at `python wav2lip_train.py --help` for more details. You can also set additional less commonly-used hyper-parameters at the bottom of the `hparams.py` file.\nTraining on datasets other than LRS2\n------------------------------------\nTraining on other datasets might require modifications to the code. Please read the following before you raise an issue:\n- You might not get good results by training\u002Ffine-tuning on a few minutes of a single speaker. This is a separate research problem, to which we do not have a solution yet. Thus, we would most likely not be able to resolve your issue. \n- You must train the expert discriminator for your own dataset before training Wav2Lip.\n- If it is your own dataset downloaded from the web, in most cases, needs to be sync-corrected.\n- Be mindful of the FPS of the videos of your dataset. Changes to FPS would need significant code changes. \n- The expert discriminator's eval loss should go down to ~0.25 and the Wav2Lip eval sync loss should go down to ~0.2 to get good results. \nWhen raising an issue on this topic, please let us know that you are aware of all these points.\nWe have an HD model trained on a dataset allowing commercial usage. The size of the generated face will be 192 x 288 in our new model.\nEvaluation\n----------\nPlease check the `evaluation\u002F` folder for the instructions.\nLicense and Citation\n----------\nThis repository can only be used for personal\u002Fresearch\u002Fnon-commercial purposes. However, for commercial requests, please contact us directly at rudrabha@synclabs.so or prajwal@synclabs.so. We have a turn-key hosted API with new and improved lip-syncing models here: https:\u002F\u002Fsynclabs.so\u002F\nThe size of the generated face will be 192 x 288 in our new models. Please cite the following paper if you use this repository:\n```\n@inproceedings{10.1145\u002F3394171.3413532,\nauthor = {Prajwal, K R and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P. and Jawahar, C.V.},\ntitle = {A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild},\nyear = {2020},\nisbn = {9781450379885},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https:\u002F\u002Fdoi.org\u002F10.1145\u002F3394171.3413532},\ndoi = {10.1145\u002F3394171.3413532},\nbooktitle = {Proceedings of the 28th ACM International Conference on Multimedia},\npages = {484–492},\nnumpages = {9},\nkeywords = {lip sync, talking face generation, video generation},\nlocation = {Seattle, WA, USA},\nseries = {MM '20}\n}\n```\nAcknowledgments\n----------\nParts of the code structure are inspired by this [TTS repository](https:\u002F\u002Fgithub.com\u002Fr9y9\u002Fdeepvoice3_pytorch). We thank the author for this wonderful code. The code for Face Detection has been taken from the [face_alignment](https:\u002F\u002Fgithub.com\u002F1adrianb\u002Fface-alignment) repository. We thank the authors for releasing their code and models. We thank [zabique](https:\u002F\u002Fgithub.com\u002Fzabique) for the tutorial collab notebook.\n## Acknowledgements\n - [Awesome Readme Templates](https:\u002F\u002Fawesomeopensource.com\u002Fproject\u002Felangosundar\u002Fawesome-README-templates)\n - [Awesome README](https:\u002F\u002Fgithub.com\u002Fmatiassingers\u002Fawesome-readme)\n - [How to write a Good readme](https:\u002F\u002Fbulldogjob.com\u002Fnews\u002F449-how-to-write-a-good-readme-for-your-github-project)\n","Wav2Lip 是一个用于生成高质量唇形同步视频的工具。该项目基于Python开发，能够根据输入的音频自动生成与之匹配的口型动作视频，特别适用于演讲、电影制作或虚拟角色动画等场景中提高内容的真实性和吸引力。其核心技术特点在于即使在复杂多变的现实环境中也能实现准确的唇部动作合成，为用户提供了一种简便快捷的方法来创建高度自然的视觉效果。此外，项目还提供了商业版本和API服务，以满足更专业的需求。",2,"2026-06-11 03:34:00","high_star"]