[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72392":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":17,"stars30d":17,"stars90d":16,"forks30d":16,"starsTrendScore":18,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":22,"defaultBranch":23,"hasWiki":22,"hasPages":22,"topics":24,"createdAt":10,"pushedAt":10,"updatedAt":25,"readmeContent":26,"aiSummary":27,"trendingCount":16,"starSnapshotCount":16,"syncStatus":14,"lastSyncTime":28,"discoverSource":29},72392,"audio2photoreal","facebookresearch\u002Faudio2photoreal","facebookresearch","Code and dataset for photorealistic Codec Avatars driven from audio","",null,"Python",2855,282,2,6,0,1,3,61.96,"Other",true,false,"main",[],"2026-06-12 04:01:04","# From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations\nThis repository contains a pytorch implementation of [\"From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations\"](https:\u002F\u002Fevonneng.github.io\u002Fprojects\u002Faudio2photoreal\u002Findex.html)\n\n:hatching_chick: **Try out our demo [here](https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1A6EwKM3PeX7dcKV66zxQWuP-v_dKlX_0?usp=sharing)** or continue following the steps below to run code locally!\nAnd thanks everyone for the support via contributions\u002Fcomments\u002Fissues!\n\nhttps:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Faudio2photoreal\u002Fassets\u002F17986358\u002F5cba4079-275e-48b6-aecc-f84f3108c810\n\nThis codebase provides:\n- train code\n- test code\n- pretrained motion models\n- access to dataset\n\nIf you use the dataset or code, please cite our [Paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.01885)\n\n```\n@inproceedings{ng2024audio2photoreal,\n  title={From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations},\n  author={Ng, Evonne and Romero, Javier and Bagautdinov, Timur and Bai, Shaojie and Darrell, Trevor and Kanazawa, Angjoo and Richard, Alexander},\n  booktitle={IEEE Conference on Computer Vision and Pattern Recognition},\n  year={2024}\n}\n```\n\n### Repository Contents\n\n- [**Quickstart:**](#quickstart) easy gradio demo that lets you record audio and render a video\n- [**Installation:**](#installation) environment setup and installation (for more details on the rendering pipeline, please refer to [Codec Avatar Body](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fca_body))\n- [**Download data and models:**](#download-data-and-models) download annotations and pre-trained models\n    - [Dataset desc.](#dataset): description of dataset annotations\n    - [Visualize Dataset](#visualize-ground-truth): script for visualizing ground truth annotations\n    - [model desc.](#pretrained-models): description of pretrained models\n- [**Running the pretrained models:**](#running-the-pretrained-models) how to generate results files and visualize the results using the rendering pipeline.\n    - [Face generation](#face-generation): commands to generate the results file for the faces\n    - [Body generation](#body-generation): commands to generate the results file for the bodies\n    - [Visualization](#visualization): how to call into the rendering api. For full details, please refer to [this repo](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fca_body).\n- [**Training from scratch (3 models):**](#training-from-scratch) scripts to get the training pipeline running from scratch for face, guide poses, and body models.\n    - [Face diffusion model](#1-face-diffusion-model)\n    - [Body diffusion](#2-body-diffusion-model)\n    - [Body vq vae](#3-body-vq-vae)\n    - [Body guide transformer](#4-body-guide-transformer)\n\nWe annotate code that you can directly copy and paste into your terminal using the :point_down: icon.\n\n# Quickstart\nWith this demo, you can record an audio clip and select the number of samples you want to generate. \n\nMake sure you have CUDA 11.7 and gcc\u002F++ 9.0 for pytorch3d compatibility\n\n:point_down: Install necessary components. This will do the environment configuration and install the corresponding rendering assets, prerequisite models, and pretrained models:\n```\nconda create --name a2p_env python=3.9\nconda activate a2p_env\nsh demo\u002Finstall.sh\n```\n:point_down: Run the demo. You can record your audio and then render corresponding results!\n```\npython -m demo.demo\n```\n\n:microphone: First, record your audio\n\n![](assets\u002Fdemo1.gif)\n\n:hourglass: Hold tight because the rendering can take a while! \n\nYou can change the number of samples (1-10) you want to generate, and download your favorite video by clicking on the download button on the top right of each video.\n\n![](assets\u002Fdemo2.gif)\n\n# Installation\nThe code has been tested with CUDA 11.7 and python 3.9, gcc\u002F++ 9.0\n\n:point_down: If you haven't done so already via the demo setup, configure the environments and download prerequisite models:\n```\nconda create --name a2p_env python=3.9\nconda activate a2p_env\npip install -r scripts\u002Frequirements.txt\nsh scripts\u002Fdownload_prereq.sh\n```\n:point_down: To get the rendering working, please also make sure you install [pytorch3d](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fpytorch3d\u002Fblob\u002Fmain\u002FINSTALL.md). \n```\npip install \"git+https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fpytorch3d.git\"\n```\nPlease see [CA Bodies repo](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fca_body) for more details on the renderer.\n\n# Download data and models\nTo download any of the datasets, you can find them at `https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Faudio2photoreal\u002Freleases\u002Fdownload\u002Fv1.0\u002F\u003Cperson_id>.zip`, where you can replace `\u003Cperson_id>` with any of `PXB184`, `RLW104`, `TXB805`, or `GQS883`.\nDownload over the command line can be done with this commands.\n```\ncurl -L https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Faudio2photoreal\u002Freleases\u002Fdownload\u002Fv1.0\u002F\u003Cperson_id>.zip -o \u003Cperson_id>.zip\nunzip \u003Cperson_id>.zip -d dataset\u002F\nrm \u003Cperson_id>.zip\n```\n:point_down: To download *all* of the datasets, you can simply run the following which will download and unpack all the models.\n```\nsh scripts\u002Fdownload_alldatasets.sh\n```\n\nSimilarly, to download any of the models, you can find them at `http:\u002F\u002Faudio2photoreal_models.berkeleyvision.org\u002F\u003Cperson_id>_models.tar`.\n```\n# download the motion generation\nwget http:\u002F\u002Faudio2photoreal_models.berkeleyvision.org\u002F\u003Cperson_id>_models.tar\ntar xvf \u003Cperson_id>_models.tar\nrm \u003Cperson_id>_models.tar\n\n# download the body decoder\u002Frendering assets and place them in the right place\nmkdir -p checkpoints\u002Fca_body\u002Fdata\u002F\nwget https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fca_body\u002Freleases\u002Fdownload\u002Fv0.0.1-alpha\u002F\u003Cperson_id>.tar.gz\ntar xvf \u003Cperson_id>.tar.gz --directory checkpoints\u002Fca_body\u002Fdata\u002F\nrm \u003Cperson_id>.tar.gz\n```\n:point_down: You can also download all of the models with this script:\n```\nsh scripts\u002Fdownload_allmodels.sh\n```\nThe above model script will download *both* the models for motion generation and the body decoder\u002Frendering models. Please view the script for more details.\n\n### Dataset \nOnce the dataset is downloaded and unzipped (via `scripts\u002Fdownload_datasets.sh`), it should unfold into the following directory structure:\n```\n|-- dataset\u002F\n    |-- PXB184\u002F\n        |-- data_stats.pth \n        |-- scene01_audio.wav\n        |-- scene01_body_pose.npy\n        |-- scene01_face_expression.npy\n        |-- scene01_missing_face_frames.npy\n        |-- ...\n        |-- scene30_audio.wav\n        |-- scene30_body_pose.npy\n        |-- scene30_face_expression.npy\n        |-- scene30_missing_face_frames.npy\n    |-- RLW104\u002F\n    |-- TXB805\u002F\n    |-- GQS883\u002F\n```\nEach of the four participants (`PXB184`, `RLW104`, `TXB805`, `GQS883`) should have independent \"scenes\" (1 to 26 or so).\nFor each scene, there are 3 types of data annotations that we save. \n```\n*audio.wav: wavefile containing the raw audio (two channels, 1600*T samples) at 48kHz; channel 0 is the audio associated with the current person, channel 1 is the audio associated with their conversational partner.\n\n*body_pose.npy: (T x 104) array of joint angles in a kinematic skeleton. Not all of the joints are represented with 3DoF. Each 104-d vector can be used to reconstruct a full-body skeleton.\n\n*face_expression.npy: (T x 256) array of facial codes, where each 256-d vector reconstructs a face mesh.\n\n*missing_face_frames.npy: List of indices (t) where the facial code is missing or corrupted. \n\ndata_stats.pth: carries the mean and std for each modality of each person.\n```\n\nFor the train\u002Fval\u002Ftest split the indices are defined in `data_loaders\u002Fdata.py` as:\n```\ntrain_idx = list(range(0, len(data_dict[\"data\"]) - 6))\nval_idx = list(range(len(data_dict[\"data\"]) - 6, len(data_dict[\"data\"]) - 4))\ntest_idx = list(range(len(data_dict[\"data\"]) - 4, len(data_dict[\"data\"])))\n```\nfor any of the four dataset participants we train on.\n\n### Visualize ground truth\nIf you've properly installed the rendering requirements, you can then visualize the full dataset with the following command:\n```\npython -m visualize.render_anno \n    --save_dir \u003Cpath\u002Fto\u002Fsave\u002Fdir> \n    --data_root \u003Cpath\u002Fto\u002Fdata\u002Froot> \n    --max_seq_length \u003Cnum>\n```\n\nThe videos will be chunked lengths according to specified `--max_seq_length` arg, which you can specify (the default is 600).\n\n:point_down: For example, to visualize ground truth annotations for `PXB184`, you can run the following.\n```\npython -m visualize.render_anno --save_dir vis_anno_test --data_root dataset\u002FPXB184 --max_seq_length 600\n```\n\n### Pretrained models\nWe train person-specific models, so each person should have an associated directory. For instance, for `PXB184`, their complete models should unzip into the following structure.\n```\n|-- checkpoints\u002F\n    |-- diffusion\u002F\n        |-- c1_face\u002F\n            |-- args.json\n            |-- model:09d.pt\n        |-- c1_pose\u002F\n            |-- args.json\n            |-- model:09d.pt\n    |-- guide\u002F\n        |-- c1_pose\u002F\n            |-- args.json\n            |-- checkpoints\u002F\n                |-- iter-:07d.pt\n    |-- vq\u002F\n        |-- c1_pose\u002F\n            |-- args.json\n            |-- net_iter:06d.pth\n```\nThere are 4 models for each person and each model has an associated `args.json`.\n1. a face diffusion model that outputs 256 facial codes conditioned on audio\n2. a pose diffusion model that outputs 104 joint rotations conditioned on audio and guide poses\n3. a guide vq pose model that outputs vq tokens conditioned on audio at 1 fps\n4. a vq encoder-decoder model that vector quantizes the continuous 104-d pose space.\n\n# Running the pretrained models \nTo run the actual models, you will need to run the pretrained models and generate the associated results files before visualizing them. \n\n### Face generation\nTo generate the results file for the face, \n```\npython -m sample.generate \n    --model_path \u003Cpath\u002Fto\u002Fmodel> \n    --num_samples \u003Cxsamples> \n    --num_repetitions \u003Cxreps> \n    --timestep_respacing ddim500 \n    --guidance_param 10.0\n```\n\nThe `\u003Cpath\u002Fto\u002Fmodel>` should be the path to the diffusion model that is associated with generating the face. \nE.g. for participant `PXB184`, the path might be `.\u002Fcheckpoints\u002Fdiffusion\u002Fc1_face\u002Fmodel000155000.pt`\nThe other parameters are:\n```\n--num_samples: number of samples to generate. To sample the full dataset, use 56 (except for TXB805, whcih is 58).\n--num_repetitions: number of times to repeat the sampling, such that total number of sequences generated is (num_samples * num_repetitions). \n--timestep_respacing: how many diffusion steps to take. Format will always be ddim\u003Cnumber>.\n--guidance_param: how influential the conditioning is on the results. I usually use range 2.0-10.0, and tend towards higher for the face.\n```\n\n:point_down: A full example of running the face model for `PXB184` with the provided pretrained models would then be:\n```\npython -m sample.generate --model_path checkpoints\u002Fdiffusion\u002Fc1_face\u002Fmodel000155000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 10.0\n```\nThis generates 10 samples from the dataset 1 time. The output results file will be saved to:\n`.\u002Fcheckpoints\u002Fdiffusion\u002Fc1_face\u002Fsamples_c1_face_000155000_seed10_\u002Fresults.npy`\n\n### Body generation\nTo generate the corresponding body, it will be very similar to generating the face, except now we have to feed in the model for generating the guide poses as well.\n```\npython -m sample.generate \n    --model_path \u003Cpath\u002Fto\u002Fmodel> \n    --resume_trans \u003Cpath\u002Fto\u002Fguide\u002Fmodel> \n    --num_samples \u003Cxsamples> \n    --num_repetitions \u003Cxreps> \n    --timestep_respacing ddim500 \n    --guidance_param 2.0\n```\n\n:point_down: Here, `\u003Cpath\u002Fto\u002Fguide\u002Fmodel>` should point to the guide transformer. The full command would be:\n```\npython -m sample.generate --model_path checkpoints\u002Fdiffusion\u002Fc1_pose\u002Fmodel000340000.pt --resume_trans checkpoints\u002Fguide\u002Fc1_pose\u002Fcheckpoints\u002Fiter-0100000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 2.0\n```\nSimilarly, the output will be saved to:\n`.\u002Fcheckpoints\u002Fdiffusion\u002Fc1_pose\u002Fsamples_c1_pose_000340000_seed10_guide_iter-0100000.pt\u002Fresults.npy`\n\n### Visualization\nOn the body generation side of things, you can also optionally pass in the `--plot` flag in order to render out the photorealistic avatar. You will also need to pass in the corresponding generated face codes with the `--face_codes` flag.\nOptionally, if you already have the poses precomputed, you an also pass in the generated body with the `--pose_codes` flag.\nThis will save videos in the same directory as where the body's `results.npy` is stored. \n\n:point_down: An example of the full command with *the three new flags added is*:\n```\npython -m sample.generate --model_path checkpoints\u002Fdiffusion\u002Fc1_pose\u002Fmodel000340000.pt --resume_trans checkpoints\u002Fguide\u002Fc1_pose\u002Fcheckpoints\u002Fiter-0100000.pt --num_samples 10 --num_repetitions 5 --timestep_respacing ddim500 --guidance_param 2.0 --face_codes .\u002Fcheckpoints\u002Fdiffusion\u002Fc1_face\u002Fsamples_c1_face_000155000_seed10_\u002Fresults.npy --pose_codes .\u002Fcheckpoints\u002Fdiffusion\u002Fc1_pose\u002Fsamples_c1_pose_000340000_seed10_guide_iter-0100000.pt\u002Fresults.npy --plot\n```\nThe remaining flags can be the same as before. For the actual rendering api, please see [Codec Avatar Body](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fca_body) for installation etc.\n*Important: in order to visualize the full photorealistic avatar, you will need to run the face codes first, then pass them into the body generation code.* It will not work if you try to call generate with `--plot` for the face codes.\n\n# Training from scratch\nThere are four possible models you will need to train: 1) the face diffusion model, 2) the body diffusion model, 3) the body vq vae, 4) the body guide transformer.\nThe only dependency is that 3) is needed for 4). All other models can be trained in parallel.\n\n### 1) Face diffusion model\nTo train the face model, you will need to run the following script:\n```\npython -m train.train_diffusion \n    --save_dir \u003Cpath\u002Fto\u002Fsave\u002Fdir>\n    --data_root \u003Cpath\u002Fto\u002Fdata\u002Froot>\n    --batch_size \u003Cbs>\n    --dataset social  \n    --data_format face \n    --layers 8 \n    --heads 8 \n    --timestep_respacing ''\n    --max_seq_length 600\n```\nImportantly, a few of the flags are as follows:\n```\n--save_dir: path to directory where all outputs are stored\n--data_root: path to the directory of where to load the data from\n--dataset: name of dataset to load; right now we only support the 'social' dataset\n--data_format: set to 'face' for the face, as opposed to pose\n--timestep_respacing: set to '' which does the default spacing of 1k diffusion steps\n--max_seq_length: the maximum number of frames for a given sequence to train on\n```\n:point_down: A full example for training on person `PXB184` is:\n```\npython -m train.train_diffusion --save_dir checkpoints\u002Fdiffusion\u002Fc1_face_test --data_root .\u002Fdataset\u002FPXB184\u002F --batch_size 4 --dataset social --data_format face --layers 8 --heads 8 --timestep_respacing '' --max_seq_length 600\n```\n\n### 2) Body diffusion model\nTraining the body model is similar to the face model, but with the following additional parameters\n```\npython -m train.train_diffusion \n    --save_dir \u003Cpath\u002Fto\u002Fsave\u002Fdir> \n    --data_root \u003Cpath\u002Fto\u002Fdata\u002Froot>\n    --lambda_vel \u003Cnum>\n    --batch_size \u003Cbs> \n    --dataset social \n    --add_frame_cond 1 \n    --data_format pose \n    --layers 6 \n    --heads 8 \n    --timestep_respacing '' \n    --max_seq_length 600\n```\nThe flags that differ from the face training are as follows:\n```\n--lambda_vel: additional auxilary loss for training with velocity\n--add_frame_cond: set to '1' for 1 fps. if not specified, it will default to 30 fps.\n--data_format: set to 'pose' for the body, as opposed to face\n```\n:point_down: A full example for training on person `PXB184` is:\n```\npython -m train.train_diffusion --save_dir checkpoints\u002Fdiffusion\u002Fc1_pose_test --data_root .\u002Fdataset\u002FPXB184\u002F --lambda_vel 2.0 --batch_size 4 --dataset social --add_frame_cond 1 --data_format pose --layers 6 --heads 8 --timestep_respacing '' --max_seq_length 600\n```\n\n### 3) Body VQ VAE\nTo train a vq encoder-decoder, you will need to run the following script:\n```\npython -m train.train_vq \n    --out_dir \u003Cpath\u002Fto\u002Fout\u002Fdir> \n    --data_root \u003Cpath\u002Fto\u002Fdata\u002Froot>\n    --batch_size \u003Cbs>\n    --lr 1e-3 \n    --code_dim 1024 \n    --output_emb_width 64 \n    --depth 4 \n    --dataname social \n    --loss_vel 0.0 \n    --add_frame_cond 1 \n    --data_format pose \n    --max_seq_length 600\n```\n:point_down: For person `PXB184`, it would be:\n```\npython -m train.train_vq --out_dir checkpoints\u002Fvq\u002Fc1_vq_test --data_root .\u002Fdataset\u002FPXB184\u002F --lr 1e-3 --code_dim 1024 --output_emb_width 64 --depth 4 --dataname social --loss_vel 0.0 --data_format pose --batch_size 4 --add_frame_cond 1 --max_seq_length 600\n```\n\n### 4) Body guide transformer\nOnce you have the vq trained from 3) you can then pass it in to train the body guide pose transformer:\n```\npython -m train.train_guide \n    --out_dir \u003Cpath\u002Fto\u002Fout\u002Fdir>\n    --data_root \u003Cpath\u002Fto\u002Fdata\u002Froot>\n    --batch_size \u003Cbs>\n    --resume_pth \u003Cpath\u002Fto\u002Fvq\u002Fmodel>\n    --add_frame_cond 1 \n    --layers 6 \n    --lr 2e-4 \n    --gn \n    --dim 64 \n```\n:point_down: For person `PXB184`, it would be:\n```\npython -m train.train_guide --out_dir checkpoints\u002Fguide\u002Fc1_trans_test --data_root .\u002Fdataset\u002FPXB184\u002F --batch_size 4 --resume_pth checkpoints\u002Fvq\u002Fc1_vq_test\u002Fnet_iter300000.pth --add_frame_cond 1 --layers 6 --lr 2e-4 --gn --dim 64\n```\n\nAfter training these 4 models, you can now follow the [\"Running the pretrained models\"](#running-the-pretrained-models) section to generate samples and visualize results.\n\nYou can also visualize the corresponding ground truth sequences by passing in the `--render_gt` flag.\n\n\n# License\nThe code and dataset are released under [CC-NC 4.0 International license](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Faudio2photoreal\u002Fblob\u002Fmain\u002FLICENSE).\n","该项目通过音频生成逼真的虚拟人物，适用于对话场景。其核心功能包括训练代码、测试代码以及预训练的运动模型，并提供了从音频到面部和身体动作合成的完整流程。基于PyTorch实现，用户可以通过简单的Gradio界面快速体验从录音到视频渲染的过程。此外，项目还详细介绍了数据集和预训练模型的使用方法，支持用户根据自身需求进行深度定制或从零开始训练模型。适合需要创建高质量虚拟交互角色的应用场景，如虚拟助手、在线教育或远程会议等。","2026-06-11 03:41:50","high_star"]