[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-92374":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":9,"language":10,"languages":9,"totalLinesOfCode":9,"stars":11,"forks":12,"watchers":13,"openIssues":14,"contributorsCount":14,"subscribersCount":14,"size":14,"stars1d":14,"stars7d":14,"stars30d":14,"stars90d":14,"forks30d":14,"starsTrendScore":14,"compositeScore":15,"rankGlobal":9,"rankLanguage":9,"license":16,"archived":17,"fork":17,"defaultBranch":18,"hasWiki":17,"hasPages":17,"topics":19,"createdAt":9,"pushedAt":9,"updatedAt":37,"readmeContent":38,"aiSummary":39,"trendingCount":14,"starSnapshotCount":14,"syncStatus":12,"lastSyncTime":40,"discoverSource":41},92374,"vllm-chatterbox-stream","wuxuedaifu\u002Fvllm-chatterbox-stream","wuxuedaifu","OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.",null,"Python",55,2,1,0,41.43,"MIT License",false,"main",[20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36],"audio","chatterbox","cuda","fastapi","gpu","llm-inference","low-latency","multilingual","openai-api","pytorch","real-time","speech-synthesis","streaming","text-to-speech","tts","vllm","voice-cloning","2026-07-22 04:02:06","# vLLM Chatterbox Stream — OpenAI-compatible TTS with real streaming\n\nMultilingual text-to-speech ([Chatterbox](https:\u002F\u002Fgithub.com\u002Fresemble-ai\u002Fchatterbox))\nserved on **vLLM**, exposed through the **OpenAI `\u002Fv1\u002Faudio\u002Fspeech` API** with **real\naudio streaming** and low time-to-first-byte. 23 languages, voice cloning.\n\n- **Two-stage TTS:** an autoregressive **T3** model (a Llama backbone) emits speech\n  tokens — served by vLLM's `AsyncLLMEngine` with continuous batching — and **S3Gen**\n  (CosyVoice2 flow-matching decoder + HiFi-GAN vocoder) turns them into a 24 kHz waveform.\n- **Real streaming:** with `stream: true` the server emits raw PCM chunks as they are\n  generated, so playback starts in ~1 second instead of waiting for the whole clip.\n\nMore detail: [`docs\u002Fserving.md`](docs\u002Fserving.md) (full API & tuning) ·\n[`docs\u002Fmaintainers-runbook.md`](docs\u002Fmaintainers-runbook.md) (ops) ·\n[`docs\u002Ftechnical-primer.md`](docs\u002Ftechnical-primer.md) (how it works).\n\n---\n\n## Install\n\nLinux + NVIDIA GPU. Requires `git` and Python 3.10+ (and\n[`uv`](https:\u002F\u002Fpypi.org\u002Fproject\u002Fuv\u002F) for the fast path).\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fwuxuedaifu\u002Fvllm-chatterbox-stream.git\ncd vllm-chatterbox-stream\nuv venv && source .venv\u002Fbin\u002Factivate\nuv pip install -e .            # or: pip install -e .\n```\n\nModel weights download automatically from the Hugging Face Hub on first run.\n\n## Start the server\n\n```bash\nchatterbox-serve --variant multilingual --host 0.0.0.0 --port 8000\n```\n\nRecommended low-latency flag (≈12% faster TTFB, identical audio):\n\n```bash\nCHATTERBOX_T3_CUDA_GRAPHS=1 chatterbox-serve --variant multilingual --port 8000\n```\n\n| Flag | Default | Meaning |\n|---|---|---|\n| `--variant` | `multilingual` | `multilingual` (23 langs) or `english` |\n| `--host` \u002F `--port` | `0.0.0.0` \u002F `8000` | bind address |\n| `--voices` | `voices\u002Fvoices.json` | voice-preset file |\n| `--max-batch-size` | `16` | vLLM `max_num_seqs` + scheduler concurrency budget |\n| `--max-model-len` | `1000` | max speech tokens per request |\n\n---\n\n## Run with Docker\n\nThe image bundles the server and runs `chatterbox-serve` on port 8000. An NVIDIA GPU\nand the NVIDIA Container Toolkit are required at runtime.\n\n**Build:**\n\n```bash\ndocker build -t vllm-chatterbox-stream .\n```\n\n**Run** (mount a Hugging Face cache so weights download only once):\n\n```bash\ndocker run --gpus all -p 8000:8000 \\\n  -v $HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  vllm-chatterbox-stream\n```\n\nEnable the low-latency CUDA-graphs path:\n\n```bash\ndocker run --gpus all -p 8000:8000 \\\n  -e CHATTERBOX_T3_CUDA_GRAPHS=1 \\\n  -v $HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  vllm-chatterbox-stream\n```\n\nThe server is then reachable at `http:\u002F\u002Flocalhost:8000\u002Fv1\u002Faudio\u002Fspeech`.\n\n---\n\n## Real streaming & TTFB\n\nWith `stream: true` the response is a chunked `audio\u002Fpcm` body: T3 speech tokens are\nconsumed as deltas and an `S3GenStreamer` re-decodes the growing prefix into ~1 s audio\nchunks (HiFi-GAN `cache_source` + overlap-fade), so the first audio arrives long before\ngeneration finishes.\n\n**Measured on one A100 (vLLM 0.10.0):**\n\n| Metric | Streaming | Non-streaming |\n|---|---|---|\n| Time-to-first-byte (Arabic, ~17 s clip) | **≈ 1.1 s** | ≈ 7.6 s (~7× slower) |\n| Single-request TTFB p50 (short EN, CUDA graphs on) | **≈ 0.68 s** | — |\n| Inter-chunk gap | ≈ 0.6 s | n\u002Fa |\n| Real-time factor (RTF) | ≈ 0.5 | ≈ 0.5 |\n\n### TTFB tuning (streaming)\n\n| Setting | Default | Effect (measured) |\n|---|---|---|\n| Startup warmup | **on** | Throwaway generation at startup so the first real request skips cold-start (~1.5 s → ~0.81 s). Disable with `CHATTERBOX_SKIP_WARMUP=1`. |\n| `CHATTERBOX_T3_CUDA_GRAPHS=1` | off | vLLM CUDA graphs for T3. **TTFB 809 → 714 ms (−12%)**, audio identical (corr 0.9999). **Recommended on.** |\n| `CHATTERBOX_STREAM_FIRST_HOP` | 12 | Smaller first hop emits the first chunk sooner (tune to taste). |\n| `CHATTERBOX_STREAM_NTIMESTEPS` | — | Fewer CFM decode steps roughly halve decode time (some quality cost). |\n\n> Note: PCM is currently the only streaming format (`mp3`\u002F`opus` containers are\n> non-streaming for now). Streaming re-decodes the prefix each chunk (O(N²) total\n> compute — fine for sentence-length inputs; incremental caching is a future optimization).\n\n---\n\n## Concurrency\n\n`chatterbox-serve` runs on vLLM's `AsyncLLMEngine` with continuous batching. The\nscheduler admits up to `--max-batch-size` (default 16) requests so they batch together;\na separate, smaller semaphore bounds concurrent S3Gen decodes\n(`CHATTERBOX_S3GEN_CONC`, default 2) so they don't thrash VRAM.\n\n**Closed-loop HTTP load test (A100, T3 CUDA graphs on, 79-char EN input)** — at each\nconcurrency level `N`, `N` virtual users each stream a request, await completion, and\nrepeat for a fixed window. Metrics over successful requests:\n\n| Concurrency (N) | TTFB p50 | TTFB p95 | Throughput (req\u002Fs) | Errors |\n|---|---|---|---|---|\n| 1 | 0.68 s | 0.71 s | 0.45 | 0% |\n| 2 | 1.07 s | 1.19 s | 0.33 | 0% |\n| 4 | 1.98 s | — | 0.47 | 0% |\n| 8 | 3.72 s | — | 0.43 | 0% |\n| 16 | 4.47 s | — | 0.37 | 0% |\n| 32 | 18.6 s | — | 0.45 | 0% |\n\nTTFB stays roughly flat up to the batch capacity (16); beyond it, requests queue and\nTTFB climbs — raise `--max-batch-size` (more VRAM) or add GPU replicas. **Throughput\nholds at ~0.4 req\u002Fs regardless of N: that is the single-GPU compute ceiling** (≈2\nconcurrent requests already saturate the GPU), not a bug. Scale total throughput with\nfaster per-request compute (CUDA graphs, fewer CFM steps) or more replicas.\n\nReproduce with `test_scripts\u002Fconcurrency_test.py` (closed-loop tester; writes a\nCSV + Markdown table).\n\n---\n\n## API endpoints\n\n| Method & path | Purpose |\n|---|---|\n| `POST \u002Fv1\u002Faudio\u002Fspeech` | synthesize speech (non-streaming or streaming) |\n| `GET \u002Fv1\u002Fmodels` | list the served model (`chatterbox`) |\n| `GET \u002Fhealth` | readiness probe → `{\"status\":\"ok\"}` |\n\n### Usage\n\n**curl — non-streaming** (returns the whole clip):\n\n```bash\ncurl -s -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Faudio\u002Fspeech \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"chatterbox\",\"input\":\"Hello from Chatterbox.\",\"voice\":\"uae_man\",\"language\":\"en\",\"response_format\":\"mp3\"}' \\\n  -o out.mp3\n```\n\n**curl — streaming** (low TTFB; raw PCM chunks piped straight to a player):\n\n```bash\ncurl -N -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Faudio\u002Fspeech \\\n  -H 'Content-Type: application\u002Fjson' \\\n  -d '{\"model\":\"chatterbox\",\"input\":\"مرحبا بالعالم\",\"voice\":\"uae_man\",\"language\":\"ar\",\"stream\":true,\"response_format\":\"pcm\"}' \\\n  --output - | aplay -r 24000 -f S16_LE -c 1\n```\n\n**OpenAI Python client** (Chatterbox extras go through `extra_body`):\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\", api_key=\"not-needed\")\n\n# Non-streaming\nresp = client.audio.speech.create(\n    model=\"chatterbox\",\n    voice=\"uae_man\",\n    input=\"Hello from Chatterbox.\",\n    response_format=\"mp3\",\n    extra_body={\"language\": \"en\", \"exaggeration\": 0.5},\n)\nresp.stream_to_file(\"out.mp3\")\n\n# Streaming: write PCM chunks as they arrive\nwith client.audio.speech.with_streaming_response.create(\n    model=\"chatterbox\",\n    voice=\"uae_man\",\n    input=\"Hello from Chatterbox.\",\n    response_format=\"pcm\",\n    extra_body={\"language\": \"en\", \"stream\": True},\n) as resp:\n    with open(\"out.pcm\", \"wb\") as f:\n        for chunk in resp.iter_bytes():\n            f.write(chunk)   # raw int16, 24 kHz, mono\n```\n\n### Request — `POST \u002Fv1\u002Faudio\u002Fspeech` (JSON)\n\n| Field | Type | Default | Meaning |\n|---|---|---|---|\n| `model` | string | `chatterbox` | model id |\n| `input` | string | **required** | text to synthesize |\n| `voice` | string | `default` | preset name from `voices\u002Fvoices.json` |\n| `reference_audio` | string | `null` | base64 or `http(s)` URL clip to clone (overrides `voice`) |\n| `response_format` | string | `mp3` | `mp3` \\| `wav` \\| `opus` \\| `flac` \\| `pcm` \\| `aac` |\n| `stream` | bool | `false` | stream audio chunks as generated (use `pcm`) |\n| `speed` | float | `1.0` | playback speed, `0 \u003C speed ≤ 4` |\n| `language` | string | `en` | one of 23 codes (see below) |\n| `exaggeration` | float | `0.5` | emotion intensity, `0–1` |\n| `vad` | bool | `true` | drop hallucinated non-speech (Silero VAD; non-streaming) |\n| `temperature` | float | `0.8` | sampling |\n| `top_p` \u002F `min_p` \u002F `repetition_penalty` | float | `0.8` \u002F `0.1` \u002F `2.0` | sampling |\n\n### Response\n\n- **Non-streaming:** HTTP body = the audio file, `Content-Type: audio\u002F\u003Cresponse_format>`.\n- **Streaming** (`stream: true`): chunked `audio\u002Fpcm` body emitted as generated — raw\n  little-endian **int16, 24 kHz, mono**.\n- **Errors:** JSON `{\"error\": \"...\"}` — `400` (bad voice\u002Flanguage), `503` (engine not\n  ready), `500` (other).\n\n**Voices:** edit `voices\u002Fvoices.json` (`name → reference clip path`, or `null` for the\nbuilt-in default). Keep reference clips ~6–10 s of clean speech.\n\n---\n\n## Language samples\n\n23 languages, all generated by this server. Codes:\n`ar da de el en es fi fr he hi it ja ko ms nl no pl pt ru sv sw tr zh`.\n\n| Language | Code | Sample | Language | Code | Sample |\n|---|---|---|---|---|---|\n| Arabic | `ar` | [▶](docs\u002Fsamples\u002Flang\u002Flang_ar.mp3) | Japanese | `ja` | [▶](docs\u002Fsamples\u002Flang\u002Flang_ja.mp3) |\n| Danish | `da` | [▶](docs\u002Fsamples\u002Flang\u002Flang_da.mp3) | Korean | `ko` | [▶](docs\u002Fsamples\u002Flang\u002Flang_ko.mp3) |\n| German | `de` | [▶](docs\u002Fsamples\u002Flang\u002Flang_de.mp3) | Malay | `ms` | [▶](docs\u002Fsamples\u002Flang\u002Flang_ms.mp3) |\n| Greek | `el` | [▶](docs\u002Fsamples\u002Flang\u002Flang_el.mp3) | Dutch | `nl` | [▶](docs\u002Fsamples\u002Flang\u002Flang_nl.mp3) |\n| English | `en` | [▶](docs\u002Fsamples\u002Flang\u002Flang_en.mp3) | Norwegian | `no` | [▶](docs\u002Fsamples\u002Flang\u002Flang_no.mp3) |\n| Spanish | `es` | [▶](docs\u002Fsamples\u002Flang\u002Flang_es.mp3) | Polish | `pl` | [▶](docs\u002Fsamples\u002Flang\u002Flang_pl.mp3) |\n| Finnish | `fi` | [▶](docs\u002Fsamples\u002Flang\u002Flang_fi.mp3) | Portuguese | `pt` | [▶](docs\u002Fsamples\u002Flang\u002Flang_pt.mp3) |\n| French | `fr` | [▶](docs\u002Fsamples\u002Flang\u002Flang_fr.mp3) | Russian | `ru` | [▶](docs\u002Fsamples\u002Flang\u002Flang_ru.mp3) |\n| Hebrew | `he` | [▶](docs\u002Fsamples\u002Flang\u002Flang_he.mp3) | Swedish | `sv` | [▶](docs\u002Fsamples\u002Flang\u002Flang_sv.mp3) |\n| Hindi | `hi` | [▶](docs\u002Fsamples\u002Flang\u002Flang_hi.mp3) | Swahili | `sw` | [▶](docs\u002Fsamples\u002Flang\u002Flang_sw.mp3) |\n| Italian | `it` | [▶](docs\u002Fsamples\u002Flang\u002Flang_it.mp3) | Turkish | `tr` | [▶](docs\u002Fsamples\u002Flang\u002Flang_tr.mp3) |\n|  |  |  | Chinese | `zh` | [▶](docs\u002Fsamples\u002Flang\u002Flang_zh.mp3) |\n\nRegenerate the set with `test_scripts\u002Fmultilang_test.py`.\n\n---\n\n## Known limitation — audio hallucinations\n\nChatterbox itself has a known defect: generated speech can occasionally contain\n**hallucinated, blurry, or garbled non-speech artifacts** (buzzing, breaths, or\nmeaningful-sounding rambling past the end of the text). The non-streaming path mitigates\nthis with a **Silero VAD post-filter** (enabled by default; disable per request with\n`\"vad\": false`) that keeps only speech regions, plus a text-proportional `max_tokens`\ncap to bound rambling. These reduce the problem substantially but **do not eliminate it\nentirely** — it is an upstream model limitation, not a server bug. VAD is not yet applied\nto the streaming path (it needs a causal hangover buffer — planned).\n\n---\n\n## License\n\nMIT. Chatterbox © 2025 Resemble AI; vLLM port © 2025 David Jia Wei Li. See\n[`LICENSE`](LICENSE).\n","这是一个基于vLLM加速的开源多语言文本转语音（TTS）服务，提供与OpenAI API兼容的实时PCM音频流式响应。核心采用两阶段架构：T3模型（Llama架构）生成语音token，由vLLM异步引擎支持连续批处理；S3Gen解码器（CosyVoice2流匹配+HiFi-GAN）实时合成24kHz波形。支持23种语言、语音克隆及低首字节延迟（约0.7秒）。适用于需要低延迟、高并发、多语言TTS能力的对话系统、智能客服、AIGC音视频生成等实时交互场景。","2026-07-08 04:30:13","CREATED_QUERY"]