[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-72004":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":17,"stars7d":18,"stars30d":19,"stars90d":16,"forks30d":16,"starsTrendScore":20,"compositeScore":21,"rankGlobal":10,"rankLanguage":10,"license":22,"archived":23,"fork":23,"defaultBranch":24,"hasWiki":23,"hasPages":23,"topics":25,"createdAt":10,"pushedAt":10,"updatedAt":37,"readmeContent":38,"aiSummary":39,"trendingCount":16,"starSnapshotCount":16,"syncStatus":15,"lastSyncTime":40,"discoverSource":41},72004,"Vision-Agents","GetStream\u002FVision-Agents","GetStream","Open Vision Agents by Stream. Build voice and vision agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.","https:\u002F\u002Fvisionagents.ai",null,"Python",7909,659,55,2,0,12,42,111,36,39.46,"Apache License 2.0",false,"main",[26,27,28,29,30,31,32,33,34,35,36],"agentic-ai","agents","ai","ai-agents","realtime","stt","tts","video-agents","video-ai","vision-ai","voice-ai","2026-06-12 02:02:57","![VisionAgents](assets\u002Frepo_image.png)\n\n# Open Vision Agents by Stream\n\n[![build](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Factions\u002Fworkflows\u002Fci.yml\u002Fbadge.svg)](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Factions)\n[![PyPI version](https:\u002F\u002Fbadge.fury.io\u002Fpy\u002Fvision-agents.svg)](http:\u002F\u002Fbadge.fury.io\u002Fpy\u002Fvision-agents)\n![PyPI - Python Version](https:\u002F\u002Fimg.shields.io\u002Fpypi\u002Fpyversions\u002Fvision-agents.svg)\n[![License](https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002FGetStream\u002FVision-Agents)](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Fblob\u002Fmain\u002FLICENSE)\n[![Discord](https:\u002F\u002Fimg.shields.io\u002Fdiscord\u002F1108586339550638090)](https:\u002F\u002Fdiscord.gg\u002FRkhX9PxMS6)\n[![X (Twitter)](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FX-@visionagents__ai-000000?logo=x&logoColor=white)](https:\u002F\u002Fx.com\u002Fvisionagents_ai)\n\n### Multi-modal AI agents that watch, listen, and understand video.\n\n[Vision Agents](https:\u002F\u002Fvisionagents.ai\u002F) give you the building blocks to create intelligent, low-latency video experiences powered by your models,\nyour infrastructure, and your use cases.\n\n### Key Highlights\n\n- **Video AI:** Built for real-time video AI. Combine YOLO, Roboflow, and others with Gemini\u002FOpenAI in real-time.\n- **Low Latency:** Join quickly (500ms) and maintain audio\u002Fvideo latency under 30ms\n  using [Stream's edge network](https:\u002F\u002Fgetstream.io\u002Fvideo\u002F?utm_source=github.com&utm_medium=referral&utm_campaign=vision_agents).\n- **Open:** Built by Stream, but works with any video edge network.\n- **Native APIs:** Native SDK methods from OpenAI (`create response`), Gemini (`generate`), and Claude (\n  `create message`) — always access the latest LLM capabilities.\n- **SDKs:** SDKs for React, Android, iOS, Flutter, React Native, and Unity, powered by Stream's ultra-low-latency\n  network.\n\n## Getting Started\n\n**Step 1: Install via uv**\n\n`uv add vision-agents`\n\n**Step 2: (Optional) Install with extra integrations**\n\n`uv add \"vision-agents[getstream, openai, elevenlabs, deepgram]\"`\n\n**Step 3: Obtain your Stream API credentials**\n\nGet a free API key from [Stream](https:\u002F\u002Fgetstream.io\u002Ftry-for-free\u002F?utm_source=github.com&utm_medium=referral&utm_campaign=vision_agents). Developers receive **333,000 participant minutes** per month,\nplus extra credits via the Maker Program.\n\nFollow the [quickstart guide](https:\u002F\u002Fvisionagents.ai\u002Fintroduction\u002Fquickstart) to build your first agent.\n\n## See It In Action\n\nhttps:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fd1258ac2-ca98-4019-80e4-41ec5530117e\n\nThis example shows you how to build golf coaching AI with YOLO and Gemini Live.\nCombining a fast object detection model (like YOLO) with a full realtime AI is useful for many different video AI use\ncases.\nFor example: Drone fire detection, sports\u002Fvideo game coaching, physical therapy, workout coaching, just dance style\ngames etc.\n\n```python\n# partial example, full example: examples\u002F02_golf_coach_example\u002Fgolf_coach_example.py\nagent = Agent(\n    edge=getstream.Edge(),\n    agent_user=agent_user,\n    instructions=\"Read @golf_coach.md\",\n    llm=gemini.Realtime(fps=10),\n    processors=[ultralytics.YOLOPoseProcessor(model_path=\"yolo11n-pose.pt\", device=\"cuda\")],\n)\n```\n\n## Features\n\n| **Feature**              | **Description**                                                                                         |\n|--------------------------|---------------------------------------------------------------------------------------------------------|\n| **Real-time WebRTC**     | Stream video directly to model providers for instant visual understanding.                              |\n| **Video Processing**     | Pluggable processor pipeline for YOLO, Roboflow, or custom PyTorch\u002FONNX models before\u002Fafter LLM calls. |\n| **Turn Detection**       | Natural conversation flow with VAD, diarization, and smart turn-taking.                                 |\n| **Tool Calling & MCP**   | Execute code and APIs mid-conversation — Linear issues, weather, telephony, or any MCP server.          |\n| **Phone Integration**    | Inbound and outbound voice calls via Twilio with bidirectional audio streaming.                         |\n| **RAG**                  | Retrieval-augmented generation with TurboPuffer vector search or Gemini FileSearch.                     |\n| **Memory**               | Agents recall context across turns and sessions via Stream Chat.                                        |\n| **Text Back-channel**    | Message the agent silently during a call — coaching overlays, silent instructions, etc.                 |\n| **Production Ready**     | Built-in HTTP server, Prometheus metrics, horizontal scaling, and Kubernetes deployment.                |\n\n## Out-of-the-Box Integrations\n\n**LLMs:** [OpenAI](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fopenai) · [Gemini](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fgemini) · [xAI](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fxai) · [OpenRouter](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fopenrouter) · [Hugging Face](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fhuggingface) · [Kimi AI](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fkimi)\n\n**Realtime:** [OpenAI Realtime](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fopenai) · [Gemini Live](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fgemini) · [AWS Nova Sonic](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Faws-bedrock) · [Qwen](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fqwen) · [Inworld](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Finworld)\n\n**STT:** [Deepgram](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fdeepgram) · [AssemblyAI](https:\u002F\u002Fwww.assemblyai.com\u002Fdocs\u002Fstreaming\u002Funiversal-3-pro) · [Fast-Whisper](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Ffast-whisper) · [Fish Audio](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Ffish) · [Wizper](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fwizper) · [Mistral Voxtral](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fmistral)\n\n**TTS:** [ElevenLabs](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Felevenlabs) · [Cartesia](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fcartesia) · [Deepgram](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fdeepgram) · [AWS Polly](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Faws-polly) · [Pocket](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fpocket) · [Kokoro](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fkokoro) · [Inworld](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Finworld) · [Fish Audio](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Ffish)\n\n**Vision:** [Ultralytics](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fultralytics) · [Roboflow](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Froboflow) · [Moondream](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fmoondream) · [NVIDIA Cosmos](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fnvidia) · [Decart](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fdecart)\n\n**Avatars:** [LemonSlice](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Flemonslice)\n\n**Turn Detection:** [Vogent](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fvogent) · [Smart Turn](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fsmart-turn)\n\n**Other:** [Twilio](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fexamples\u002F03_phone_and_rag_example) · [TurboPuffer](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Frag)\n\n## Documentation\n\nCheck out the full docs at [VisionAgents.ai](https:\u002F\u002Fvisionagents.ai\u002F).\n\n**Quickstart:** [Voice AI](https:\u002F\u002Fvisionagents.ai\u002Fintroduction\u002Fvoice-agents) · [Video AI](https:\u002F\u002Fvisionagents.ai\u002Fintroduction\u002Fvideo-agents)\n\n**Guides:** [MCP & Function Calling](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fmcp-tool-calling) · [Video Processors](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fvideo-processors) · [Phone Calling](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fcalling) · [RAG](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Frag) · [Testing](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Ftesting)\n\n**Production:** [HTTP Server](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fhttp-server) · [Deployment](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fdeployment) · [Kubernetes](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fkubernetes-deployment) · [Horizontal Scaling](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fhorizontal-scaling) · [Prometheus Metrics](https:\u002F\u002Fvisionagents.ai\u002Fguides\u002Fprometheus-metrics)\n\n## Examples\n\n| 🔮 Demo Applications                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                                                                                         |\n|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------|\n| \u003Cbr>\u003Ch3>Voice Agents (Low Latency + RAG + File Search)\u003C\u002Fh3>Build fast voice agents that can reason over knowledge, search files, and respond in real time.\u003Cbr>\u003Cbr>• Low-latency voice interactions\u003Cbr>• Retrieval-augmented responses\u003Cbr>• File and knowledge search\u003Cbr>\u003Cbr> [>Source Code and tutorial](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fplugins\u002Fcartesia\u002Fexample)                                                                                                                                                    | \u003Cimg src=\"assets\u002Fdemo_gifs\u002Fcartesia.gif\" width=\"320\" alt=\"Voice Agent Demo\">               |\n| \u003Cbr>\u003Ch3>Realtime Coaching and Video Understanding\u003C\u002Fh3>Power interactive coaching flows with live pose tracking and processor pipelines for frame-by-frame understanding.\u003Cbr>\u003Cbr>• Real-time pose tracking\u003Cbr>• Actionable coaching feedback\u003Cbr>• Video processor pipeline support\u003Cbr>\u003Cbr> [>Source Code and tutorial](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fexamples\u002F02_golf_coach_example)                                                     | \u003Cimg src=\"assets\u002Fdemo_gifs\u002Fgolf.gif\" width=\"320\" alt=\"Realtime Coaching Demo\">                 |\n| \u003Cbr>\u003Ch3>Video Restyling and Avatars\u003C\u002Fh3>Use models like Decart Lucy to build virtual try-ons, stylized scenes, or give your agents a visual identity.\u003Cbr>\u003Cbr>• Real-time video restyling\u003Cbr>• Virtual try-on experiences\u003Cbr>• Avatar-like visual presence\u003Cbr>\u003Cbr> [>Source Code and tutorial](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fplugins\u002Fdecart\u002Fexample)                                                                                                    | \u003Cimg src=\"assets\u002Fdemo_gifs\u002Fmirage.gif\" width=\"320\" alt=\"Video Restyling Demo\">           |\n| \u003Cbr>\u003Ch3>Custom Video Models (Roboflow, YOLO, and More)\u003C\u002Fh3>Train and run custom computer vision models for security monitoring, moderation, and other domain-specific workflows.\u003Cbr>\u003Cbr>• Bring your own CV models\u003Cbr>• Real-time moderation pipelines\u003Cbr>• Security and detection use cases\u003Cbr>\u003Cbr> [>Source Code and tutorial](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fexamples\u002F11_moderation_example) | \u003Cimg src=\"assets\u002Fdemo_gifs\u002Fsecurity_camera.gif\" width=\"320\" alt=\"Custom Video Models Demo\">          |\n| \u003Cbr>\u003Ch3>Tools, MCP, and Phone Calling\u003C\u002Fh3>Connect external APIs and services so agents can validate data and take real-world actions during live conversations.\u003Cbr>\u003Cbr>• MCP and function calling support\u003Cbr>• Twilio-based phone workflows\u003Cbr>• Real-time fraud response automation\u003Cbr>\u003Cbr> [>Phone + RAG example](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fexamples\u002F03_phone_and_rag_example) · [>Fraud workflow example](https:\u002F\u002Fgithub.com\u002FGetStream\u002FVision-Agents\u002Ftree\u002Fmain\u002Fplugins\u002Fopenai\u002Fexamples\u002Fnemotron_example) | \u003Cimg src=\"assets\u002Fdemo_gifs\u002Ffraud_detection.gif\" width=\"320\" alt=\"Tools and Phone Demo\"> |\n\n## Development\n\nSee [DEVELOPMENT.md](DEVELOPMENT.md)\n\nWant to add your platform or provider? See [Create Your Own Plugin](https:\u002F\u002Fvisionagents.ai\u002Fintegrations\u002Fcreate-your-own-plugin) or reach out to **nash@getstream.io**.\n\n## Current Limitations\n\n- Video AI struggles with small text — models may hallucinate scores, signs, etc.\n- Context degrades on longer sessions (~30s+) for continuous video understanding\n- Most use cases need a mix of specialized models (YOLO, Roboflow) with larger LLMs\n- Real-time models require audio\u002Ftext to trigger responses — video alone won't prompt output\n\n## Star History\n\n[![Star History Chart](https:\u002F\u002Fapi.star-history.com\u002Fsvg?repos=GetStream\u002Fvision-agents&type=timeline&legend=top-left)](https:\u002F\u002Fwww.star-history.com\u002F#GetStream\u002Fvision-agents&type=timeline&legend=top-left)\n","GetStream\u002FVision-Agents 是一个用于快速构建语音和视觉代理的开源项目，支持任何模型或视频提供商。该项目利用 Stream 的边缘网络提供超低延迟服务，核心功能包括实时视频AI处理、多种模型（如YOLO、Roboflow）与语言模型（Gemini\u002FOpenAI）的无缝集成，以及通过Stream的基础设施实现的500毫秒内快速连接和30毫秒以下的音视频延迟。它提供了广泛的原生API接口和跨平台SDK支持，适用于开发需要实时分析与响应的应用场景，例如无人机火灾检测、体育训练辅助、物理治疗指导等。","2026-06-11 03:39:56","high_star"]