[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-9821":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":16,"subscribersCount":16,"size":16,"stars1d":16,"stars7d":17,"stars30d":18,"stars90d":16,"forks30d":16,"starsTrendScore":16,"compositeScore":19,"rankGlobal":10,"rankLanguage":10,"license":20,"archived":21,"fork":21,"defaultBranch":22,"hasWiki":21,"hasPages":21,"topics":23,"createdAt":10,"pushedAt":10,"updatedAt":33,"readmeContent":34,"aiSummary":35,"trendingCount":16,"starSnapshotCount":16,"syncStatus":36,"lastSyncTime":37,"discoverSource":38},9821,"LitServe","Lightning-AI\u002FLitServe","Lightning-AI","A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.","https:\u002F\u002Flightning.ai\u002Flitserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme",null,"Python",3889,294,27,29,0,1,9,29.41,"Apache License 2.0",false,"main",[24,25,26,27,28,29,30,31,32],"ai","api","artificial-intelligence","deep-learning","developer-tools","fastapi","rest-api","serving","web","2026-06-12 02:02:13","\u003Cdiv align='center'>\n\n\u003Ch1>\n  Build custom inference servers in pure Python\n  \u003Cbr\u002F>\n\u003C\u002Fh1> \n\u003Ch4>\n  Define exactly how inference works for models, agents, RAG, or pipelines. \n  \u003Cbr\u002F>\n  Control batching, routing, streaming, and orchestration without MLOps glue or config files.\n\u003C\u002Fh4> \n\n\u003Cimg alt=\"Lightning\" src=\"https:\u002F\u002Fpl-bolts-doc-images.s3.us-east-2.amazonaws.com\u002Fapp-2\u002Fls_banner2.png\" width=\"800px\" style=\"max-width: 100%;\">\n\n  \n\u003C\u002Fdiv>\n\n\u003Cdiv align='center'>\n  \n\u003Cpre>\n✅ Custom inference logic  ✅ 2× faster than FastAPI     ✅ Agents, RAG, pipelines, more\n✅ Custom logic + control  ✅ Any PyTorch model          ✅ Self-host or managed        \n✅ Multi-GPU autoscaling   ✅ Batching + streaming       ✅ BYO model or vLLM           \n✅ No MLOps glue code      ✅ Easy setup in Python       ✅ Serverless support          \n\n\u003C\u002Fpre>\n\n\u003Cdiv align='center'>\n\n[![PyPI Downloads](https:\u002F\u002Fstatic.pepy.tech\u002Fbadge\u002Flitserve)](https:\u002F\u002Fpepy.tech\u002Fprojects\u002Flitserve)\n[![Discord](https:\u002F\u002Fimg.shields.io\u002Fdiscord\u002F1077906959069626439?label=Get%20help%20on%20Discord)](https:\u002F\u002Fdiscord.gg\u002FWajDThKAur)\n![cpu-tests](https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Flitserve\u002Factions\u002Fworkflows\u002Fci-testing.yml\u002Fbadge.svg)\n[![codecov](https:\u002F\u002Fcodecov.io\u002Fgh\u002FLightning-AI\u002Flitserve\u002Fgraph\u002Fbadge.svg?token=SmzX8mnKlA)](https:\u002F\u002Fcodecov.io\u002Fgh\u002FLightning-AI\u002Flitserve)\n[![license](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FLicense-Apache%202.0-blue.svg)](https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Flitserve\u002Fblob\u002Fmain\u002FLICENSE)\n\n\u003C\u002Fdiv>\n\u003C\u002Fdiv>\n\u003Cdiv align=\"center\">\n  \u003Cdiv style=\"text-align: center;\">\n    \u003Ca target=\"_blank\" href=\"#quick-start\" style=\"margin: 0 10px;\">Quick start\u003C\u002Fa> •\n    \u003Ca target=\"_blank\" href=\"#featured-examples\" style=\"margin: 0 10px;\">Examples\u003C\u002Fa> •\n    \u003Ca target=\"_blank\" href=\"#features\" style=\"margin: 0 10px;\">Features\u003C\u002Fa> •\n    \u003Ca target=\"_blank\" href=\"#performance\" style=\"margin: 0 10px;\">Performance\u003C\u002Fa> •\n    \u003Ca target=\"_blank\" href=\"#host-anywhere\" style=\"margin: 0 10px;\">Hosting\u003C\u002Fa> •\n    \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\" style=\"margin: 0 10px;\">Docs\u003C\u002Fa>\n  \u003C\u002Fdiv>\n\u003C\u002Fdiv>\n\n \n\n\u003Cdiv align=\"center\">\n\u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fhome\u002Fget-started?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">\n  \u003Cimg src=\"https:\u002F\u002Fpl-bolts-doc-images.s3.us-east-2.amazonaws.com\u002Fapp-2\u002Fget-started-badge.svg\" height=\"36px\" alt=\"Get started\"\u002F>\n\u003C\u002Fa>\n\u003C\u002Fdiv>\n\n  \n\n# Why LitServe?\nMost serving tools (vLLM, etc..) are built for a single model type and enforce rigid abstractions. They work well until you need custom logic, multiple models, agents, or non standard pipelines. LitServe lets you write your own inference engine in Python. You define how requests are handled, how models are loaded, how batching and routing work, and how outputs are produced. LitServe handles performance, concurrency, scaling, and deployment. Use LitServe to build inference APIs, agents, chatbots, RAG systems, MCP servers, or multi model pipelines. \n\nRun it locally, self host anywhere, or deploy with one click on [Lightning AI](https:\u002F\u002Flightning.ai\u002Flitserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme).\n\n \n\n# Want the easiest way to host inference?\nOver 380,000 developers use [Lightning Cloud](https:\u002F\u002Flightning.ai\u002F?utm_source=ptl_readme&utm_medium=referral&utm_campaign=ptl_readme), the simplest way to run LitServe without managing infrastructure. Deploy with one command, get autoscaling GPUs, monitoring, and a free tier. No cloud setup required. Or self host anywhere.\n\n# Quick start\n\nInstall LitServe via pip ([more options](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fhome\u002Finstall)):\n\n```bash\npip install litserve\n```\n\n[Example 1](#inference-engine-example): Toy inference pipeline with multiple models.   \n[Example 2](#agent-example): Minimal agent to fetch the news (with OpenAI API).    \n([Advanced examples](#featured-examples)):    \n\n### Inference engine example   \n\n```python\nimport litserve as ls\n\n# define the api to include any number of models, dbs, etc...\nclass InferenceEngine(ls.LitAPI):\n    def setup(self, device):\n        self.text_model = lambda x: x**2\n        self.vision_model = lambda x: x**3\n\n    def predict(self, request):\n        x = request[\"input\"]    \n        # perform calculations using both models\n        a = self.text_model(x)\n        b = self.vision_model(x)\n        c = a + b\n        return {\"output\": c}\n\nif __name__ == \"__main__\":\n    # 12+ features like batching, streaming, etc...\n    server = ls.LitServer(InferenceEngine(max_batch_size=1), accelerator=\"auto\")\n    server.run(port=8000)\n```\n\nDeploy for free to [Lightning cloud](#hosting-options) (or self host anywhere):\n\n```bash\n# Deploy for free with autoscaling, monitoring, etc...\nlightning deploy server.py --cloud\n\n# Or run locally (self host anywhere)\nlightning deploy server.py\n# python server.py\n```\n\nTest the server: Simulate an http request (run this on any terminal):\n```bash\ncurl -X POST http:\u002F\u002F127.0.0.1:8000\u002Fpredict -H \"Content-Type: application\u002Fjson\" -d '{\"input\": 4.0}'\n```\n\n### Agent example\n\n```python\nimport re, requests, openai\nimport litserve as ls\n\nclass NewsAgent(ls.LitAPI):\n    def setup(self, device):\n        self.openai_client = openai.OpenAI(api_key=\"OPENAI_API_KEY\")\n\n    def predict(self, request):\n        website_url = request.get(\"website_url\", \"https:\u002F\u002Ftext.npr.org\u002F\")\n        website_text = re.sub(r'\u003C[^>]+>', ' ', requests.get(website_url).text)\n\n        # ask the LLM to tell you about the news\n        llm_response = self.openai_client.chat.completions.create(\n           model=\"gpt-3.5-turbo\", \n           messages=[{\"role\": \"user\", \"content\": f\"Based on this, what is the latest: {website_text}\"}],\n        )\n        output = llm_response.choices[0].message.content.strip()\n        return {\"output\": output}\n\nif __name__ == \"__main__\":\n    server = ls.LitServer(NewsAgent())\n    server.run(port=8000)\n```\nTest it:\n```bash\ncurl -X POST http:\u002F\u002F127.0.0.1:8000\u002Fpredict -H \"Content-Type: application\u002Fjson\" -d '{\"website_url\": \"https:\u002F\u002Ftext.npr.org\u002F\"}'\n```\n\n \n\n# Key benefits   \n\nA few key benefits:\n\n- **Deploy any pipeline or model**: Agents, pipelines, RAG, chatbots, image models, video, speech, text, etc...\n- **No MLOps glue:** LitAPI lets you build full AI systems (multi-model, agent, RAG) in one place ([more](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fapi-reference\u002Flitapi?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)).   \n- **Instant setup:** Connect models, DBs, and data in a few lines with `setup()` ([more](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fapi-reference\u002Flitapi?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme#setup)).    \n- **Optimized:** autoscaling, GPU support, and fast inference included ([more](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fapi-reference\u002Flitserver?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)).    \n- **Deploy anywhere:** self-host or one-click deploy with Lightning ([more](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fdeploy-on-cloud?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)).\n- **FastAPI for AI:** Built on FastAPI but optimized for AI - 2× faster with AI-specific multi-worker handling ([more]((#performance))).   \n- **Expert-friendly:** Use vLLM, or build your own with full control over batching, caching, and logic ([more](https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-private-llama-3-2-rag-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)).    \n\n> ⚠️ Not a vLLM or Ollama alternative out of the box. LitServe gives you lower-level flexibility to build what they do (and more) if you need it.\n\n \n\n# Featured examples    \nHere are examples of inference pipelines for common model types and use cases.      \n  \n\u003Cpre>\n\u003Cstrong>Toy model:\u003C\u002Fstrong>      \u003Ca target=\"_blank\" href=\"#define-a-server\">Hello world\u003C\u002Fa>\n\u003Cstrong>LLMs:\u003C\u002Fstrong>           \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-llama-3-2-vision-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Llama 3.2\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fopenai-fault-tolerant-proxy-server?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">LLM Proxy server\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-ai-agent-with-tool-use?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Agent with tool use\u003C\u002Fa>\n\u003Cstrong>RAG:\u003C\u002Fstrong>            \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-private-llama-3-2-rag-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">vLLM RAG (Llama 3.2)\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-private-llama-3-1-rag-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">RAG API (LlamaIndex)\u003C\u002Fa>\n\u003Cstrong>NLP:\u003C\u002Fstrong>            \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-any-hugging-face-model-instantly?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Hugging face\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-hugging-face-bert-model?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">BERT\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-text-embedding-api-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Text embedding API\u003C\u002Fa>\n\u003Cstrong>Multimodal:\u003C\u002Fstrong>     \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-open-ai-clip-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">OpenAI Clip\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-multi-modal-llm-with-minicpm?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">MiniCPM\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-phi3-5-vision-api-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Phi-3.5 Vision Instruct\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Fbhimrajyadav\u002Fstudios\u002Fdeploy-and-chat-with-qwen2-vl-using-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Qwen2-VL\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-multi-modal-llm-with-pixtral?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Pixtral\u003C\u002Fa>\n\u003Cstrong>Audio:\u003C\u002Fstrong>          \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-open-ai-s-whisper-model?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Whisper\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-an-music-generation-api-with-meta-s-audio-craft?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">AudioCraft\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-an-audio-generation-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">StableAudio\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-noise-cancellation-api-with-deepfilternet?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Noise cancellation (DeepFilterNet)\u003C\u002Fa>\n\u003Cstrong>Vision:\u003C\u002Fstrong>         \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-private-api-for-stable-diffusion-2?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Stable diffusion 2\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-an-image-generation-api-with-auraflow?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">AuraFlow\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-an-image-generation-api-with-flux?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Flux\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-super-resolution-image-api-with-aura-sr?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Image Super Resolution (Aura SR)\u003C\u002Fa>,\n                \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Fbhimrajyadav\u002Fstudios\u002Fdeploy-background-removal-api-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Background Removal\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-controlled-image-generation-api-controlnet?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Control Stable Diffusion (ControlNet)\u003C\u002Fa>\n\u003Cstrong>Speech:\u003C\u002Fstrong>         \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-voice-clone-api-coqui-xtts-v2-model?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Text-speech (XTTS V2)\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Fbhimrajyadav\u002Fstudios\u002Fdeploy-a-speech-generation-api-using-parler-tts-powered-by-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Parler-TTS\u003C\u002Fa>\n\u003Cstrong>Classical ML:\u003C\u002Fstrong>   \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-random-forest-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Random forest\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-xgboost-with-litserve?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">XGBoost\u003C\u002Fa>\n\u003Cstrong>Miscellaneous:\u003C\u002Fstrong>  \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-an-media-conversion-api-with-ffmpeg?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">Media conversion API (ffmpeg)\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-both-pytorch-and-tensorflow-in-a-single-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">PyTorch + TensorFlow in one API\u003C\u002Fa>, \u003Ca target=\"_blank\" href=\"https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fopenai-fault-tolerant-proxy-server?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme\">LLM proxy server\u003C\u002Fa>\n\u003C\u002Fpre>\n\u003C\u002Fpre>\n\n[Browse 100+ community-built templates](https:\u002F\u002Flightning.ai\u002Fstudios?section=serving&utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)\n\n \n\n# Host anywhere\n\nSelf-host with full control, or deploy with [Lightning AI](https:\u002F\u002Flightning.ai\u002F?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme) in seconds with autoscaling, security, and 99.995% uptime.  \n**Free tier included. No setup required. Run on your cloud**   \n\n```bash\nlightning deploy server.py --cloud\n```\n\nhttps:\u002F\u002Fgithub.com\u002Fuser-attachments\u002Fassets\u002Fff83dab9-0c9f-4453-8dcb-fb9526726344\n\n \n\n# Features\n\n\u003Cdiv align='center'>\n\n| [Feature](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)               | Self Managed                      | [Fully Managed on Lightning](https:\u002F\u002Flightning.ai\u002Fdeploy?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)         |\n|----------------------------------------------------------------------|-----------------------------------|------------------------------------|\n| Docker-first deployment          | ✅ DIY                             | ✅ One-click deploy                |\n| Cost                             | ✅ Free (DIY)                      | ✅ Generous [free tier](https:\u002F\u002Flightning.ai\u002Fpricing?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme) with pay as you go                |\n| Full control                     | ✅                                 | ✅                                 |\n| Use any engine (vLLM, etc.)      | ✅                                 | ✅ vLLM, Ollama, LitServe, etc.    |\n| Own VPC                          | ✅ (manual setup)                  | ✅ Connect your own VPC            |\n| [(2x)+ faster than plain FastAPI](#performance)                                               | ✅       | ✅                                 |\n| [Bring your own model](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Ffull-control?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)              | ✅       | ✅                                 |\n| [Build compound systems (1+ models)](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fhome?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                 | ✅       | ✅                                 |\n| [GPU autoscaling](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fgpu-inference?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                  | ✅       | ✅                                 |\n| [Batching](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fbatching?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                              | ✅       | ✅                                 |\n| [Streaming](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fstreaming?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                            | ✅       | ✅                                 |\n| [Worker autoscaling](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fautoscaling?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                 | ✅       | ✅                                 |\n| [Serve all models: (LLMs, vision, etc.)](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fexamples?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)         | ✅       | ✅                                 |\n| [Supports PyTorch, JAX, TF, etc...](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Ffull-control?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme) | ✅       | ✅                                 |\n| [OpenAPI compliant](https:\u002F\u002Fwww.openapis.org\u002F)                                                | ✅       | ✅                                 |\n| [Open AI compatibility](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fopen-ai-spec?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)             | ✅       | ✅                                 |\n| [MCP server support](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fmcp?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                         | ✅       | ✅                                 |\n| [Asynchronous](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fasync-concurrency?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                 | ✅       | ✅                                 |\n| [Authentication](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures\u002Fauthentication?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                  | ❌ DIY   | ✅ Token, password, custom         |\n| GPUs                             | ❌ DIY                             | ✅ 8+ GPU types, H100s from $1.75  |\n| Load balancing                   | ❌                                 | ✅ Built-in                        |\n| Scale to zero (serverless)       | ❌                                 | ✅ No machine runs when idle       |\n| Autoscale up on demand           | ❌                                 | ✅ Auto scale up\u002Fdown              |\n| Multi-node inference             | ❌                                 | ✅ Distribute across nodes         |\n| Use AWS\u002FGCP credits              | ❌                                 | ✅ Use existing cloud commits      |\n| Versioning                       | ❌                                 | ✅ Make and roll back releases     |\n| Enterprise-grade uptime (99.95%) | ❌                                 | ✅ SLA-backed                      |\n| SOC2 \u002F HIPAA compliance          | ❌                                 | ✅ Certified & secure              |\n| Observability                    | ❌                                 | ✅ Built-in, connect 3rd party tools|\n| CI\u002FCD ready                      | ❌                                 | ✅ Lightning SDK                   |\n| 24\u002F7 enterprise support          | ❌                                 | ✅ Dedicated support               |\n| Cost controls & audit logs       | ❌                                 | ✅ Budgets, breakdowns, logs       |\n| Debug on GPUs                    | ❌                                 | ✅ Studio integration              |\n| [20+ features](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Ffeatures?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme)                    | -                                 | -                                  |\n\n\u003C\u002Fdiv>\n\n \n\n# Performance  \nLitServe is designed for AI workloads. Specialized multi-worker handling delivers a minimum **2x speedup over FastAPI**.    \n\nAdditional features like batching and GPU autoscaling can drive performance well beyond 2x, scaling efficiently to handle more simultaneous requests than FastAPI and TorchServe.\n    \nReproduce the full benchmarks [here](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fhome\u002Fbenchmarks?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme) (higher is better).  \n\n\u003Cdiv align=\"center\">\n  \u003Cimg alt=\"LitServe\" src=\"https:\u002F\u002Fpl-bolts-doc-images.s3.us-east-2.amazonaws.com\u002Fapp-2\u002Fls_charts_v6.png\" width=\"1000px\" style=\"max-width: 100%;\">\n\u003C\u002Fdiv> \n\nThese results are for image and text classification ML tasks. The performance relationships hold for other ML tasks (embedding, LLM serving, audio, segmentation, object detection, summarization etc...).   \n    \n***💡 Note on LLM serving:*** For high-performance LLM serving (like Ollama\u002FvLLM), integrate [vLLM with LitServe](https:\u002F\u002Flightning.ai\u002Flightning-ai\u002Fstudios\u002Fdeploy-a-private-llama-3-2-rag-api?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme), use [LitGPT](https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Flitgpt?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme#deploy-an-llm), or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.\n\n \n\n\n# Community\nLitServe is a [community project accepting contributions](https:\u002F\u002Flightning.ai\u002Fdocs\u002Flitserve\u002Fcommunity?utm_source=litserve_readme&utm_medium=referral&utm_campaign=litserve_readme) - Let's make the world's most advanced AI inference engine.\n\n💬 [Get help on Discord](https:\u002F\u002Fdiscord.com\u002Finvite\u002FXncpTy7DSt)    \n📋 [License: Apache 2.0](https:\u002F\u002Fgithub.com\u002FLightning-AI\u002Flitserve\u002Fblob\u002Fmain\u002FLICENSE)    \n","LitServe 是一个用于构建自定义AI推理服务器的轻量级Python框架，提供了对逻辑、批处理和扩展的完全控制。其核心功能包括自定义推理逻辑、多GPU自动扩展、无MLOps粘合代码以及支持任何PyTorch模型。技术上，它比FastAPI快两倍，并且支持流式处理和简单部署。适用于需要高度定制化推理流程的场景，如构建复杂的AI服务、代理、检索增强生成(RAG)系统或涉及多个模型协同工作的项目中。",2,"2026-06-11 03:24:54","top_topic"]