📄️ Quick Start
Quick start CLI, Config, Docker
📄️ 🐳 Docker, Deploying LiteLLM Proxy
You can find the Dockerfile to build litellm proxy here
📄️ ⚡ Best Practices for Production
Expected Performance in Production
📄️ Proxy Config.yaml
Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.
🔗 📖 All Endpoints
📄️ ✨ Enterprise Features - Content Mod
Features here are behind a commercial license in our /enterprise folder. See Code
📄️ Use with Langchain, OpenAI SDK, LlamaIndex, Curl
Input, Output, Exceptions are mapped to the OpenAI format for all supported models
📄️ 🔑 Virtual Keys
Track Spend, and control model access via virtual keys for the proxy
📄️ 💰 Budgets, Rate Limits
Requirements:
📄️ 👥 Team-based Routing + Logging
Routing
📄️ [BETA] Proxy UI
Create + delete keys through a UI
📄️ 🚨 Budget Alerting
Alerts when a project will exceed it’s planned limit
📄️ Cost Tracking - Azure
Set base model for cost tracking azure image-gen call
📄️ [BETA] JWT-based Auth
Use JWT's to auth admins / projects into the proxy.
🗃️ 🔥 Load Balancing
2 items
📄️ Model Management
Add new models + Get model info without restarting proxy.
📄️ Health Checks
Use this to health check all LLMs defined in your config.yaml
📄️ Debugging
2 levels of debugging supported.
📄️ PII Masking
LiteLLM supports Microsoft Presidio for PII masking.
📄️ Prompt Injection
LiteLLM supports similarity checking against a pre-generated list of prompt injection attacks, to identify if a request contains an attack.
📄️ Caching
Cache LLM Responses
🗃️ Logging, Alerting
3 items
📄️ Modify / Reject Incoming Requests
- Modify data before making llm api calls on proxy
📄️ Post-Call Rules
Use this to fail a request based on the output of an llm api call.
📄️ CLI Arguments
Cli arguments, --host, --port, --num_workers