AI Gateway Multi-Model Intelligent Routing Architect

You are a senior AI infrastructure architect specializing in model gateway design. I need to build a production-grade AI gateway that routes requests across multiple LLM providers. Requirements: - Supported providers: [OpenAI / Anthropic / Google / DeepSeek / local models] - Expected RPS: [specify] - Latency target: [e.g., <200ms overhead] - Budget constraints: [monthly budget] Design the following: ## 1. Routing Strategy - Cost-optimized routing: Route simple tasks to cheaper models, complex tasks to premium models - Latency-based routing: Automatic failover when a provider is slow - Capability-based routing: Match task type (code/creative/analysis) to best model - Provide the routing decision tree as pseudocode ## 2. Load Balancing - Weighted round-robin across providers - Circuit breaker patterns for provider outages - Request queue management and backpressure handling - Rate limit awareness per provider ## 3. Cost Control - Per-user/per-team token budgets - Real-time cost tracking and alerting thresholds - Automatic downgrade rules when budget is near limit - Token usage analytics dashboard schema ## 4. Observability - Metrics to collect (latency p50/p95/p99, error rates, token usage, cost per request) - Structured logging format - Health check endpoints - Alerting rules ## 5. Implementation - Provide a working config file for the gateway - Include Docker Compose setup for local testing - Add a simple benchmark script to validate routing logic Output as a complete, deployable architecture document with configs.