AI Gateway 多模型智能路由架构设计师

You are a senior AI infrastructure architect specializing in model gateway design. I need to build a production-grade AI gateway that routes requests across multiple LLM providers.

Requirements:

• Supported providers: [OpenAI / Anthropic / Google / DeepSeek / local models]
• Expected RPS: [specify]
• Latency target: [e.g., <200ms overhead]
• Budget constraints: [monthly budget]

Design the following:

1. Routing Strategy

• Cost-optimized routing: Route simple tasks to cheaper models, complex tasks to premium models
• Latency-based routing: Automatic failover when a provider is slow
• Capability-based routing: Match task type (code/creative/analysis) to best model
• Provide the routing decision tree as pseudocode

2. Load Balancing

• Weighted round-robin across providers
• Circuit breaker patterns for provider outages
• Request queue management and backpressure handling
• Rate limit awareness per provider

3. Cost Control

• Per-user/per-team token budgets
• Real-time cost tracking and alerting thresholds
• Automatic downgrade rules when budget is near limit
• Token usage analytics dashboard schema

4. Observability

• Metrics to collect (latency p50/p95/p99, error rates, token usage, cost per request)
• Structured logging format
• Health check endpoints
• Alerting rules

5. Implementation

• Provide a working config file for the gateway
• Include Docker Compose setup for local testing
• Add a simple benchmark script to validate routing logic

Output as a complete, deployable architecture document with configs.