LLM 多模型混合推理集群部署方案生成器

You are an expert LLM infrastructure architect. Help me design a multi-model inference cluster deployment plan.

My Setup

• Hardware: [describe GPU/CPU resources, e.g., 4x A100 80GB, 2x RTX 4090]
• Budget: [monthly budget]
• Use cases: [list your use cases, e.g., code gen, RAG Q&A, translation, summarization]
• Expected QPS: [queries per second per use case]
• Latency requirement: [e.g., <2s for chat, <10s for code gen]

Generate the following:

1. Model Selection Matrix

Use Case	Recommended Model	Size	Quantization	Why

2. Hardware Allocation

• Which model runs on which GPU
• Memory budget per model
• Batch size recommendations
• KV cache allocation strategy

3. Routing Strategy

• Semantic router rules (which query → which model)
• Fallback chain (primary → secondary → tertiary)
• Load balancing algorithm recommendation
• Cost-aware routing rules

4. Serving Stack

• Inference engine (vLLM / TensorRT-LLM / SGLang)
• Gateway/proxy (LiteLLM / Bifrost / custom)
• Monitoring (Langfuse / Prometheus metrics)
• Auto-scaling triggers

5. Cost Analysis

| Model | GPU Hours/day | Est. Monthly Cost | Cost per 1K tokens |

6. Docker Compose / K8s Manifest

Provide a ready-to-deploy configuration file.

7. Optimization Tips

• Speculative decoding opportunities
• Prefix caching strategy
• Continuous batching tuning

Be practical and specific. Prefer open-source solutions.