本地LLM模型热切换方案设计师

You are a local LLM deployment specialist focused on multi-model serving and hot-swapping.

My Setup

• Hardware: [GPU model and VRAM / Apple Silicon and RAM / CPU only]
• OS: [macOS / Linux / Windows]
• Models I want to run: [list models, e.g., Qwen3 7B, DeepSeek-V3 8B, Llama 3.3 8B, Gemma 3 4B]
• Use cases: [coding / chat / translation / summarization - which model for which task]
• Acceptable cold-start latency: [< 5s / < 15s / < 30s]

Design a Hot-Swap Architecture:

1. Model Loading Strategy

• Which models should stay resident vs. load on demand?
• Memory budget allocation per model
• Quantization recommendations (Q4, Q5, Q8, FP16) per model based on my VRAM
• KV cache management across model switches

2. Routing Layer

• Design the request routing logic (which prompt to which model)
• Implement automatic model selection based on task type
• Fallback chain when preferred model is loading
• Concurrency handling (queue vs reject vs swap)

3. Implementation Options

Compare and recommend from:

• llama-swap (Go-based proxy for llama.cpp/vllm)
• Ollama with model management
• vLLM with multi-model serving
• LiteLLM proxy with local backends
• Custom solution

4. Configuration Template

Provide a ready-to-use configuration file for the recommended solution, including:

• Model aliases and paths
• Memory limits and swap policies
• Health check endpoints
• Monitoring and logging

Optimize for minimal idle VRAM usage while keeping frequently-used models warm.