LLM Multimodal Inference Service Performance Optimization Plan

You are a senior ML infrastructure engineer specializing in high-throughput LLM serving systems. Design a comprehensive optimization plan for deploying an omni-modal model (text + vision + audio) inference service with the following constraints: **Current Setup**: - Model: 70B parameter omni-modal model (similar to GPT-4o architecture) - Hardware: 4x NVIDIA H100 80GB GPUs, 512GB system RAM - Target: 200 concurrent users, <2s time-to-first-token, 60 tokens/s throughput **Optimize across these dimensions**: 1. **Quantization Strategy**: Compare AWQ vs GPTQ vs FP8 for this model class. Impact on quality per modality. Memory savings vs quality tradeoff analysis. 2. **Batching & Scheduling**: Continuous batching implementation. Priority queue design for mixed-modality requests. Variable-length image/audio input handling. 3. **KV Cache Management**: PagedAttention configuration. Cache eviction policy for long conversations with media. Prefix caching for common system prompts. 4. **Tensor Parallelism**: Optimal TP degree for 4-GPU setup. Pipeline vs tensor parallelism tradeoffs. NVLink topology-aware placement. 5. **Monitoring & Autoscaling**: Key metrics (TTFT, TPS, queue depth, GPU util). Autoscaling triggers. Graceful degradation under load. For each optimization, provide expected speedup, implementation complexity, risk assessment, and recommended order. Output as a prioritized optimization roadmap with estimated timeline.