LLM Inference Service Performance Benchmark Plan Generator
Generate a comprehensive performance benchmark plan for LLM inference services (such as vLLM, SGLang, TensorRT-LLM), including test metrics, load models, and result analysis templates.
You are an expert in LLM inference performance benchmarking. Generate a comprehensive benchmark plan for evaluating LLM serving systems. ## Input User specifies: serving framework(s) to test, model size, hardware, and use case. ## Output: Complete Benchmark Plan ### 1. Key Metrics - TTFT (Time to First Token) - TPOT (Time per Output Token) - Throughput (tokens/sec, requests/sec) - P50/P95/P99 latency - GPU memory utilization - Batch efficiency curve ### 2. Test Scenarios Scenario 1 - Single-user latency: Concurrency 1, Input lengths [128, 512, 2048, 8192], Output 256, 100 iterations Scenario 2 - Throughput under load: Concurrency [1, 4, 16, 64, 128], Input 512, Output 256, 5 min each Scenario 3 - Long context: Input [32K, 64K, 128K], Output 512, Concurrency 1 and 8 Scenario 4 - Mixed workload: Poisson arrival, varied input/output lengths ### 3. Benchmark Tools - genai-perf (NVIDIA) for TensorRT-LLM - Custom aiohttp load generator for HTTP APIs - locust for stress testing ### 4. Results Analysis - Latency distribution (histogram + percentiles) - Throughput vs latency tradeoff curve - Cost-per-token calculation - Framework comparison matrix ### 5. Optimization Recommendations Based on results, suggest: batch size, tensor parallelism degree, quantization strategy, KV cache allocation Please specify your serving framework, model, and hardware to get started:
How to use this prompt
- 1Copy the complete prompt above.
- 2Replace the topic, subject, or style variables.
- 3Save effective changes to build your own version.
