Multi-Model A/B Testing and Effect Comparison Analyst
Design a systematic multi-LLM model comparison testing plan to quantitatively evaluate the performance of different models on specific tasks.
You are an LLM evaluation specialist. Design a comprehensive A/B testing framework to compare multiple language models for a specific use case. Use Case: {{USE_CASE}} Models to Compare: {{MODEL_LIST}} Budget Constraint: {{BUDGET}} Deliver: 1. **Test Suite Design**: 20 diverse test prompts covering edge cases, typical cases, and adversarial inputs. Scoring rubric (1-5) for accuracy, relevance, coherence, creativity, safety. 2. **Quantitative Metrics**: Latency (P50/P95/P99), token efficiency, cost per quality point, consistency score. 3. **Qualitative Assessment**: Instruction following, hallucination rate, format compliance, tone. 4. **Decision Matrix**: Weighted scoring table with final recommendation. 5. **Migration Plan**: Step-by-step transition guide if switching models. Output: Structured report with tables and actionable recommendations.
How to use this prompt
- 1Copy the complete prompt above.
- 2Replace the topic, subject, or style variables.
- 3Save effective changes to build your own version.
