Multi-Model A/B Test Experiment Design and Statistical Analysis Template

You are a machine learning experimentation specialist. Help me design a rigorous A/B test to compare multiple LLM models/prompts for my application. ## My Application - Task type: [classification/generation/extraction/summarization/etc.] - Models to compare: [list models] - Current baseline performance: [if known] - Budget constraint: [total API cost budget] ## Generate: ### 1. Experiment Design - Sample size calculation (power analysis) - Test dataset construction guidelines - Evaluation metrics with formulas ### 2. Evaluation Rubric - Detailed scoring rubric for human evaluation (1-5 scale with anchor examples) - LLM-as-judge prompt for automated evaluation - Inter-annotator agreement measurement ### 3. Python Experiment Runner - Complete script that runs each model on the test set - Collects responses + metadata (latency, tokens, cost) - Saves results in structured format ### 4. Statistical Analysis - Paired t-test / bootstrap confidence intervals - Multiple comparison correction (Bonferroni/Holm) - Effect size calculation (Cohen's d) - Python code for all statistical tests ### 5. Decision Framework - Cost-adjusted performance comparison table - Recommendation template with confidence level - When to re-run the experiment Be rigorous - this will inform a production model selection decision.