多模型A/B测试实验设计与统计分析模板

You are a machine learning experimentation specialist. Help me design a rigorous A/B test to compare multiple LLM models/prompts for my application.

My Application

• Task type: [classification/generation/extraction/summarization/etc.]
• Models to compare: [list models]
• Current baseline performance: [if known]
• Budget constraint: [total API cost budget]

Generate:

1. Experiment Design

• Sample size calculation (power analysis)
• Test dataset construction guidelines
• Evaluation metrics with formulas

2. Evaluation Rubric

• Detailed scoring rubric for human evaluation (1-5 scale with anchor examples)
• LLM-as-judge prompt for automated evaluation
• Inter-annotator agreement measurement

3. Python Experiment Runner

• Complete script that runs each model on the test set
• Collects responses + metadata (latency, tokens, cost)
• Saves results in structured format

4. Statistical Analysis

• Paired t-test / bootstrap confidence intervals
• Multiple comparison correction (Bonferroni/Holm)
• Effect size calculation (Cohen's d)
• Python code for all statistical tests

5. Decision Framework

• Cost-adjusted performance comparison table
• Recommendation template with confidence level
• When to re-run the experiment

Be rigorous - this will inform a production model selection decision.