Multi-Model Benchmark Auto-Evaluation Script Generator

You are an LLM benchmarking engineer. Generate a complete, runnable evaluation script based on my requirements. ## My Evaluation Needs: - **Models to compare**: {MODEL_LIST} (e.g., GPT-4o, Claude Sonnet, Gemini 2.5 Pro, Qwen3) - **Task type**: {TASK_TYPE} (e.g., code generation, reasoning, summarization, translation, multi-turn dialogue) - **Test dataset**: {DATASET_DESCRIPTION} (e.g., 50 coding problems from LeetCode medium, 100 news articles for summarization) - **Metrics**: {METRICS} (e.g., accuracy, latency, token cost, BLEU score, human preference) - **Budget constraint**: {BUDGET} (e.g., $50 total, or unlimited) ## Generate: 1. **Python evaluation script** using litellm or openai SDK for unified API calls 2. **Test case loader** (support JSON/JSONL input format) 3. **Scoring functions** for each metric with clear rubrics 4. **Rate limiting & retry logic** to handle API throttling 5. **Results aggregation** with: - Per-model scores (mean, median, p95) - Statistical significance tests (paired t-test or bootstrap) - Cost-per-quality analysis 6. **Visualization code** (matplotlib/plotly) generating: - Radar chart comparing models across dimensions - Box plots for score distribution - Latency vs quality scatter plot 7. **README** explaining how to run, configure API keys, and interpret results Output the complete project structure with all files. Use modern Python (3.11+), type hints, and async where beneficial.