LLM应用全链路可观测性监控仪表板设计

You are an LLM operations (LLMOps) expert. Help me design a comprehensive observability dashboard for my LLM-powered application.

Context

I have a production LLM application that uses multiple models (GPT-4o, Claude 3.5, Gemini) via a routing layer. I need full observability.

Requirements

1. Tracing Layer

Design the trace schema:

• Trace ID -> Span hierarchy (user request -> routing decision -> LLM call -> tool calls -> response)
• Capture: model, prompt tokens, completion tokens, latency, cost, temperature, top_p
• Parent-child span relationships for multi-step agent workflows
• Structured logging format (OpenTelemetry compatible)

2. Metrics Dashboard

Define key metrics and their alert thresholds:

• Latency: P50, P95, P99 per model, per endpoint
• Cost: Daily/weekly/monthly burn rate, cost per request, cost per user
• Quality: Success rate, hallucination rate (via eval), user satisfaction scores
• Usage: Requests per minute, token consumption trends, model distribution
• Errors: Rate by error type (rate limit, context overflow, timeout, safety filter)

3. Evaluation Pipeline

Design automated eval workflows:

• Factuality checks against ground truth
• Relevance scoring (query-response alignment)
• Safety/toxicity screening
• Regression detection on prompt template changes
• A/B testing framework for model/prompt variants

4. Alerting Rules

Provide specific alerting configurations:

• Cost spike > 2x daily average
• Latency P95 > 5s for 5 consecutive minutes
• Error rate > 5% over 10-minute window
• Eval score drop > 10% on any dimension

5. Implementation

Recommend tech stack and provide:

• Docker Compose setup for self-hosted monitoring
• Integration code snippets for Python (OpenAI SDK, Anthropic SDK)
• Grafana dashboard JSON template
• Cost allocation tagging strategy

Output a complete implementation guide with architecture diagrams, config files, and deployment instructions.