LLM application full-link observable monitoring dashboard design

You are an LLM operations (LLMOps) expert. Help me design a comprehensive observability dashboard for my LLM-powered application. ## Context I have a production LLM application that uses multiple models (GPT-4o, Claude 3.5, Gemini) via a routing layer. I need full observability. ## Requirements ### 1. Tracing Layer Design the trace schema: - Trace ID -> Span hierarchy (user request -> routing decision -> LLM call -> tool calls -> response) - Capture: model, prompt tokens, completion tokens, latency, cost, temperature, top_p - Parent-child span relationships for multi-step agent workflows - Structured logging format (OpenTelemetry compatible) ### 2. Metrics Dashboard Define key metrics and their alert thresholds: - **Latency**: P50, P95, P99 per model, per endpoint - **Cost**: Daily/weekly/monthly burn rate, cost per request, cost per user - **Quality**: Success rate, hallucination rate (via eval), user satisfaction scores - **Usage**: Requests per minute, token consumption trends, model distribution - **Errors**: Rate by error type (rate limit, context overflow, timeout, safety filter) ### 3. Evaluation Pipeline Design automated eval workflows: - Factuality checks against ground truth - Relevance scoring (query-response alignment) - Safety/toxicity screening - Regression detection on prompt template changes - A/B testing framework for model/prompt variants ### 4. Alerting Rules Provide specific alerting configurations: - Cost spike > 2x daily average - Latency P95 > 5s for 5 consecutive minutes - Error rate > 5% over 10-minute window - Eval score drop > 10% on any dimension ### 5. Implementation Recommend tech stack and provide: - Docker Compose setup for self-hosted monitoring - Integration code snippets for Python (OpenAI SDK, Anthropic SDK) - Grafana dashboard JSON template - Cost allocation tagging strategy Output a complete implementation guide with architecture diagrams, config files, and deployment instructions.