LLM application observability monitoring solution designer

You are an expert LLM Observability Architect. Help me design a comprehensive observability strategy for my LLM application. ## Context - Application type: [chatbot / RAG / agent / code assistant] - Scale: [requests per day] - Models used: [GPT-4 / Claude / local models] - Current pain points: [latency / cost / quality / debugging] ## Your Tasks 1. **Trace Design**: Design a tracing schema that captures the full lifecycle of each LLM request (prompt construction → model call → post-processing → response). Include parent-child span relationships for multi-step agent workflows. 2. **Key Metrics Dashboard**: Define the top 10 metrics I should track: - Latency percentiles (p50, p95, p99) - Token usage and cost per request/user/feature - Error rates and retry patterns - Model quality scores (user feedback, auto-eval) - Cache hit rates 3. **Prompt Version Management**: Design a prompt versioning strategy: - How to A/B test prompt variants - Rollback procedures - Performance comparison framework 4. **Evaluation Pipeline**: Create an automated eval framework: - Define eval criteria (relevance, faithfulness, toxicity) - Design golden dataset management - Set up regression detection alerts 5. **Cost Optimization**: Analyze current usage and recommend: - Model routing strategies (cheap model for simple queries) - Caching layers (semantic cache design) - Token optimization techniques 6. **Alert Rules**: Define actionable alert thresholds for: - Latency spikes - Cost anomalies - Quality degradation - Error rate increases Output a complete implementation plan with architecture diagrams (in Mermaid), code snippets for instrumentation, and a 30-day rollout timeline.