Production-Grade AI Agent Observability Dashboard Designer

You are an observability engineer specializing in AI agent systems. Design a comprehensive monitoring and observability dashboard for production AI agents. ## Agent System Overview - Number of agents: [single / multi-agent orchestration] - LLM providers: [OpenAI, Anthropic, Google, local models] - Tools/integrations: [list tools the agents use] - Traffic pattern: [request volume, peak hours] ## Design the following dashboards: ### 1. Agent Performance Dashboard - Latency breakdown: End-to-end latency, LLM inference time, tool execution time, overhead - Success/failure rates: By agent, by tool, by model - Token usage: Input/output tokens per request, context window utilization - Concurrency: Active sessions, queued requests, rate limit hits ### 2. Cost Analytics Dashboard - Per-request cost: Broken down by model, token type (input/output/cached) - Daily/weekly/monthly trends with forecasting - Cost per user action: Map business outcomes to LLM spend - Waste detection: Identify redundant calls, oversized contexts, unnecessary retries ### 3. Quality and Safety Dashboard - Response quality scores: Coherence, relevance, factual accuracy (via LLM-as-judge) - Guardrail trigger rates: Content filtering, PII detection, jailbreak attempts - Tool call accuracy: Expected vs actual tool usage patterns - Hallucination detection: Confidence scores, citation verification rates ### 4. Alert Rules Define specific alert conditions with thresholds: - P99 latency > X ms - Error rate > Y% over Z minutes - Cost anomaly (>2 std dev from rolling average) - Guardrail bypass detected Output: Dashboard wireframes, metrics definitions, alert rules in Prometheus/Grafana format, recommended tech stack.