Architect for Unified Compute Engine for Big Data and AI Workloads

You are a senior data infrastructure architect specializing in unified compute engines. Help me design a modern data platform that consolidates batch processing, stream processing, and AI/ML workloads into a single engine. ## Current Pain Points - Separate clusters for Spark (batch), Flink (streaming), and Ray (ML) - Data duplication across systems - High operational cost maintaining 3+ compute frameworks - Slow iteration: moving data between batch and ML pipelines ## Design Requirements 1. **Unified Query Layer**: Single SQL interface for batch queries, streaming aggregations, and ML feature computation 2. **Compute Architecture**: - Analyze Rust-based alternatives to JVM compute engines - Arrow-native columnar processing - GPU acceleration for AI workloads within the same engine 3. **Migration Plan**: Generate a phased migration from Spark/Flink/Ray: - Phase 1: Batch SQL workloads - Phase 2: Streaming pipelines - Phase 3: ML training and inference 4. **Performance Benchmarks**: Design benchmark suite comparing: - TPC-DS queries vs Spark - Streaming throughput vs Flink - ML pipeline latency vs Ray 5. **Cost Analysis**: TCO comparison over 12 months 6. **Risk Assessment**: Compatibility gaps, missing connectors, team skill gaps My current stack: [DESCRIBE CURRENT INFRASTRUCTURE] Data volume: [DAILY DATA VOLUME] Team size: [ENGINEERING TEAM SIZE]