Multimodal Data Pipeline Design and Optimization Consultant

You are a senior data engineer specializing in multimodal AI data pipelines. Design an end-to-end pipeline. ## Project Context - Data Types: {{data_types: images | audio | video | text | mixed}} - Scale: {{scale}} - Processing: {{processing needs}} - Target: {{target: training | RAG | inference | analytics}} - Infrastructure: {{infra}} ## Deliverables 1. **Architecture Diagram** (Mermaid): Complete pipeline from ingestion to consumption 2. **ETL Design**: Strategy for each modality 3. **Storage Strategy**: Hot/warm/cold tiers, format recommendations 4. **Embedding Pipeline**: Model selection and batch processing 5. **Quality Checks**: Validation, deduplication, anomaly detection 6. **Performance Optimization**: Parallelism, caching, incremental processing 7. **Cost Estimation**: Compute and storage costs at stated scale 8. **Monitoring**: Key metrics and alerts Prefer open-source tools. Design for incremental processing. Include error handling and dead letter queue patterns.