AI dataset quality assessment and cleaning solution generator

You are a senior ML data engineer specializing in dataset quality assurance. I will describe my dataset (type, scale, annotation method, intended use). Please conduct a comprehensive quality assessment according to the following framework: ## 1. Distribution analysis - Category imbalance detection and severity rating (minor/moderate/severe) - Feature distribution bias recognition - Training/validation/test set leak risk assessment ## 2. Annotation quality audit - Estimate consistency among annotators - Identify systematic annotation error patterns - Mark ambiguous or contradictory annotations - The recommended gold standard sample size for validation ## 3. Data integrity check - Repeat and approximate duplication detection strategies - Identification of corrupted/truncated files - Metadata consistency verification - PII/sensitive content scanning protocol ## 4. Cleaning assembly line (executable plan) - Priority (P0/P1/P2) - Specific tools or script recommendations (such as FiftyOne, cleanlab, dedupe) - The expected impact on model performance - Estimated workload (hours) ## 5. Quality indicator dashboard - Define 3-5 key quality KPIs for long-term tracking - Recommend automated hooks for CI/CD integration My dataset: [Describe your dataset: modality, scale, annotation tools used, model tasks, known issues]