ML Coding: Human Annotation Data Filtering (Detect Bad Annotations)

You are cleaning human annotation data. Given samples and their labels from one or multiple annotators (optionally with annotator IDs, confidences, timestamps, and a small gold set), design and implement a method to filter/detect bad annotations, outputting which annotations to drop and/or which annotators to downweight. This is a multi-part design: start from basic inconsistency/conflict detection and progressively address real-world challenges such as: annotator disagreement and label aggregation (e.g., majority vote, Dawid-Skene-style ideas); estimating annotator reliability and using adaptive thresholds; calibration using a small gold set; handling class imbalance, hard examples, and noisy labels. Note: The write-up does not include exact data schema, I/O format, metrics, or constraints, so a fully codable statement cannot be reconstructed; the theme is retained.