Identify Low-Quality Annotators Under Label Noise and Evaluate Classifier Improvement

Problem: ML Coding — Noisy Human Labels and Annotator Quality You have a classification dataset labeled by multiple human annotators. Some annotators are low-quality and introduce label noise that hurts the classifier. Implement a pipeline (or complete functions in a provided scaffold) to: Identify low-quality annotators based on agreement with others, model predictions, consistency statistics, etc. Filter/downweight those annotators' labels to produce a cleaner training set (e.g., re-aggregation per item via voting or annotator weighting). Retrain and evaluate the classifier and determine whether performance improves. Notes The exact metric (accuracy/F1/AUC), data format (multiple labels per item), and baselines are provided in the interview. Focus on writing runnable analysis/training code and a defensible statistical approach.