scikit-learn Binary Classifier

For an AI Engineer phone screen, train a binary classifier on a provided toy dataset using Python and scikit-learn, then improve the initial evaluation metric through model choice or tuning. Problem Statement This is a practical machine learning coding task. You need to build a binary classifier (a "yes/no" predictor) using scikit-learn. First, you will train a basic model on a simple dataset. Then, you will use different methods to make the model perform better. The interviewer expects you to: Pick a dataset from scikit-learn. Train a binary classification model. Check how well the model works using metrics. Write code to improve the results. Explain your choices. This tests if you know the Machine Learning workflow and how to measure success. Step 1: Choose Data and Setup Dataset Options You can use any of these standard datasets that support binary classification: from sklearn.datasets import ( load_breast_cancer, # 2 classes (Malignant/Benign). Good standard choice. load_iris, # 3 classes. You can pick just 2 to make it binary. make_classification, # Creates fake data. You control how hard it is. make_moons, # Data shaped like moons. Good for complex patterns. make_circles # Data shaped like circles. Tests non-linear models. ) Basic Setup Code from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score # Load the dataset data = load_breast_cancer() X, y = data.data, data.target # Split into training and testing sets X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y ) # Standardize features (helps models learn better) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) Step 2: Pick a Model Select a binary classifier and train it. Here are common options: Model Strengths When to Use Logistic Regression Fast and easy to explain. Simple, linear data. Random Forest Handles complex patterns well. Messy data or mixed features. SVM Good for data with many features. Clean, smaller datasets. Gradient Boosting Very accurate. When you need the best performance. Initial Training Code from sklearn.linear_model import LogisticRegression # Train the model model = LogisticRegression(random_state=42, max_iter=1000) model.fit(X_train_scaled, y_train) # Check performance y_pred = model.predict(X_test_scaled) print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}") print(f"Precision: {precision_score(y_test, y_pred):.4f}") print(f"Recall: {recall_score(y_test, y_pred):.4f}") print(f"F1 Score: {f1_score(y_test, y_pred):.4f}") Step 3: Ways to Improve Performance Once you have a baseline score, the interviewer will ask you to improve it. Use these strategies: 1. Tune Hyperparameters Adjust the settings of the model to find the best configuration. from sklearn.model_selection import GridSearchCV param_grid = { 'C': [0.01, 0.1, 1, 10, 100], 'penalty': ['l1', 'l2'], 'solver': ['liblinear', 'saga'] } # Search for the best settings grid_search = GridSearchCV( LogisticRegression(random_state=42, max_iter=1000), param_grid, cv=5, scoring='f1', # Optimize for F1 score n_jobs=-1 ) grid_search.fit(X_train_scaled, y_train) print(f"Best params: {grid_search.best_params_}") print(f"Best CV F1: {grid_search.best_score_:.4f}") 2. Test Other Algorithms Try different models to see if one fits the data better. from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier from sklearn.svm import SVC models = { 'Logistic Regression': LogisticRegression(random_state=42, max_iter=1000), 'Random Forest': RandomForestClassifier(random_state=42), 'Gradient Boosting': GradientBoostingClassifier(random_state=42), 'SVM': SVC(random_state=42) } for name, model in models.items(): model.fit(X_train_scaled, y_train) y_pred = model.predict(X_test_scaled) print(f"{name}: F1={f1_score(y_test, y_pred):.4f}") 3. Create New Features Transform the data to help the model find patterns. from sklearn.preprocessing import PolynomialFeatures from sklearn.feature_selection import SelectKBest, f_classif # Add polynomial features (combinations of existing features) poly = PolynomialFeatures(degree=2, include_bias=False) X_train_poly = poly.fit_transform(X_train_scaled) X_test_poly = poly.transform(X_test_scaled) # Apply same transform to test # Keep only the top 20 best features selector = SelectKBest(f_classif, k=20) X_train_selected = selector.fit_transform(X_train_poly, y_train) X_test_selected = selector.transform(X_test_poly) # Apply same selection to test 4. Fix Class Imbalance If one class appears much more often than the other, adjust the weights so the model pays attention to the smaller class. from sklearn.utils.class_weight import compute_class_weight import numpy as np # Calculate weights based on class frequency class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train) weights_dict = dict(zip(np.unique(y_train), class_weights)) # Train using these weights model = LogisticRegression(class_weight=weights_dict, random_state=42, max_iter=1000) model.fit(X_train_scaled, y_train) 5. Adjust Probability Thresholds Instead of using the default 50% cutoff for a "yes" prediction, find a cutoff that gives a better F1 score. from sklearn.metrics import precision_recall_curve # Get probability scores instead of hard predictions y_proba = model.predict_proba(X_test_scaled)[:, 1] # Find the threshold that maximizes F1 precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba) f1_scores = 2 * (precisions * recalls) / (precisions + recalls + 1e-8) best_threshold = thresholds[np.argmax(f1_scores[:-1])] # Apply the new optimal threshold y_pred_optimal = (y_proba >= best_threshold).astype(int) print(f"Optimal threshold: {best_threshold:.3f}") print(f"F1 with optimal threshold: {f1_score(y_test, y_pred_optimal):.4f}") 6. Use Cross-Validation This ensures your results are reliable and not just luck based on how you split the data. from sklearn.model_selection import cross_val_score # Check performance across 5 different splits cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5, scoring='f1') print(f"CV F1: {cv_scores.mean():.4f} (+/- {cv_scores.std()*2:.4f})") Important Interview Questions 1. Why did you pick this metric? Question: "Why did you choose this score to measure success?" Answer: It depends on the business goal: Accuracy: Use only if classes are equal. Bad for imbalanced data. Precision: Use if False Positives are expensive (e.g., marking a real email as spam). Recall: Use if False Negatives are dangerous (e.g., missing a cancer diagnosis). F1 Score: Use when you need a balance of Precision and Recall. AUC-ROC: Good for general ranking ability. 2. How do you spot overfitting? Question: "How do you know if your model is memorizing data instead of learning?" Answer: Check if the training score is very high but the test score is low. Use learning curves to visualize the gap. Use Cross-Validation to get a stable estimate. from sklearn.model_selection import learning_curve import matplotlib.pyplot as plt train_sizes, train_scores, val_scores = learning_curve( model, X_train_scaled, y_train, cv=5, scoring='f1', train_sizes=np.linspace(0.1, 1.0, 10) ) # Plot these scores to check for gaps 3. Which features matter most? Question: "What data points is the model using to make decisions?" Answer: # For Random Forest / Gradient Boosting importances = model.feature_importances_ indices = np.argsort(importances)[::-1] for i in range(10): print(f"{data.feature_names[indices[i]]}: {importances[indices[i]]:.4f}") # For Logistic Regression coefficients = model.coef_[0] for name, coef in sorted(zip(data.feature_names, coefficients), key=lambda x: abs(x[1]), reverse=True)[:10]: print(f"{name}: {coef:.4f}") 4. How would you put this in production? Question: "What steps are needed to deploy this model?" Answer: Save the model (using pickle or joblib). Check Latency (how fast it predicts). Ensure the data pipeline is consistent (apply the same scaling/transforms). Monitor for Data Drift (changes in data over time). Plan for A/B testing. Have a fallback plan if the model fails.