Why accuracy is not enough

A classifier that predicts "no" for every sample can look impressive on an imbalanced dataset. Before celebrating a score, ask what the model is measured on and what mistakes cost in the real world. This note is a placeholder essay that demonstrates the article layout, code, tables, images and math.

Start with the split

Keep a test set untouched until the very end. If you tune against it, it stops being a test set and becomes another training set.

  • Use a stratified split when classes are imbalanced.
  • Group related samples together when they come from the same user or session.
  • Record the random seed and the exact preprocessing steps.
from sklearn.model_selection import train_test_split
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

Pick metrics that match the cost

Different errors have different consequences. A spam filter and a medical screening tool should not optimize the same number.

MetricBest whenWatch out for
AccuracyBalanced classes, equal error costMisleading on imbalanced data
PrecisionFalse positives are expensiveCan hide missed positives
RecallFalse negatives are expensiveCan flood you with false alarms
F1You need a single balanced scoreStill hides the precision/recall trade-off

The F1 score is the harmonic mean of precision and recall:

F1=2⋅precision⋅recallprecision+recallF_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

A metric is a compressed story. Always open it up and look at the examples behind the number.

Look at the errors

Plot a confusion matrix, then read a sample of the mistakes by hand. You will often find a data issue, a label ambiguity or a subgroup that the average score hides.

Placeholder chart showing model metrics
Placeholder figure: replace with a real evaluation chart before publishing.

A short checklist

  1. Freeze the test set before tuning.
  2. Report more than one metric.
  3. Break results down by important subgroups.
  4. Inspect misclassified examples.
  5. Write down what the model should not be used for.

Replace this placeholder article with your own writing before the site goes live.