ROC and AUC for classification

ROC

Receiver Operating Charasteristic Curve

  • X - False Positive Ratio (range from 0 to 1)

    • FP/(FP+TN)

  • Y - True Positive Ratio (range from 0 to 1)

    • TP/(TP+FN)

  1. Sorting the samples from highest confidence score to lowest.

  2. Using the confidence score as the threshold. Samples with score higher than the threshold are positive. Otherwise, negative.

  3. Connecting the dots we get ROC. The more samples, the smoother the curve.

  4. (0,0) is where we set the threshold at 1.0 and (1,1) is threshold at 0.0.

AUC

Area Under the Curve

The (ROC-)AUC value is equivalent to the probability that a randomly chosen positive example is ranked higher than a randomly chosen negative example.

  • AUC = 1: Perfect classifier. There is a threshold perfectly classifies all the samples.

  • AUC = 0.5: Just like flipping a coin.

  • AUO < 0.5: Worse than flipping a coin. Inverse classification gives better results.

PR-AUC

F1



Micro-F1: Sum up all the TP, FP, TN, FN across multi-class and fo F1.

Macro-F1: Sum up TP, FP, TN, FN per class for F1 then do averaging over all the F1s.