What's the difference between Keras' AUC(curve='PR') and Scikit-learn's average_precision_score?

Question

I am quite confused on the difference between Keras' AUC(curve='PR') and Scikit-learn's average_precision_score. My objective is to compute the Area Under the Precision-Recall Curve (AUPRC), for both Scikit-learn and Keras models. However, these two metrics yield vastly different results!

Did I miss something out on the TensorFlow-Keras documentation at https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC, with regards to the use of the AUC function?

Answer 1

As stated in the Scikit-learn documentation, they use a different implementation method:

References [Manning2008] and [Everingham2010] present alternative variants of AP that interpolate the precision-recall curve. Currently, average_precision_score does not implement any interpolated variant. References [Davis2006] and [Flach2015] describe why a linear interpolation of points on the precision-recall curve provides an overly-optimistic measure of classifier performance. This linear interpolation is used when computing area under the curve with the trapezoidal rule in auc.

In the average_precision_score function documentation, you can also read:

This implementation is not interpolated and is different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic.

I encourage you to look in detail at the different functions and their descriptions available in the metrics module. I also highly recommend to read the related paper.

Lastly, there's also a potentially interested thread here: [AUC] result of tf.metrics.auc doesnot match with sklearn's.