sklearn, Keras, DeepStack - ValueError: multi_class must be in ('ovo', 'ovr')

Question

I trained a set of DNNs and I want to use them in a deep ensemble. The code is implemented in TF2, but the package deepstack works with Keras as well. The code looks something like this

from deepstack.base import KerasMember
from deepstack.ensemble import DirichletEnsemble

dirichletEnsemble = DirichletEnsemble(N=2000 * ensemble_size)

for net_idx in range(0,ensemble_size):  

   member = KerasMember(name=model_name, keras_model=model,
                         train_batches=(train_images,train_labels), val_batches=(valid_images, valid_labels))

   dirichletEnsemble.add_member(member)

dirichletEnsemble.fit()

where 'model' is essentially a Keras model, thus you need to load one model at each loop (I am using my own implementation). 'ensemble_size' represents the number of DNNs used in the ensemble.

As a result, I get the following error

ValueError: multi_class must be in ('ovo', 'ovr')

which is generated by the sklearn package.

FURTHER DETAILS: deepstack creates a metric

metric = metrics.roc_auc_score

and then returns it as

return metric(y_t, y_p)

which then calls sklearn

if multi_class == 'raise':
      raise ValueError("multi_class must be in ('ovo', 'ovr')")

In my specific case, the labels are respectively y_t

[ 7 10 18 52 10 13 10 4 7 7 24 26 7 26 13 13]

and y_p

[ 73 250 250 250 281 281 250 281 281 174 281 250 281 250 250 250]

How do I set multi_class as 'ovo' or 'ovr'?

Answer 1

The documentation for roc_auc_score indicates the following:

roc_auc_score(
    y_true,
    y_score,
    *,
    average='macro',
    sample_weight=None,
    max_fpr=None,
    multi_class='raise',
    labels=None
)

The second last parameter there is multi_class, which has the following explanation:

Multiclass only. Determines the type of configuration to use. The default value raises an error, so either 'ovr' or 'ovo' must be passed explicitly.

So, it seems that there is some variation in how roc auc is calculated and they are forcing you to explicitly choose which variation you want them to use. If you don't make the choice, the default will result in an exception being raised. And that exception is the error that you are reporting in your question title.