XGBoost hyperparameter tunning with RandomizedSearchCV with multiple classes

Question

I am doing a hyperparameter tunning for my model and my code is like this:

para_tunning = {
        'learning_rate': [0.01,0.05,0.1],
        'min_child_weight': [1, 5, 10],
        'gamma': [0.5, 1, 1.5, 2, 5],
        'subsample': [0.6, 0.8, 1.0],
        'colsample_bytree': [0.6, 0.8, 1.0],
        'max_depth': [3, 4, 5, 6, 7, 8, 9, 10],
        "n_estimators": [100, 200, 300, 400, 500],
        "objective": "multi:softmax",
        "aplha":[0,2,4,6,8]
        }

clf_rndcv = RandomizedSearchCV(clf, 
                         param_distributions = para_tunning,
                         cv = 5,  
                         n_iter = 5,
                         scoring = 'accuracy', 
                         error_score = 0, 
                         verbose = 3, 
                         n_jobs = -1,
                         random_state = 42)
clf_rndcv.fit(X_train, y_train)

It shows that Fitting 5 folds for each of 5 candidates, totalling 25 fits, I suppose it just randomly pick 5 from the para_tunning dict and do a 5 fold cv? If I want to test all of the parameter do I switch over to gridsearchcv? And any suggestion for the tunning? I am doing a multiple class classifier with 100 classes, 500 sample per classes: so 50000 in total. Thanks!

Answer 1

Yes, if you want to search for ALL the hyperparameters you have to use GridSearchCV. GridSearch searches all possible combinations of the hyperparameters (it can be quite large given your case).

For your information, XGBoost has it own hyperparameters tuning. XGBoost CV