GridSearchCV & RandomizedSearchCV has a ranking issue when test scores are same. #21895
Description
Describe the bug
During best parameter candidate search when two parameter candidates have the same mean_test_<scorer_name> then the rank_test_<scorer_name> ranks them the same, and if both have rank 1 then the algorithm picks the one that come first.
Steps/Code to Reproduce
from sklearn import svm, datasets
import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()
def grid_search(parameters):
svc = svm.SVC()
clf = GridSearchCV(svc, parameters,return_train_score=True)
clf.fit(iris.data, iris.target)
print(pd.DataFrame(clf.cv_results_)[['param_C','param_kernel','mean_test_score','rank_test_score','mean_train_score']])
print('best_parameters = ',clf.best_params_,'\n')
params = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
grid_search(params)
params = {'kernel':('rbf', 'linear'), 'C':[10, 1]}
grid_search(params)Expected Results
Both function calls should return the same output for clf.best_params_
i.e best_parameters = {'C': 1, 'kernel': 'linear'} because the difference between the mean_train_score and the mean_test_score is the least for this parameter candidate.
Actual Results
Versions
System:
python: 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]
executable: C:\Anaconda3\anaconda3\envs\sklearndev\python.exe
machine: Windows-10-10.0.19043-SP0
Python dependencies:
pip: 21.2.4
setuptools: 58.0.4
sklearn: 1.1.dev0
numpy: 1.21.2
scipy: 1.7.1
Cython: 0.29.24
pandas: 1.3.4
matplotlib: 3.5.0
joblib: 1.1.0
threadpoolctl: 3.0.0
Built with OpenMP: True
threadpoolctl info:
user_api: blas
internal_api: mkl
prefix: mkl_rt
filepath: C:\Anaconda3\anaconda3\envs\sklearndev\Library\bin\mkl_rt.1.dll
version: 2021.4-Product
threading_layer: intel
num_threads: 2
user_api: openmp
internal_api: openmp
prefix: vcomp
filepath: C:\Windows\System32\vcomp140.dll
version: None
num_threads: 4