@ogrisel Do you intend to finish this one? It would be nice to have it prior to #28971.

Two strings in the scorer string "binary_brier_score", "multiclass_brier_score"

I guess the scorer strings would instead be "neg_binary_brier_score" and "neg_multiclass_brier_score" (to enforce the greater is better model selection convention that is common to all our scorer string names) while we would also introduce two binary_brier_score, multiclass_brier_score functions that would probably share some common private helper code under the hood. We would ensure that the docstring of those functions would cross-reference each other explain the difference in normalization constant.

The binary_brier_score would raise if passed a y_observed value (or y_proba_pred) with n_classes > 2 with a message pointing to use the multiclass_ variant instead.

The multiclass_ variants on the other hand could accept binary data without raising or warning I think.

Then we would deprecate the "neg_brier_score" name and brier_score function in favor or the binary_ prefixed counter part.

If that's right I can start working on reviving this PR to implement this plan.

The first step, if I recall correctly, would be to add keyword to brier_score_loss to switch between normalizations.
A 2nd step is about the different strings to select a score. I'm still not too happy about a proliferation of names because a user can always use the the score directly with the normalization parameter.

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 4f00c63. Link to the linter CI: here}

For the changelog, I first tried two entries 22046.feature.rst and 18699.feature.rst with the same content. That indeed works for towncrier, it merges them into one single entry and credits both PR #22046 and #18699. But then unfortunately the "Check Changelog" complains, because I guess #18699 has not been merged, so I only keep this PR entry 22046.feature.rst.

Should we add array api support for multiclass_brier_score_loss and brier_score_loss or do this in a follow up PR ?

Follow-up PR. Let's keep different features well separated by PRs.

@ogrisel @lorentzenchr have you settled on the scorer strings / normalization keyword debate ? It's not clear to me with the above comments whether you prefer:

1. deprecate brier_score_loss in favor of binary_brier_score_loss and corresponding scoring strings
2. introduce a normalization keyword to brier_score_loss

I am -1000 for 2 different functions for binary and multiclass. I want a single function dealing with both, as does log_loss.

Let's keep y_prob, it's still used a lot elsewhere and y_proba sounds like a weird Frenchism.

The normalize keyword was added to brier_score_loss with the meaning "divides by the number of classes".

Then b60cb3b results in the following for brier_score_loss:

• the unnormalized Brier score ranges from 0 to 2
• the normalized Brier score ranges from 0 to 2/n_classes
• labels was added for the multi-class support (required when not all classes appear in y_true)
• pos_label was kept for binary classification (required when y_true contains strings)

There is some ambiguity on binary vs multiclass when n_classes=2, especially about the usage of pos_label vs labels. Currently the following choice has been made:

• targets y_true must be passed as a 1D array (n_samples,)
• pos_label is only used when y_prob.shape=(n_samples,) which is considered as a binary classification task
• labels is only used when y_prob.shape=(n_sample, n_classes) which is considered as a multiclass task (even when n_classes=2)

@lorentzenchr @ogrisel are you happy with the current API ?
If so, I can continue on cleaning up the tests, docs and scoring strings.

I think the API makes sense, apart from the meaning of normalize=True.

However, I am not sure about the normalize=True parameter with 2/n_classes. For three class problems, that would mean Brier score values between 0 and 2/3? Who would want to compute this?

Shall we not just error when normalize=True and n_classes > 2? Then we issue a FutureWarning to state that in the future normalize=False will become the default, meaning that Brier score will always be between 0 and 2, including for binary classification problems.

Alternatively, normalize=True could mean to divide by 2 instead of by n_classes so that normalize=True means that Brier score is always scaled between 0 (best) and 1 (worst), whatever the number of classes.

This parameter could be named scaled_by_half=True instead or something else more explicit than normalize.

I think the API makes sense, apart from the meaning of normalize=True.

However, I am not sure about the normalize=True parameter with 2/n_classes. For three class problems, that would mean Brier score values between 0 and 2/3? Who would want to compute this?

I guess only for the correspondence with the MSE, if we divide by n_classes the Brier score is indeed the MSE. But it doesn't seem to be a common practice, I prefer your scaled_by_half parameter.

What about scaled_by_half with a default "auto" option meaning scaled_by_half=(n_classes==2) ?
The Brier score then follows the convention by default and remains backward compatible.

+1 for scaled_by_half="auto" that divides by 2 whenever n_classes=True.

This way we can automatically switch between the two formula of the Wikipedia page:

https://en.wikipedia.org/wiki/Brier_score

It's also backward compatible with the current implementation, while also giving control to the user if they prefer to use a different convention.

What do you think of this proposal @lorentzenchr?

In my opinion, it's important to keep the goal in sight: one Brier scoring function, ideally with no scaling (always range [0, 2]).

• So yes, I'm ok with scaled_by_half.
• I would rename it to scale_by_half.
• I would start with the default scale_by_half="deprecated" and later set the default to True.

I would rename it to scale_by_half.

+1

I would start with the default scale_by_half="deprecated" and later set the default to True.

To me, scale_by_half=True means a BS range of [0, 1] (instead of [0, 2]) and would typically be used for the binary classification setting where it's common to use the [0, 1] range.

I am ok with raising a FutureWarning to change the scale_by_half default value to False in the future (to always use the [0, 2] range): in the long term this will make our code simpler to maintain by only implementing the convention introduced in the original definition by Brier.