This bug report replaces bug#19654, which misattributed the bug to libsvm_sparse specifically.

Thanks for the report. Indeed this could be fixed. So I'll tag this accordingly. Would be nice if you could submit a PR to fix it @andrewdelong

/take

I am trying to work on this issue, and trying the workaround as suggested by @andrewdelong

import numpy as np
from sklearn.svm import SVC

X = np.array([[0., 0.], [1., 0.], [0., 1.]])   # or sp.csr_matrix(...)
y = [0, 1, 2]
w = np.array([0., 1., 1.])                          
w[w <= 0] = 1e-16
print(w)
clf = SVC().fit(X, y, w)

However, the results are not as expected

Results

w: [1.e-16 1.e+00 1.e+00]
classes_:  [0 1 2]
support_:  [0 1 2]
support_vectors_:  [[0. 0.]
 [1. 0.]
 [0. 1.]]
n_support_:  [1 1 1]
dual_coef_:  [[ 1.e-16 -1.e-16 -1.e-16]
 [ 1.e-16  1.e+00 -1.e+00]]

Need your suggestions on what to try next? @adrinjalali @andrewdelong

I certainly wouldn't try to change sample weights as a workaround. We probably want to run svm with those classes omitted in this case.

So one way we can do this is to remove values from X, y, and w, where w == 0. Or we can find a way to skip this in training ?

@rand0wn figuring it out would be a part of solving this issue 😁 I haven't jumped deep into that part of the codebase in a while.

Got it will try to figure out ways it's possible, and run the testcases

On Investigation, all the variations except multiclass SVC remove zero_weights

if(svm_type == C_SVC ||
	   svm_type == EPSILON_SVR ||
	   svm_type == NU_SVR ||
	   svm_type == ONE_CLASS)
	{
		PREFIX(problem) newprob;
		// filter samples with negative and null weights
		remove_zero_weight(&newprob, prob);

Either we can do the same with multiclass as well or return an error

For example in LinearSVC

import numpy as np
from sklearn.svm import LinearSVC

X = np.array([[0., 0.], [1., 0.], [0., 1.]])   # or sp.csr_matrix(...)
y = [0, 1, 2]
w = [0., 1., 1.]                               # class 0 has zero weight
clf = LinearSVC().fit(X, y, w)
/Users/abhishek/scikit-learn/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.
  warnings.warn(
WARNING: class label 0 specified in weight is not found

classes_:  [0 1 2]
coef_:  [[-0.66666488  0.66666219]] # 2 Coef for 3 classes
intercept_:  [-2.68435456e-06]
intercept_:  [-2.68435456e-06]

While with Linear Kernel in SVC

import numpy as np
from sklearn.svm import SVC

X = np.array([[0., 0.], [1., 0.], [0., 1.]])   # or sp.csr_matrix(...)
y = np.array([0, 1, 2])
w = np.array([0., 1., 1.])                          

clf = SVC(kernel='linear').fit(X, y, w)

classes_:  [0 1 2]
support_:  [0 1]
support_vectors_:  [[1. 0.]
 [0. 1.]]
n_support_:  [1 1]
dual_coef_:  [[ 1. -1.]]

Also when removing zero-weight classes

import numpy as np
from sklearn.svm import SVC

X = np.array([[0., 0.], [1., 0.], [0., 1.]])   # or sp.csr_matrix(...)
y = np.array([0, 1, 2])
w = np.array([0., 1., 1.])                          
zero_weight_index = (w == 0)
X = X[~zero_weight_index]
y = y[~zero_weight_index]
w = w[~zero_weight_index]

clf = SVC().fit(X, y, w)

classes_:  [1 2]
support_:  [0 1]
support_vectors_:  [[1. 0.]
 [0. 1.]]
n_support_:  [1 1]
dual_coef_:  [[-1.  1.]]

Gives the same thing without warning, my suggestion is either to remove zero weight classes for all to make it consistent or return an error, thoughts?

@glemaitre @adrinjalali

I think making it consistent with the other cases makes sense. Thanks for investigating it @rand0wn

Looking at the code I don't think that the results of the investigation is right.
The negative and null sample-weights are removed even in the multiclass case: C-SVC is the same type for binary or multiclass.

As mentioned in the original post, the dense and sparse case are wrong. The sparse case fail for inconsistent shape but the dense case shows the issue: the internal fit of libsvm was fit only on 2 class and thus we have an issue in the reported values. I did not check the code yet but we should take care of some offsets that should be apply once the fit is done.

The remove_zero_weight should create a mapping to store what samples have been removed because it will create an offset in the support vector index. In addition, we need to find out that we completely remove a specific class and inject this information in the dual_coef_ and n_support_.

All of these need to be done in the svm.cpp file to be meaningful.

At least we have a good non-regression case. But as far I can say, we should suffer from the same issue for the OneClassSVM.