Why SGDClassifier with hinge loss is faster than SVC implementation in scikit-learn

Question

As we know For the support vector machine we can use SVC as well as SGDClassifier with hinge loss implementation. Is SGDClassifier with hinge loss implementation is faster than SVC. Why?

Links of both implementations of SVC in scikit-learn:
SVC
SGDClassifier

I read on the documentation page of the sci-kit learn that SVC uses some algorithm of libsvm library for optimization. While SGDClassifier uses SGD(obviously).

Answer 1

Maybe it is better to start trying some practical cases and read the code. Let's start...

First of all, if we read the documentation of SGDC, it says the linear SVM is used only:

Linear classifiers (SVM, logistic regression, a.o.) with SGD training

What if instead of using the usual SVC, we use the LinearSVC?

Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.

Let's add an example for the three types of algorithms:

from sklearn.svm import SVC
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC
from sklearn import datasets
import numpy as np

iris = datasets.load_iris()
X = np.random.rand(20000,2)

Y = np.random.choice(a=[False, True], size=(20000, 1))

# hinge is used as the default
svc = SVC(kernel='linear')

sgd = SGDClassifier(loss='hinge')

svcl = LinearSVC(loss='hinge')

Using jupyter and the command %%time we get the execution time (you can use similar ways in normal python, but this is how I did it):

%%time
svc.fit(X, Y)

Wall time: 5.61 s

%%time
sgd.fit(X, Y)

Wall time: 24ms

%%time
svcl.fit(X, Y)

Wall time: 26.5ms

As we can see there is a huge difference between all of them, but linear and SGDC have more or less the same time. The time keeps being a little bit different, but this will always happen since the execution of each algorithm does not come from the same code.

If you are interested in each implementation, I suggest you read the github code using the new github reading tool which is really good!

Code of linearSVC

Code of SGDC