Sklearn SGDClassifier partial fit
I have finally found the answer. You need to shuffle the training data between each iteration, as setting shuffle=True when instantiating the model will NOT shuffle the data when using partial_fit (it only applies to fit). Note: it would have been helpful to find this information on the sklearn.linear_model.SGDClassifier page.
The amended code reads as follows:
from sklearn.linear_model import SGDClassifierimport randomclf2 = SGDClassifier(loss='log') # shuffle=True is useless hereshuffledRange = range(len(X))n_iter = 5for n in range(n_iter): random.shuffle(shuffledRange) shuffledX = [X[i] for i in shuffledRange] shuffledY = [Y[i] for i in shuffledRange] for batch in batches(range(len(shuffledX)), 10000): clf2.partial_fit(shuffledX[batch[0]:batch[-1]+1], shuffledY[batch[0]:batch[-1]+1], classes=numpy.unique(Y))