How to use the a k-fold cross validation in scikit with naive bayes classifier and NLTK

Your options are to either set this up yourself or use something like NLTK-Trainer since NLTK doesn't directly support cross-validation for machine learning algorithms.

I'd recommend probably just using another module to do this for you but if you really want to write your own code you could do something like the following.

Supposing you want 10-fold, you would have to partition your training set into 10 subsets, train on 9/10, test on the remaining 1/10, and do this for each combination of subsets (10).

Assuming your training set is in a list named training, a simple way to accomplish this would be,

num_folds = 10subset_size = len(training)/num_foldsfor i in range(num_folds):    testing_this_round = training[i*subset_size:][:subset_size]    training_this_round = training[:i*subset_size] + training[(i+1)*subset_size:]    # train using training_this_round    # evaluate against testing_this_round    # save accuracy# find mean accuracy over all rounds

Actually there is no need for a long loop iterations that are provided in the most upvoted answer. Also the choice of classifier is irrelevant (it can be any classifier).

Scikit provides cross_val_score, which does all the looping under the hood.

from sklearn.cross_validation import KFold, cross_val_scorek_fold = KFold(len(y), n_folds=10, shuffle=True, random_state=0)clf = <any classifier>print cross_val_score(clf, X, y, cv=k_fold, n_jobs=1)

I've used both libraries and NLTK for naivebayes sklearn for crossvalidation as follows:

import nltkfrom sklearn import cross_validationtraining_set = nltk.classify.apply_features(extract_features, documents)cv = cross_validation.KFold(len(training_set), n_folds=10, indices=True, shuffle=False, random_state=None, k=None)for traincv, testcv in cv:    classifier = nltk.NaiveBayesClassifier.train(training_set[traincv[0]:traincv[len(traincv)-1]])    print 'accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv[0]:testcv[len(testcv)-1]])

and at the end I calculated the average accuracy