Scikit-learn cross validation scoring for regression

I dont have the reputation to comment but I want to provide this link for you and/or a passersby where the negative output of the MSE in scikit learn is discussed - https://github.com/scikit-learn/scikit-learn/issues/2439

In addition (to make this a real answer) your first option is correct in that not only is MSE the metric you want to use to compare models but R^2 cannot be calculated depending (I think) on the type of cross-val you are using.

If you choose MSE as a scorer, it outputs a list of errors which you can then take the mean of, like so:

# Doing linear regression with leave one out cross valfrom sklearn import cross_validation, linear_modelimport numpy as np# Including this to remind you that it is necessary to use numpy arrays rather # than lists otherwise you will get an errorX_digits = np.array(x)Y_digits = np.array(y)loo = cross_validation.LeaveOneOut(len(Y_digits))regr = linear_model.LinearRegression()scores = cross_validation.cross_val_score(regr, X_digits, Y_digits, scoring='mean_squared_error', cv=loo,)# This will print the mean of the list of errors that were output and # provide your metric for evaluationprint scores.mean()

The first one is correct. It outputs the negative of the MSE, as it always tries to maximize the score. Please help us by suggesting an improvement to the documentation.