13
我有一个具有〜300点和32个不同标签的数据集,我想通过使用网格搜索和LabelKFold验证绘制其学习曲线来评估LinearSVR模型。如何嵌套LabelKFold?
我的代码看起来是这样的:
import numpy as np
from sklearn import preprocessing
from sklearn.svm import LinearSVR
from sklearn.pipeline import Pipeline
from sklearn.cross_validation import LabelKFold
from sklearn.grid_search import GridSearchCV
from sklearn.learning_curve import learning_curve
...
#get data (x, y, labels)
...
C_space = np.logspace(-3, 3, 10)
epsilon_space = np.logspace(-3, 3, 10)
svr_estimator = Pipeline([
("scale", preprocessing.StandardScaler()),
("svr", LinearSVR),
])
search_params = dict(
svr__C = C_space,
svr__epsilon = epsilon_space
)
kfold = LabelKFold(labels, 5)
svr_search = GridSearchCV(svr_estimator, param_grid = search_params, cv = ???)
train_space = np.linspace(.5, 1, 10)
train_sizes, train_scores, valid_scores = learning_curve(svr_search, x, y, train_sizes = train_space, cv = ???, n_jobs = 4)
...
#plot learning curve
我的问题是如何设置的CV属性网格搜索和学习曲线,这样它会打破我原来设置成训练和测试集,唐不共享用于计算学习曲线的任何标签。然后从这些训练集中,进一步将它们分成训练集和测试集,而不共享网格搜索标签?
本质上,我该如何运行一个嵌套的LabelKFold?
我,谁创造了赏金这一问题的用户,使用数据可从sklearn
写了下面的可重复的例子。
import numpy as np
from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import make_scorer, roc_auc_score
from sklearn.grid_search import GridSearchCV
from sklearn.cross_validation import cross_val_score, LabelKFold
digits = load_digits()
X = digits['data']
Y = digits['target']
Z = np.zeros_like(Y) ## this is just to make a 2-class problem, purely for the sake of an example
Z[np.where(Y>4)]=1
strata = [x % 13 for x in xrange(Y.size)] # define the strata for use in
## define stuff for nested cv...
mtry = [5, 10]
tuned_par = {'max_features': mtry}
toy_rf = RandomForestClassifier(n_estimators=10, max_depth=10, random_state=10,
class_weight="balanced")
roc_auc_scorer = make_scorer(roc_auc_score, needs_threshold=True)
## define outer k-fold label-aware cv
outer_cv = LabelKFold(labels=strata, n_folds=5)
#############################################################################
## this works: using regular randomly-allocated 10-fold CV in the inner folds
#############################################################################
vanilla_clf = GridSearchCV(estimator=toy_rf, param_grid=tuned_par, scoring=roc_auc_scorer,
cv=5, n_jobs=1)
vanilla_results = cross_val_score(vanilla_clf, X=X, y=Z, cv=outer_cv, n_jobs=1)
##########################################################################
## this does not work: attempting to use label-aware CV in the inner loop
##########################################################################
inner_cv = LabelKFold(labels=strata, n_folds=5)
nested_kfold_clf = GridSearchCV(estimator=toy_rf, param_grid=tuned_par, scoring=roc_auc_scorer,
cv=inner_cv, n_jobs=1)
nested_kfold_results = cross_val_score(nested_kfold_clf, X=X, y=Y, cv=outer_cv, n_jobs=1)
这几乎是我不得不做的,我自己做了kfold循环,并在单独的折叠上运行网格搜索。我是最初的提问者,但我不是那个在这个问题上付出恩典的人。我不确定这是如何工作的,但是我会积极回答这个问题,因为这是我所知道的最佳解决方案。但是,我会在接受答案之前等待赏金持有人的回复。 – Alex
这看起来非常可行 - 我会给这个旋转。看起来,这是做到这一点的正确方法。非常感谢你。 FWIW,@Alex,我相信只有我可以奖励赏金,所以geompalik可以在接下来的24小时内期待这一点。 – Sycorax