Cross Validated
2022-08-09 16:42 UTC
By thesecond
AI-113-20220809-social-media-3387c8c0
How to properly impute values on the test set using imputer (missForest)
I'm trying to impute some missing values on my dataset $X$ . So first I shuffle and split data to obatin the train set X_train and the test set X_test . X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, shuffle = True, test_size = 0.25) Then I am imputing values using missForest algorithm, but only in the train set imputer = MissForest(max_iter = 10, verbose = 0) X_train_imp = imputer.fit_transform(X_train) Now, what's the proper way of imputing the test set? Use transform() function as follows X_test_imp = imputer.transform(X_test) OR Combine imputed train set and test set and only then impute values in the test set X_combined = pd.concat([X_train_imp, Xtest]) imputer_conc = MissForest(max_iter = 10, verbose = 0) X_conc_imp = imputer_conc.fit_transform(X_combined) X_conc_imp = pd.DataFrame(Xconc_imp) Xtrain_imp = Xconc_imp.loc[:X_train_imp.shape[0] - 1] Xtest_imp = Xconc_imp.loc[X_train_imp.shape[0]:] The first approach seems to allow no data leakage, however the second one takes advantage of the available test set (I simply have it) - but I'm curious to what degree it could cause data leakage.
I'm trying to impute some missing values on my dataset $X$ . So first I shuffle and split data to obatin the train set X_train and the test set X_test . X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, shuffle = True, test_size = 0.25) Then I am imputing values using missForest algorithm, but only in the train set imputer = MissForest(max_iter = 10, verbose = 0) X_train_imp = imputer.fit_transform(X_train) Now, what's the proper way of imputing the test set? Use transform() function as follows X_test_imp = imputer.transform(X_test) OR Combine imputed train set and test set and only then impute values in the test set X_combined = pd.concat([X_train_imp, Xtest]) imputer_conc = MissForest(max_iter = 10, verbose = 0) X_conc_imp = imputer_conc.fit_transform(X_combined) X_conc_imp = pd.DataFrame(Xconc_imp) Xtrain_imp = Xconc_imp.loc[:X_train_imp.shape[0] - 1] Xtest_imp = Xconc_imp.loc[X_train_imp.shape[0]:] The first approach seems to allow no data leakage, however the second one takes advantage of the available test set (I simply have it) - but I'm curious to what degree it could cause data leakage.
Full article content could not be extracted automatically. Read the original below.
Source:
Cross Validated
· stats.stackexchange.com