Custom transformer for sklearn Pipeline that alters both X and y

Modifying the sample axis, e.g. removing samples, does not (yet?) comply with the scikit-learn transformer API. So if you need to do this, you should do it outside any calls to scikit learn, as preprocessing.

As it is now, the transformer API is used to transform the features of a given sample into something new. This can implicitly contain information from other samples, but samples are never deleted.

Another option is to attempt to impute the missing values. But again, if you need to delete samples, treat it as preprocessing before using scikit learn.

@eickenberg is the proper and clean answer. Nevertheless, I like to keep everything into one Pipeline, so if you are interested, I created a library (not yet deployed on pypi) that allow to apply transformation on Y:

https://gitlab.com/thibaultB/transformers/

Usage is the following:

df = pd.DataFrame([[0, 1, 2], [3, 4, 5], [6, 7, 8]])df.columns = ["a", "b", "target"]spliter = SplitXY("target") # Create a new step and give it name of column targetpipe = Pipeline([        ("imputer", SklearnPandasWrapper(KNNImputer())),        ("spliter", spliter),         ("scaler", StandardScaler()),        ("rf",            EstimatorWithoutYWrapper(RandomForestRegressor(random_state=45),                                    spliter)) # EstimatorWithoutYWrapper overwrite RandomForestRegressor to get y from spliter just before calling fit or transform    ])pipe.fit(df)res = pipe.predict(df)

Using this code, you can alter the number of rows if you put all the transformer that modify the numbers of rows before the "SplitXY" transformer. Transformer before the SplitXY transformer should keep columns name, it is why I also added a SklearnPandasWrapper that wrap sklearn transformer (that usually return numpy array) to keep columns name.

Use "deep-copies" further on, down the pipeline and X, y remain protected

.fit() can first assign on each call deep-copy to new class-variables

self.X_without_NaNs = X.copy()self.y_without_NaNs = y.copy()

and then reduce / transform these not to have more NaN-s than ordered by self.treshold