如何解决为 sklearn 管道实现自定义单热编码功能
与 One Hot Encoding preserve the NAs for imputation 中发布的问题相关,我正在尝试创建一个自定义函数,用于在对分类变量进行热编码时处理 NA。该设置应适用于使用 sklearn pipeline
的训练/测试拆分和建模。
我的问题的一个简单的可重现示例:
#Packages
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.impute import KNNImputer
from sklearn.base import BaseEstimator,TransformerMixin
from sklearn.linear_model import Ridge
from sklearn.impute import SimpleImputer
# Make some categorical data X and a response y and split it.
X = pd.DataFrame(columns=["1","2"],data = [["A",np.nan],["B","A"],[np.nan,"B"],["A",["C",["D","E"]])
y = pd.DataFrame(data = np.array([1,5,4,6,2,3,9,9]))
X_train,X_test,Y_train,Y_test = train_test_split(X,y,test_size=0.2,random_state=42)
然后我创建了一个使用 nan 执行 OHE 的自定义函数(使用 Cyclical Loop Between OneHotEncoder and KNNImpute in Scikit-learn 中描述的过程)
class OHE_with_nan(BaseEstimator,TransformerMixin):
""" OHE with NAN. Not super pretty but works..
"""
def __init__(self,copy=True):
self.copy = copy
def fit(self,X,y = None):
""" This transformer does not use a fit procedure """
return self
def transform(self,y = None):
""" Return the new object here"""
# Replace nans with "Missing" such that OneHotEncoder can work.
enc_missing = SimpleImputer(strategy="constant",fill_value="missing")
data1 = pd.DataFrame(columns=X.columns,data = enc_missing.fit_transform(X))
#Perform standard OHE
OHE = OneHotEncoder(sparse=False,handle_unknown="ignore")
OHE_fit = OHE.fit_transform(data1)
#save feature names of the OHE dataframe
data_OHE = pd.DataFrame(columns=OHE.get_feature_names(data1.columns),data = OHE_fit)
# Initialize
Column_names = data1.columns
Final_OHE = pd.DataFrame()
# Loop over columns to replace 0s with nan the correct places.
for i in range(len(data1.columns)):
tmp_data = data_OHE[data_OHE.columns[pd.Series(data_OHE.columns).str.startswith(Column_names[i])]]
missing_name = tmp_data.iloc[:,-1:].columns
missing_index = np.where(tmp_data[missing_name]==1)[0]
tmp_data.loc[missing_index,:] = np.nan
tmp_data1 = tmp_data.drop(missing_name,axis=1)
Final_OHE = pd.concat([Final_OHE,tmp_data1],axis=1)
return Final_OHE
然后将其组合成一个使用岭回归预测 y 的管道(模型的随机选择,仅用于示例..)
Estimator = Pipeline([
('Ohe_with_NA',OHE_with_nan()),("Imputer",KNNImputer(n_neighbors=1)),('Model',Ridge(alpha = 0.01))
])
可以拟合的程序:
pipe_fit = Estimator.fit(X_train,Y_train)
但是对看不见的数据的测试失败了:
pipe_fit.score(X_test,Y_test)
ValueError: X has 2 features,but KNNImputer is expecting 7 features as input.
这是因为 handle_unknown = "ignore
中的 OneHotEncoder
中的 OHE_with_nan
不再“活动”,因为它已被包装到我的自定义函数中。
如果只是在管道中直接使用 OneHotEncoder(handle_unknown = "ignore")
,一切正常(但这不是我的意图,因为这会从我尝试估算的数据中“删除”nans。)
我的问题
我如何在我的自定义函数中启用 handle_unknown = "ignore"
,以便它也可以在管道设置中对看不见的数据执行?
希望您了解我的情况 - 非常感谢您的帮助!
解决方法
我认为主要问题是您需要在合适的时候保存更多信息(尤其是内部 OneHotEncoder
)。我还使缺失列识别更加强大(我想也许您依赖于将其放在最后的排序,但由于字母顺序,这仅适用于您的样本数据?)。我没有花太多时间清理东西或寻求效率。
class OHE_with_nan(BaseEstimator,TransformerMixin):
"""One-hot encode,propagating NaNs.
Requires a dataframe as input!
"""
def fit(self,X,y=None):
self.orig_cols_ = X.columns
self.imputer_ = SimpleImputer(strategy="constant",fill_value="MISSING")
X_filled = self.imputer_.fit_transform(X)
self.ohe_ = OneHotEncoder(sparse=False,handle_unknown="ignore")
self.ohe_.fit(X_filled)
self.ohe_colnames_ = self.ohe_.get_feature_names(X.columns)
self.missing_value_columns = np.array(["MISSING" in col for col in self.ohe_colnames_])
return self
def transform(self,y=None):
raw_ohe = pd.DataFrame(self.ohe_.transform(self.imputer_.transform(X)),columns=self.ohe_colnames_)
out_list = []
# Loop over columns to replace 0s with nan the correct places.
for orig_col in self.orig_cols_:
tmp_data = raw_ohe[self.ohe_colnames_[pd.Series(self.ohe_colnames_).str.startswith(orig_col)]]
missing_name = tmp_data.columns[["MISSING" in col for col in tmp_data.columns]]
missing_indices = np.where(tmp_data[missing_name]==1)[0]
tmp_data.loc[missing_indices,:] = np.nan
tmp_data1 = tmp_data.drop(missing_name,axis=1)
out_list.append(tmp_data1)
out = pd.concat(out_list,axis=1)
return out
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。