即使使用 csr_matrix，如何使用 BinaryRelevance 修复 ArrayMemoryError？

如何解决即使使用 csr_matrix，如何使用 BinaryRelevance 修复 ArrayMemoryError？

我正在尝试使用 Toxic Comment data from kaggle 预测有害评论：

import skmultilearn,sys
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix,issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
y = csr_matrix(data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']])
binary_rel_clf = BinaryRelevance(MultinomialNB())
binary_rel_clf.fit(Xfeatures,y)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [y.columns.values[prediction].tolist() for prediction in br_prediction]
print(predictions)

但是，我收到此错误：

Traceback (most recent call last):
  File "...\multi_label_toxic.py",line 15,in <module>
    binary_rel_clf.fit(Xfeatures,y)
  File "...\problem_transform\br.py",line 161,in fit
    classifier.fit(self._ensure_input_format(
  File "...\base\base.py",line 86,in _ensure_input_format
    return X.toarray()
  File "...\scipy\sparse\compressed.py",line 1031,in toarray
    out = self._process_toarray_args(order,out)
  File "...\scipy\sparse\base.py",line 1202,in _process_toarray_args
    return np.zeros(self.shape,dtype=self.dtype,order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 226. GiB for an array with shape (159571,189775) and data type float64

即使尝试传递参数“require_dense=False”，我还是遇到了另一个错误：

Traceback (most recent call last):
  File "...\multi_label_toxic.py",y)
  File "...\skmultilearn\problem_transform\br.py",in fit
    classifier.fit(self._ensure_input_format(
  File "...\sklearn\naive_bayes.py",line 612,in fit
    X,y = self._check_X_y(X,y)
  File "...\sklearn\naive_bayes.py",line 477,in _check_X_y
    return self._validate_data(X,y,accept_sparse='csr')
  File "...\sklearn\base.py",line 433,in _validate_data
    X,y = check_X_y(X,**check_params)
  File "...\sklearn\utils\validation.py",line 63,in inner_f
    return f(*args,**kwargs)
  File "...\sklearn\utils\validation.py",line 826,in check_X_y
    y = column_or_1d(y,warn=True)
  File "...\sklearn\utils\validation.py",line 864,in column_or_1d
    raise ValueError(
ValueError: y should be a 1d array,got an array of shape () instead.

如何解决这个问题并使用整个模型进行训练？

解决方法

您似乎错误地指定了 required_dense 参数。您需要 required_dense=[False,True] 才能以稀疏格式指定 X 值而不是 y 值。在倒数第二行（预测 = ...）中，您需要在将其转换为矩阵之前使用 y，以便您可以访问列名。 以下代码应该可以工作。

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix,issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance
import numpy as np

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
cats = data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']]
y = csr_matrix(cats)
binary_rel_clf = BinaryRelevance(MultinomialNB(),require_dense = [False,True])
binary_rel_clf.fit(Xfeatures,y) # y[:,0].toarray().reshape(-1)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [cats.columns[prediction].tolist() for prediction in br_prediction]
print(predictions)

输出：

[['toxic','insult']]