如何解决姓名匹配运行sparse_dot_topn函数会给我警告:内核重新启动了吗?
我正在尝试通过awesome_cossim_top使用余弦相似度将公司名称与政府的公司名称数据库匹配。因此,我将ngrams tf-idf转换为CSR矩阵,然后通过该函数运行它。它不会运行,并且会在每个IDE(Colab,Spyder,PyCharm和Jupyter)上重新启动内核。它根本不起作用。我想知道为什么吗?
import re
from ftfy import fix_text
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.neighbors import NearestNeighbors
import difflib
import numpy as np
from sparse_dot_topn import awesome_cossim_topn
from scipy.sparse import csr_matrix
import sparse_dot_topn.sparse_dot_topn as ct
def ngrams(string,n=3):
string = fix_text(string) # fix text encoding issues
string = string.encode("ascii",errors="ignore").decode() #remove non ascii chars
string = string.lower() #make lower case
chars_to_remove = [")","(",".","|","[","]","{","}","'"]
rx = '[' + re.escape(''.join(chars_to_remove)) + ']'
string = re.sub(rx,'',string) #remove the list of chars defined above
string = string.replace('&','and')
string = string.replace(',',' ')
string = string.replace('-',' ')
string = string.title() # normalise case - capital at start of each word
string = re.sub(' +',' ',string).strip() # get rid of multiple spaces and replace with a single space
string = ' '+ string +' ' # pad names for ngrams...
string = re.sub(r'[,-./]|\sBD',r'',string)
ngrams = zip(*[string[i:] for i in range(n)])
return [''.join(ngram) for ngram in ngrams]
def awesome_cossim_top(A,B,ntop,lower_bound=0):
# force A and B as a CSR matrix.
# If they have already been CSR,there is no overhead
A = A.tocsr()
B = B.tocsr()
M,_ = A.shape
_,N = B.shape
idx_dtype = np.int32
nnz_max = M * ntop
indptr = np.zeros(M + 1,dtype=idx_dtype)
indices = np.zeros(nnz_max,dtype=idx_dtype)
data = np.zeros(nnz_max,dtype=A.dtype)
ct.sparse_dot_topn(
M,N,np.asarray(A.indptr,dtype=idx_dtype),np.asarray(A.indices,A.data,np.asarray(B.indptr,np.asarray(B.indices,B.data,lower_bound,indptr,indices,data)
return csr_matrix((data,indptr),shape=(M,N))
def get_matches_df(sparse_matrix,A,top=100):
non_zeros = sparse_matrix.nonzero()
sparserows = non_zeros[0]
sparsecols = non_zeros[1]
if top:
nr_matches = top
else:
nr_matches = sparsecols.size
left_side = np.empty([nr_matches],dtype=object)
right_side = np.empty([nr_matches],dtype=object)
similairity = np.zeros(nr_matches)
for index in range(0,nr_matches):
left_side[index] = A[sparserows[index]]
right_side[index] = B[sparsecols[index]]
similairity[index] = sparse_matrix.data[index]
return pd.DataFrame({'left_side': left_side,'right_side': right_side,'similairity': similairity})
govdata = pd.read_csv('companydata2018.csv',encoding='utf-8')
hypxdata = pd.read_csv('enerygycomp.csv',encoding='cp1252')
#X = gov Y = hypx
vectoriser = TfidfVectorizer(analyzer=ngrams)
tfidfgov = vectoriser.fit_transform(govdata['CompanyName'])
tfidfhypx = vectoriser.fit_transform(hypxdata['Name'])
matches = awesome_cossim_top(tfidfgov,tfidfhypx.transpose(),1,0)```
解决方法
我猜你的内存不足。您是否尝试过使用较小的数据集?
此外,我认为您应该分别执行拟合和转换步骤:将向量化器与两个系列拟合(例如将它们连接起来),然后通过变换获取两个数据集的 tfidf 矩阵。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。