如何解决使用joblib.Parallel并行化相互信息的计算
我正在尝试计算两个数组之间的经验互信息。因此,我正在使用normalized_mutual_info_score
(sklearn doc)中的函数sklearn
。我使用numba
尝试了各种方法来加速cuda的计算,但是这对我不起作用...
因此,我选择尝试joblib
。通过以下代码,我试图并行化相互信息的计算:
from sklearn.metrics.cluster import normalized_mutual_info_score
def processInput(i):
mi = []
for j in range(i + 1):
mi.append(normalized_mutual_info_score(test_matrix[:,i],test_matrix[:,j]))
return mi
def get_mutual_information_sklearn(data):
mutual_inf = np.zeros((data.shape[1],data.shape[1]))
for i in numba.prange(data.shape[1]):
print('{}/{}'.format(i,data.shape[1]),end = '\r')
for j in numba.prange(i + 1):
mutual_inf[j,i] += normalized_mutual_info_score(data[:,data[:,j])
i_lower = np.tril_indices(data.shape[1],-1)
mutual_inf[i_lower] = mutual_inf.T[i_lower]
return mutual_inf
test_matrix = np.random.rand(100,500)
num_cores = multiprocessing.cpu_count()
now = time.time()
a = get_mutual_information_sklearn(test_matrix)
print("non parallel execution took : ",time.time() - now,"seconds")
now = time.time()
results = Parallel(n_jobs=1)(delayed(processInput)(i) for i in range(test_matrix.shape[1]))
print("non parallel execution with 1 n_jobs took : ","seconds")
now = time.time()
results = Parallel(n_jobs=2)(delayed(processInput)(p) for p in range(test_matrix.shape[1]))
print("parallel execution with 2 n_jobs took : ","seconds")
输出如下:
non parallel execution took : 148.19334053993225 seconds
non parallel execution with 1 n_jobs took : 146.42843222618103 seconds
parallel execution with 2 n_jobs took : 989.2411859035492 seconds
我不确定我是否做得正确。但是我很高兴能得到一些输入,以改善我的计算的并行性。变量num_cores = 14
。非常感谢!
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。