如何解决BigQuery的Python客户端不应该与多处理一起使用吗?
根据Python BigQuery client documentation,看来多处理应该可以工作。 但是,在尝试使用多处理从熊猫数据帧向BigQuery表进行简单加载时,我一直遇到错误,我想知道文档中的以下语句是否与此有关。
在多处理方案中,最佳实践是在
multiprocessing.Pool
或multiprocessing.Process
调用os.fork()
之后创建客户端实例。
我基于此GCP doc(google-cloud-bigquery)编写了代码,该代码仅尝试创建2个进程以在同一张表上加载两个不同的pandas数据框(我也尝试在两个不同的表并出现相同的错误):
from google.cloud import bigquery
import pandas as pd
import numpy as np
import multiprocessing
import random
def trying():
#This is just to create random values to create the dataframe.
values = np.round(np.random.uniform(0,1,(6,14)),2)* random.uniform(0,1)
df = pd.DataFrame(values,columns=list('abcdefghifklmn'))
#I understand that I have to create the client for each process
client = bigquery.Client()
table_id = 'mydataset.new_table'
job = client.load_table_from_dataframe(df,table_id)
job.result()
if __name__ == '__main__':
processes = []
for i in range(2):
p = multiprocessing.Process(target=trying)
p.start()
processes.append(p)
for process in processes:
process.join()
这是我得到的错误,我不知道发生了什么。两个进程都抛出相同的错误:
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py",line 258,in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py",line 114,in run
self._target(*self._args,**self._kwargs)
File "test_multiproc_load.py",line 19,in trying
job = client.load_table_from_dataframe(df,table_id)
File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py",line 1932,in load_table_from_dataframe
parquet_compression=parquet_compression,File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 485,in dataframe_to_parquet
arrow_table = dataframe_to_arrow(dataframe,bq_schema)
File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 449,in dataframe_to_arrow
bq_to_arrow_array(get_column_or_index(dataframe,bq_field.name),bq_field)
File "/home/.../snow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 224,in bq_to_arrow_array
return pyarrow.Array.from_pandas(series,type=arrow_type)
File "pyarrow/array.pxi",line 755,in pyarrow.lib.Array.from_pandas
return array(obj,mask=mask,type=type,safe=safe,from_pandas=True,File "pyarrow/array.pxi",line 269,in pyarrow.lib.array
return _sequence_to_array(obj,mask,size,type,pool,c_from_pandas)
File "pyarrow/array.pxi",line 38,in pyarrow.lib._sequence_to_array
check_status(ConvertPySequence(sequence,options,&out))
File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/frame.py",line 2927,in __getitem__
indexer = self.columns.get_loc(key)
File "/home/.../snow/local/lib/python2.7/site-packages/pandas/core/indexes/base.py",line 2659,in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx",line 108,in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx",line 127,line 147,in pandas._libs.index.IndexEngine._get_loc_duplicates
KeyError: 0
这是我的数据框创建过程中的简单索引错误吗? 你有什么建议吗?
解决方法
问题似乎出在columns=list('abcdefghifklmn')
上。
我不确定为什么它不起作用,但是如果我这样指定columns=['a','b',...]
,它就会起作用。
如果有人可以解释原因,那将是很棒的。
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。