如何解决将熊猫数据框转换为固定大小的分段数组
我正在努力将数据框转换为固定大小的段数组，并将其馈送到卷积神经网络。具体来说，我想从df到m数组的列表，每个数组包含大小为(1,5,4)的段。所以最后，我将得到一个(m,1,4)数组。
为澄清我的问题，我将使用此MWE进行说明。假设这是我的df：
df = {
'id': [1,1],'speed': [17.63,17.63,0.17,1.41,0.61,0.32,0.18,0.43,0.30,0.46,0.75,0.37],'acc': [0.00,-0.09,1.24,-0.80,-0.29,-0.14,0.25,-0.13,0.16,0.29,-0.38,0.27],'jerk': [0.00,0.01,-2.04,0.51,0.15,0.39,0.13,-0.67,0.65,0.52],'bearing': [29.03,56.12,18.49,11.85,36.75,27.52,81.08,51.06,19.85,10.76,14.51,24.27],'label' : [3,3,3] }
df = pd.DataFrame.from_dict(df)
为此，我使用以下功能：
def df_transformer(dataframe,chunk_size=5):
grouped = dataframe.groupby('id')
# initialize accumulators
X,y = np.zeros([0,chunk_size,4]),np.zeros([0,])
# loop over segments (id)
for _,group in grouped:
inputs = group.loc[:,'speed':'bearing'].values
label = group.loc[:,'label'].values[0]
# calculate number of splits
N = len(inputs) // chunk_size
if N > 0:
inputs = np.array_split(inputs,[chunk_size]*N)
else:
inputs = [inputs]
# loop over splits
for inpt in inputs:
inpt = np.pad(
inpt,[(0,chunk_size-len(inpt)),(0,0)],mode='constant')
# add each inputs split to accumulators
X = np.concatenate([X,inpt[np.newaxis,np.newaxis]],axis=0)
y = np.concatenate([y,label[np.newaxis]],axis=0)
return X,y
上面的df有12行，因此如果正确转换为预期的形式，我应该得到形状为(3,4)的数组。在上述函数中，对少于5行的段进行零填充，以使段的形状为(1,4)。
当前，此功能有两个问题：
- 该功能仅对我的df中少于10行有效。
这样（最后一行应在下面填充零）：
X,y = df_transformer(df[:9])
X
array([[[[ 1.763e+01,0.000e+00,2.903e+01],[ 1.763e+01,-9.000e-02,1.000e-02,5.612e+01],[ 1.700e-01,1.240e+00,-2.040e+00,1.849e+01],[ 1.410e+00,-8.000e-01,5.100e-01,1.185e+01],[ 6.100e-01,-2.900e-01,1.500e-01,3.675e+01]]],[[[ 3.200e-01,-1.400e-01,3.900e-01,2.752e+01],[ 1.800e-01,2.500e-01,-3.800e-01,8.108e+01],[ 4.300e-01,-1.300e-01,2.900e-01,5.106e+01],[ 3.000e-01,1.600e-01,1.300e-01,1.985e+01],[ 0.000e+00,0.000e+00]]]])
但是在这种情况下引入了全零数组（段）：
X,y = df_transformer(df[:10])
X
array([[[[ 1.763e+01,[[[ 0.000e+00,0.000e+00],0.000e+00]]],[ 4.600e-01,-6.700e-01,1.076e+01]]]])
- 如果我传递整个
df，该函数将失败（我不理解该错误，但似乎与少于5行的段的填充有关。）
因此，在这种情况下，我收到index can't contain negative values错误：
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-1fc559db37eb> in <module>()
----> 1 X,y = df_transformer(df)
2 frames
<ipython-input-4-9e1c49985863> in df_transformer(dataframe,chunk_size)
24 inpt = np.pad(
25 inpt,---> 26 mode='constant')
27 # add each inputs split to accumulators
28 X = np.concatenate([X,axis=0)
<__array_function__ internals> in pad(*args,**kwargs)
/usr/local/lib/python3.6/dist-packages/numpy/lib/arraypad.py in pad(array,pad_width,mode,**kwargs)
746
747 # Broadcast to shape (array.ndim,2)
--> 748 pad_width = _as_pairs(pad_width,array.ndim,as_index=True)
749
750 if callable(mode):
/usr/local/lib/python3.6/dist-packages/numpy/lib/arraypad.py in _as_pairs(x,ndim,as_index)
517
518 if as_index and x.min() < 0:
--> 519 raise ValueError("index can't contain negative values")
520
521 # Converting the array with `tolist` seems to improve performance
ValueError: index can't contain negative values
预期输出：
X,y = df_transformer(df)
X
array([[[[ 1.763e+01,1.076e+01]]],[[[ 7.500e-01,6.500e-01,1.451e+01],[ 3.700e-01,2.700e-01,5.200e-01,2.427e+01],0.000e+00]]]])
有人可以帮我解决这个问题吗？上面的WME可以很好地重现此错误。
编辑
RichieV的答案也有一个错误。尽管它在给定的MWE中有效，但在以下情况下却无法完成正确的任务（将df两次扩展
its size):
df = {
'id': [1]*12+[2]*12,0.37]*2,0.27]*2,0.52]*2,24.27]*2,3]*2 }
df = pd.DataFrame.from_dict(df)
X,y = df_transformer(df,chunk_size=5)
print(X[:3])
[[[[ 1.763e+01 0.000e+00 0.000e+00 2.903e+01]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]
[ 3.700e-01 2.700e-01 5.200e-01 2.427e+01]]]
[[[ 7.500e-01 -3.800e-01 6.500e-01 1.451e+01]
[ 3.000e-01 1.600e-01 1.300e-01 1.985e+01]
[ 4.600e-01 2.900e-01 -6.700e-01 1.076e+01]
[ 1.800e-01 2.500e-01 -3.800e-01 8.108e+01]
[ 3.200e-01 -1.400e-01 3.900e-01 2.752e+01]]]
[[[ 6.100e-01 -2.900e-01 1.500e-01 3.675e+01]
[ 1.410e+00 -8.000e-01 5.100e-01 1.185e+01]
[ 1.700e-01 1.240e+00 -2.040e+00 1.849e+01]
[ 1.763e+01 -9.000e-02 1.000e-02 5.612e+01]
[ 4.300e-01 -1.300e-01 2.900e-01 5.106e+01]]]]
请注意，第一个元素与答案中的元素不同（在第二，第三和第四行中得到全零。
解决方法
您可以填充df一次，而不必在每次迭代中填充。
使用第二个ID获取该数据
df = {
'id': [1,1,2,2],'speed': [17.63,17.63,0.17,1.41,0.61,0.32,0.18,0.43,0.30,0.46,0.75,0.37],'acc': [0.00,-0.09,1.24,-0.80,-0.29,-0.14,0.25,-0.13,0.16,0.29,-0.38,0.27],'jerk': [0.00,0.01,-2.04,0.51,0.15,0.39,0.13,-0.67,0.65,0.52],'bearing': [29.03,56.12,18.49,11.85,36.75,27.52,81.08,51.06,19.85,10.76,14.51,24.27],'label' : [3,3,3] }
df = pd.DataFrame.from_dict(df)
print(df)
id speed acc jerk bearing label
0 1 17.63 0.00 0.00 29.03 3
1 1 17.63 -0.09 0.01 56.12 3
2 1 0.17 1.24 -2.04 18.49 3
3 1 1.41 -0.80 0.51 11.85 3
4 1 0.61 -0.29 0.15 36.75 3
5 1 0.32 -0.14 0.39 27.52 3
6 1 0.18 0.25 -0.38 81.08 3
7 1 0.43 -0.13 0.29 51.06 3
8 1 0.30 0.16 0.13 19.85 3
9 2 0.46 0.29 -0.67 10.76 3
10 2 0.75 -0.38 0.65 14.51 3
11 2 0.37 0.27 0.52 24.27 3
和代码
def df_transformer(df,chunk_size=5):
### pad df with 0's so len(df) is exactly a multiple of chunk_size
df = pd.concat([df,pd.DataFrame([[id] + [0] * 5 # add row with zeros
for id,ct in df.groupby('id').size().iteritems() # for each id
for row in range(chunk_size - ct % chunk_size)] # as many times as needed,columns=df.columns)
]).sort_values('id',kind='mergesort',ignore_index=True)
# print(df)
X,y = [],[]
for _,group in df.groupby(df.index//5):
X.append(group.iloc[:,1:-1].values[np.newaxis,...])
y.append(group.iloc[0,-1]) # not sure how you want y to be structured
return np.array(X),np.array(y)
X,y = df_transformer(df,chunk_size=5)
print(X)
输出
[[[[ 1.763e+01 0.000e+00 0.000e+00 2.903e+01]
[ 1.763e+01 -9.000e-02 1.000e-02 5.612e+01]
[ 1.700e-01 1.240e+00 -2.040e+00 1.849e+01]
[ 1.410e+00 -8.000e-01 5.100e-01 1.185e+01]
[ 6.100e-01 -2.900e-01 1.500e-01 3.675e+01]]]
[[[ 3.200e-01 -1.400e-01 3.900e-01 2.752e+01]
[ 1.800e-01 2.500e-01 -3.800e-01 8.108e+01]
[ 4.300e-01 -1.300e-01 2.900e-01 5.106e+01]
[ 3.000e-01 1.600e-01 1.300e-01 1.985e+01]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]]]
[[[ 4.600e-01 2.900e-01 -6.700e-01 1.076e+01]
[ 7.500e-01 -3.800e-01 6.500e-01 1.451e+01]
[ 3.700e-01 2.700e-01 5.200e-01 2.427e+01]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]
[ 0.000e+00 0.000e+00 0.000e+00 0.000e+00]]]]
请注意前两个部分来自id==1，最后一个来自id==2，每个部分都有自己的零填充
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。