如何在python中提高计算速度？

如何解决如何在python中提高计算速度？

我正在建立一个计算，以将新列添加到数据框。这是我的数据： 1

我需要创建一个新列“ mob”。 “暴民”的计算是

如果某行的“ LoanID”与前一行相同。例如，如果借贷['LoanId'] [0] =借贷['LoanId'] ReactJS Website;
如果“暴民”的前一行是> 0；如果是这样，那么此行的“ mob”值将与前一行的值加1；如果不是，请尝试如果该行的loan ['repay_lbl']为1或2，如果是，则该行的“暴民”值为1；

我的代码如下：

for i in range(1,len(loan['LoanId'])):
if loan['LoanId'][i-1] == loan['LoanId'][i]:
    if loan['mob'][i-1] > 0:
        loan['mob'][i] = loan['mob'][i-1] +1 
    elif loan['repay_lbl'][i] == 1 or loan['repay_lbl'][i] == 2:
        loan['mob'][i] = 1

代码将花费O（n）。有什么方法可以改善算法并加快速度吗？我只是Python的初学者。非常感谢您的帮助。

解决方法

由于每行mob列的值取决于上一行的值，因此它取决于所有先前的行。这意味着您不能并行运行它，而您基本上陷于O(n)中。

因此，我认为numpy数组操作在这里不会有太大用处。

否则，通常会有一些技巧来加快Python代码的速度；

我不确定前两个是否适用于numpy / pandas。在这种情况下，您可能必须对数据使用常规的Python列表。

当然，在深入研究其中任何一项之前，您应该考虑自己的数据集是否足够大以保证需要付出努力。

通过更改循环方式来缩短时间

改善循环时间的依据

遍历所有N行而不进行广播，因此复杂度为O（N）
虽然都是N阶，但是不同的循环方法具有不同的复杂度缩放因子
不同的比例因子使某些方法比其他方法快得多

受-Different ways to iterate over rows in a Pandas Dataframe — performance comparison

的启发

方法

For循环-原始帖子
iterrows
itertuples
zip

摘要

对于10万行，zip方法比for循环（即OP方法）快93倍

测试代码

import pandas as pd
import numpy as np
from random import randint

def create_input(N):
    ' Creates a loan DataFrame with N rows '
    LoanId = [randint(0,N //4) for _ in range(N)]  # though random,N//4 ensures
                                                    # high likelihood some rows repeat
                                                    # LoanID
    repay_lbl = [randint(0,2) for _ in range(N)]

    data = {'LoanId':LoanId,'repay_lbl': repay_lbl,'mob':[0]*N}
    return pd.DataFrame(data)

def m_itertuples(loan):
    ' Iterating using itertuples,set single values using at '
    loan = loan.copy()  # copy since timing calls function multiple time
                        # so don't want to modify input
                        # not necessary in general
    prev_loanID,prev_mob = None,None
    for index,row in enumerate(loan.itertuples()): # iterate over rows with iterrows()
        if prev_loanID is not None:
             if prev_loanID == row.LoanId:
                if prev_mob > 0:
                    loan.at[row.Index,'mob'] = prev_mob + 1 
                elif row.repay_lbl == 1 or row.repay_lbl == 2:
                    loan.at[row.Index,'mob'] = 1
            
        # Query for latest values   
        prev_loanID,prev_mob = loan.at[index,'LoanId'],loan.at[index,'mob']
                    
    return loan
    
def m_for_loop(loan):
    ' For loop over the data frame '
    loan = loan.copy()  # copy since timing calls function multiple time
                        # so don't want to modify input
                        # not necessary in general
            
    for i in range(1,len(loan['LoanId'])):
        if loan['LoanId'][i-1] == loan['LoanId'][i]:
            if loan['mob'][i-1] > 0:
                loan['mob'][i] = loan['mob'][i-1] +1 
            elif loan['repay_lbl'][i] == 1 or loan['repay_lbl'][i] == 2:
                loan['mob'][i] = 1
    return loan

def m_iterrows(loan):
    ' Iterating using iterrows,row in loan.iterrows(): # iterate over rows with iterrows()
        if prev_loanID is not None:
             if prev_loanID == row['LoanId']:
                if prev_mob > 0:
                    loan.at[index,'mob'] = prev_mob + 1 
                elif row['repay_lbl'] == 1 or row['repay_lbl'] == 2:
                    loan.at[index,'mob'] = 1
                    
        # Query for latest values          
        prev_loanID,'mob']
        
    return loan

def m_zip(loan):
    ' Iterating using zip,prev_mob  = None,(loanID,mob,repay_lbl) in enumerate(zip(loan['LoanId'],loan['mob'],loan['repay_lbl'])):
        if prev_loanID is not None:
             if prev_loanID == loanID:
                if prev_mob > 0:
                    mob = loan.at[index,'mob'] = prev_mob + 1
                elif repay_lbl == 1 or repay_lbl == 2:
                    mob = loan.at[index,'mob'] = 1
        
        # Update to latest values
        prev_loanID,prev_mob = loanID,mob
        
    return loan

注意：迭代器代码查询数据帧以获取更新的数据，而不是从迭代器中获取warning：

您永远不要修改要迭代的内容。这不是保证在所有情况下都能正常工作。根据数据类型，迭代器返回一个副本而不是一个视图，对其进行写入将没有效果。

还使用assert df1.equals(df2)比较了DataFrame，以验证不同方法产生的结果相同

时间代码

使用benchit

inputs = [create_input(i) for i in 10**np.arange(6)]  # 1 to 10^5 rows
funcs = [m_for_loop,m_iterrows,m_itertuples,m_zip]

t = benchit.timings(funcs,inputs)

结果

运行时间以秒为单位

Functions  m_for_loop  m_iterrows  m_itertuples     m_zip
Len                                                      
1            0.000217    0.000493      0.000781  0.000327
10           0.001070    0.002002      0.001008  0.000353
100          0.007100    0.016501      0.003062  0.000498
1000         0.056940    0.162423      0.021396  0.001057
10000        0.565809    1.625043      0.210858  0.006938
100000       5.890920   16.658842      2.179602  0.062953

如何在python中提高计算速度？

如何解决如何在python中提高计算速度？

解决方法

相关推荐