python3 subprocess.Popen和线程的有线行为

如何解决python3 subprocess.Popen和线程的有线行为

我想编写一个Python脚本来使用线程和subprocess.Popen并行ping某些服务器。在subprocess.Popen中，我对ping服务器调用了“ / bin / ping”程序。对服务器执行ping操作后，我将延迟结果存储到队列中。我有16台服务器，但是每次启动多线程脚本时，结果都少于16。某些ping返回结果尚未存储到队列，某些线程已完成，而某些ping返回结果存储到队列。以下是我的脚本。

from __future__ import print_function

import platform
import subprocess
import re
import threading
from pprint import pprint
from queue import Queue

vultr_domain = {
                "(Asia)Seoul,Korea[韩国 首尔]"     :   "sel-kor-ping.vultr.com","(Asia)Tokyo,Japan[日本 东京]"     :   "hnd-jp-ping.vultr.com","Singapore[新加坡]"                 :   "sgp-ping.vultr.com","(AU) Sydney,Australia[悉尼]"      :   "syd-au-ping.vultr.com","(EU) Frankfurt,DE[德国 法兰克福]" :   "fra-de-ping.vultr.com","(EU) Amsterdam,NL[荷兰 阿姆斯特丹]"   :   "ams-nl-ping.vultr.com","(EU) London,UK[英国 伦敦]"            :   "lon-gb-ping.vultr.com","(EU) Paris,France[法国 巴黎]"         :   "par-fr-ping.vultr.com","Seattle,Washington[美东 西雅图]"      :   "wa-us-ping.vultr.com","Silicon Valley,Ca[美西 加州 硅谷]"    :   "sjo-ca-us-ping.vultr.com","Los Angeles,Ca[美西 加州 洛杉矶]"     :   "lax-ca-us-ping.vultr.com","Chicago,Illinois[美东 芝加哥]"        :   "il-us-ping.vultr.com","Dallas,Texas[美中 德克萨斯州 达拉斯]" :   "tx-us-ping.vultr.com","New York / New Jersey[美东 新泽西]"    :   "nj-us-ping.vultr.com","Atlanta,Georgiaa[美东 亚特兰大]"      :   "ga-us-ping.vultr.com","Miami,Florida[美东 迈阿密]"           :   "fl-us-ping.vultr.com"
}

# 存放结果
q = Queue()

def linux_call_ping(server,ipaddr = "fl-us-ping.vultr.com"):

    ping = subprocess.Popen(["/bin/ping","-c","2",ipaddr],stdout = subprocess.PIPE,stderr = subprocess.PIPE).communicate()

    if ping[0]:
        # ping成功，但不一定能通
        #print(ping[0].decode())
        packet_loss = "(\d+)% packet loss"
        match_receive = re.search(packet_loss,ping[0].decode())
        packet_loss_rate = match_receive.groups()[0]

        if packet_loss_rate != '100':
            rtt = "rtt min/avg/max/mdev = (.*?) ms"
            rtt_rec = re.search(rtt,ping[0].decode())
            result = rtt_rec.groups()[0].split('/')[:3]
            result.append(packet_loss_rate)
        else:
            result = [999,999,100]

    else:
        # ping失败
        print(ping[1].decode())
        result = [999,100]

    q.put((server,float(result[1])))
    print("[+] {} ping stoped.".format(server))


def linux_ping_thr(q):
    while not q.empty():
        server,domain = q.get()
        try:
            print("[+] ",server," ping started.")
            linux_call_ping(server,domain)
            
        except:
            print('[*]'," ping failed.")

def main():

    dq = Queue()

    server_latency = {}

    # 单线程
    # for server,domain in vultr_domain.items():
    #     
    #     result = linux_call_ping(domain)
    #     print(server,'\tmin: ',result[0],'\tavg: ',result[1],'\tmax: ',result[2],'\tloss rate: ',result[3])
    #     server_latency[server] = float(result[1])
    
    # 多线程
    for server,domain in vultr_domain.items():
        dq.put((server,domain))

    for i in range(6):
        threads = []
        try:
            thr = threading.Thread(target=linux_ping_thr,args=(dq,))
            thr.start()
            threads.append(thr)
            
        except:
            print('[*] ping failed.')

    # for thr in threads:
    #     thr.start()
    for thr in threads:
        thr.join()
    
    print("Length: ",q.qsize())
    while not q.empty():
        s,l = q.get()
        server_latency[s] = l

    pprint(server_latency)
    # min_latency = min(server_latency,key=server_latency.get)
    # print(min_latency)


if __name__ == "__main__":
    main()

当我将thr.start()移至循环时，就像注释一样。然后脚本运行得非常慢，就像单线程一样。

我认为可能是子进程和线程问题，但是我不知道如何解决。

这是输出。

[+]  (Asia)Seoul,Korea[韩国 首尔]  ping started.
[+]  (Asia)Tokyo,Japan[日本 东京]  ping started.
[+]  Singapore[新加坡]  ping started.
[+]  (AU) Sydney,Australia[悉尼]  ping started.
[+]  (EU) Frankfurt,DE[德国 法兰克福]  ping started.
[+]  (EU) Amsterdam,NL[荷兰 阿姆斯特丹]  ping started.
[+] (Asia)Seoul,Korea[韩国 首尔] ping stoped.
[+]  (EU) London,UK[英国 伦敦]  ping started.
[+] (AU) Sydney,Australia[悉尼] ping stoped.
[+]  (EU) Paris,France[法国 巴黎]  ping started.
[+] (EU) Amsterdam,NL[荷兰 阿姆斯特丹] ping stoped.
[+]  Seattle,Washington[美东 西雅图]  ping started.
[+] (Asia)Tokyo,Japan[日本 东京] ping stoped.
[+]  Silicon Valley,Ca[美西 加州 硅谷]  ping started.
[+] Singapore[新加坡] ping stoped.
[+]  Los Angeles,Ca[美西 加州 洛杉矶]  ping started.
[+] (EU) Frankfurt,DE[德国 法兰克福] ping stoped.
[+]  Chicago,Illinois[美东 芝加哥]  ping started.
[+] Silicon Valley,Ca[美西 加州 硅谷] ping stoped.
[+]  Dallas,Texas[美中 德克萨斯州 达拉斯]  ping started.
[+] Seattle,Washington[美东 西雅图] ping stoped.
[+]  New York / New Jersey[美东 新泽西]  ping started.
[+] (EU) London,UK[英国 伦敦] ping stoped.
[+]  Atlanta,Georgiaa[美东 亚特兰大]  ping started.
[+] Los Angeles,Ca[美西 加州 洛杉矶] ping stoped.
[+]  Miami,Florida[美东 迈阿密]  ping started.
[+] Chicago,Illinois[美东 芝加哥] ping stoped.
[+] (EU) Paris,France[法国 巴黎] ping stoped.
[+] New York / New Jersey[美东 新泽西] ping stoped.
Length:  13
{'(AU) Sydney,Australia[悉尼]': 180.772,'(Asia)Seoul,Korea[韩国 首尔]': 109.185,'(Asia)Tokyo,Japan[日本 东京]': 94.31,'(EU) Amsterdam,NL[荷兰 阿姆斯特丹]': 250.362,'(EU) Frankfurt,DE[德国 法兰克福]': 269.163,'(EU) London,UK[英国 伦敦]': 257.464,'(EU) Paris,France[法国 巴黎]': 256.591,'Chicago,Illinois[美东 芝加哥]': 266.876,'Los Angeles,Ca[美西 加州 洛杉矶]': 220.391,'New York / New Jersey[美东 新泽西]': 274.922,'Seattle,Washington[美东 西雅图]': 219.727,'Silicon Valley,Ca[美西 加州 硅谷]': 213.939,'Singapore[新加坡]': 282.892}
[+] Miami,Florida[美东 迈阿密] ping stoped.
[+] Dallas,Texas[美中 德克萨斯州 达拉斯] ping stoped.
[+] Atlanta,Georgiaa[美东 亚特兰大] ping stoped.