如何解决当iterable包含数百万个元素时，是否有zip* iterable的替代方法？
我遇到过这样的代码：
from random import randint
class Point:
def __init__(self,x,y):
self.x = x
self.y = y
points = [Point(randint(1,10),randint(1,10)) for _ in range(10)]
xs = [point.x for point in points]
ys = [point.y for point in points]
我认为这段代码是 not Pythonic的，因为它可以重复。如果将另一个维度添加到Point类中，则需要编写一个全新的循环：
zs = [point.z for point in points]
因此，我尝试通过编写如下代码使其更具有Pythonic：
xs,ys = zip(*[(point.x,point.y) for point in p])
如果添加了新尺寸，则没问题：
xs,ys,zs = zip(*[(point.x,point.y,point.z) for point in p])
尽管只有一个循环，但在有数百万个点时，这几乎比另一种解决方案慢十倍。我认为这是因为*运算符需要将数百万个参数解压缩到zip函数中，这很可怕。所以我的问题是：
是否可以更改上面的代码，使其像以前一样快速和 Pythonic （不使用第三方库）？
解决方法
我刚刚测试了几种压缩Point坐标的方法，并随着点数的增加寻找它们的性能。
下面是我用来测试的功能：
def hardcode(points):
# a hand crafted comprehension for each coordinate
return [point.x for point in points],[point.y for point in points]
def using_zip(points):
# using the "problematic" qip function
return zip(*((point.x,point.y) for point in points))
def loop_and_comprehension(points):
# making comprehension from a list of coordinate names
zipped = []
for coordinate in ('x','y'):
zipped.append([getattr(point,coordinate) for point in points])
return zipped
def nested_comprehension(points):
# making comprehension from a list of coordinate names using nested
# comprehensions
return [
[getattr(point,coordinate) for point in points]
for coordinate in ('x','y')
]
使用timeit计时每个函数具有不同点数的时间，结果如下：
comparing processing times using 10 points and 10000000 iterations
hardcode................. 14.12024447 [+0%]
using_zip................ 16.84289724 [+19%]
loop_and_comprehension... 30.83631476 [+118%]
nested_comprehension..... 30.45758349 [+116%]
comparing processing times using 100 points and 1000000 iterations
hardcode................. 9.30594717 [+0%]
using_zip................ 13.74953714 [+48%]
loop_and_comprehension... 19.46766583 [+109%]
nested_comprehension..... 19.27818860 [+107%]
comparing processing times using 1000 points and 100000 iterations
hardcode................. 7.90372457 [+0%]
using_zip................ 12.51523594 [+58%]
loop_and_comprehension... 18.25679913 [+131%]
nested_comprehension..... 18.64352790 [+136%]
comparing processing times using 10000 points and 10000 iterations
hardcode................. 8.27348382 [+0%]
using_zip................ 18.23079485 [+120%]
loop_and_comprehension... 18.00183383 [+118%]
nested_comprehension..... 17.96230063 [+117%]
comparing processing times using 100000 points and 1000 iterations
hardcode................. 9.15848662 [+0%]
using_zip................ 22.70730675 [+148%]
loop_and_comprehension... 17.81126971 [+94%]
nested_comprehension..... 17.86892597 [+95%]
comparing processing times using 1000000 points and 100 iterations
hardcode................. 9.75002857 [+0%]
using_zip................ 23.13891725 [+137%]
loop_and_comprehension... 18.08724660 [+86%]
nested_comprehension..... 18.01269820 [+85%]
comparing processing times using 10000000 points and 10 iterations
hardcode................. 9.96045920 [+0%]
using_zip................ 23.11653558 [+132%]
loop_and_comprehension... 17.98296033 [+81%]
nested_comprehension..... 18.17317708 [+82%]
comparing processing times using 100000000 points and 1 iterations
hardcode................. 64.58698246 [+0%]
using_zip................ 92.53437881 [+43%]
loop_and_comprehension... 73.62493845 [+14%]
nested_comprehension..... 62.99444739 [-2%]
我们可以看到，随着点数的增加，“经过编码”的解决方案与使用gettattr进行理解的解决方案之间的差距似乎会不断缩小。
因此，对于很多点，最好使用从坐标列表生成的理解：
[[getattr(point,coordinate) for point in points]
for coordinate in ('x','y')]
但是，对于少数几点来说，这是最糟糕的解决方案（至少从我测试过的解决方案来看）。
有关信息，这是我用于运行此基准测试的代码：
import timeit
...
def compare(nb_points,nb_iterations):
reference = None
points = [Point(randint(1,100),randint(1,100))
for _ in range(nb_points)]
print("comparing processing times using {} points and {} iterations"
.format(nb_points,nb_iterations))
for func in (hardcode,using_zip,loop_and_comprehension,nested_comprehension):
duration = timeit.timeit(lambda: func(points),number=nb_iterations)
print('{:.<25} {:0=2.8f} [{:0>+.0%}]'
.format(func.__name__,duration,0 if reference is None else (duration / reference - 1)))
if reference is None:
reference = duration
print("-" * 80)
compare(10,10000000)
compare(100,1000000)
compare(1000,100000)
compare(10000,10000)
compare(100000,1000)
compare(1000000,100)
compare(10000000,10)
compare(100000000,1)
zip(*iter)的问题在于它将迭代整个可迭代对象，并将结果序列作为args传递给zip。
因此它们在功能上是相同的：
使用*： xs,ys = zip(*[(p.x,p.y) for p in ((0,1),(0,2),3))])
使用位置： xz,ys = zip((0,3))。
很明显，如果有数百万个位置参数，这将很慢。
唯一的解决方法是迭代器方法。
我在网上搜索了python itertools unzip。可悲的是，最接近的itertools是tee。在上述要点的链接中，从itertools.tee的实现中返回了iunzip的迭代器元组：https://gist.github.com/andrix/106334。
我必须将其转换为python3：
from random import randint
import itertools
import time
from operator import itemgetter
def iunzip(iterable):
"""Iunzip is the same as zip(*iter) but returns iterators,instead of
expand the iterator. Mostly used for large sequence"""
_tmp,iterable = itertools.tee(iterable,2)
iters = itertools.tee(iterable,len(next(_tmp)))
return (map(itemgetter(i),it) for i,it in enumerate(iters))
class Point:
def __init__(self,x,y):
self.x = x
self.y = y
points = [Point(randint(1,10),10)) for _ in range(1000000)]
itime = time.time()
xs = [point.x for point in points]
ys = [point.y for point in points]
otime = time.time() - itime
itime += otime
print(f"original: {otime}")
xs,p.y) for p in points])
otime = time.time() - itime
itime += otime
print(f"unpacking into zip: {otime}")
xs,ys = iunzip(((p.x,p.y) for p in points))
for _ in zip(xs,ys): pass
otime = time.time() - itime
itime += otime
print(f"iunzip: {otime}")
输出：
original: 0.1282501220703125
unpacking into zip: 1.286362886428833
iunzip: 0.3046858310699463
因此，迭代器绝对比解压缩为位置args更好。更不用说一个事实，当我达到1000万点时，我的4GB内存就被吃光了……但是，我不相信上面的iunzip迭代器是最佳的，如果它是python内置的鉴于到目前为止，按照“原始”方法进行两次迭代以进行解压缩仍然是最快的（尝试各种长度的点时，速度要快约4倍）。
好像iunzip应该是东西。我很惊讶它不是内置的python还是itertools的一部分...
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。