最快的方式

Question

我有一个列表的列表，像这样：

import numpy as np 
import random 
import time 
import itertools 

N = 1000 
x =np.random.random((N,N)) 
y = np.zeros((N,N)) 
z = np.random.random((N,N)) 

list_of_lists = [[x, y], [y,z], [z,x]] 

并为每个子表我想计算非零的个数，均值和标准差。

我已经做到了，像这样：

distribution = [] 
alb_mean = [] 
alb_std = [] 


start = time.time() 

for i in range(len(list_of_lists)): 

    one_mean = [] 
    non_zero_l = [] 
    one_list = list_of_lists[i] 

    for n in one_list: 


     #count non_zeros 
     non_zero_count = np.count_nonzero(n) 
     non_zero_l.append(non_zero_count) 

     #assign nans 
     n = n.astype(float) 
     n[n == 0.0] = np.nan 

     #flatten the matrix 
     n = np.array(n.flatten()) 
     one_mean.append(n) 

    #append means and stds 
    distribution.append(sum(non_zero_l)) 
    alb_mean.append(np.nanmean(one_mean)) 
    alb_std.append(np.nanstd(one_mean)) 


end = time.time() 
print "Loop took {} seconds".format((end - start)) 

这需要0.23秒。

我试图使这个更快了第二个选项：

distribution = [] 
alb_mean = [] 
alb_std = [] 


start = time.time() 

for i in range(len(list_of_lists)): 

    for_mean = [] 

    #get one list 
    one_list = list_of_lists[i] 

    #flatten the list 
    chain = itertools.chain(*one_list) 
    flat = list(chain) 

    #count non_zeros 
    non_zero_count = np.count_nonzero(flat) 
    distribution.append(non_zero_count) 

    #remove zeros 
    remove_zero = np.setdiff1d(flat ,[0.0]) 
    alb_mean.append(np.nanmean(remove_zero)) 
    alb_std.append(np.nanstd(remove_zero)) 

end = time.time() 
print "Loop took {} seconds".format((end - start)) 

这实际上是慢，需要0.88秒。

绝对数量的循环让我觉得有一个更好的方法来做到这一点。我已经尝试过numba，但它并没有像在函数中追加一样。一个与3迭代，另一个2迭代 -

Answer 1

版本＃1

很好用的多圈解决您的样品中，你有两个回路循环。所以，它已经接近于矢量化了。唯一的瓶颈是append步骤。

外出时完全矢量化，这里有一个方法 -

a = np.array(list_of_lists, dtype=float) 
zm = a!=0 
avgs = np.einsum('ijkl,ijkl->i',zm,a)/zm.sum(axis=(1,2,3)).astype(float) 

a[~zm] = np.nan 
stds = np.nanstd(a, axis=(1,2,3)) 

使用相同的设置中的问题，这是我得到的时序 -

Loop took 0.150925159454 seconds 
Proposed solution took 0.121352910995 seconds 

版本＃2

我们可以使用average来计算std，因此重新使用avgs作进一步提升：

因此，修改后的版本将是 -

a = np.asarray(list_of_lists) 
zm = a!=0 
N = zm.sum(axis=(1,2,3)).astype(float) 
avgs = np.einsum('ijkl,ijkl->i',zm,a)/N 

diffs = ((a-avgs[:,None,None,None])**2) 
stds = np.sqrt(np.einsum('ijkl,ijkl->i',zm,diffs)/N) 

更新时机 -

Loop took 0.155035018921 seconds 
Proposed solution took 0.0648851394653 seconds