有没有办法使用numpy去除循环？

Question

我有一个3D numpy数组input_data（q x m x n），我正在使用它来创建直方图数据以最终绘制，它存储在plot_data（m x n x 2）中。这一步在我的流程中是一个体面的瓶颈，我想知道是否有更快，更“粗糙”的做法。

num_bins = 3 
for i in range(m): 

    for j in range(n): 

     data = input_data[:, i, j] 

     hist, bins = np.histogram(data, bins=num_bins) 

     # Create the (x, y) pairs to plot 
     plot_data[i][j] = np.stack((bins[:-1], hist), axis=1) 

Answer 1

我认为你的样本是相似的，所以直方图是相似的。在这种情况下，您可以简化比较，做一个更加量化的方式：

a=np.random.rand(100000,10,10) 


def f(): # roughly your approach. 
    plotdata=np.zeros((10,10,3),np.int32) 
    for i in range(10): 
     for j in range(10): 
      bins,hist=np.histogram(a[:,i,j],3) 
      plotdata[i,j]=bins 
    return plotdata 

def g(): #vectored comparisons 
    u=(a < 1/3).sum(axis=0) 
    w=(a > 2/3).sum(axis=0) 
    v=len(a)-u-w 
    return np.dstack((u,v,w)) 

对于一个8倍的改进：

In [213]: %timeit f() 
548 ms ± 15.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 

In [214]: %timeit g() 
77.7 ms ± 5.46 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 

Answer 2

下面是箱的通用号码一个量化的方法 -

def vectorized_app(input_data, num_bins): 
    s0 = input_data.min(0) 
    s1 = input_data.max(0) 

    m,n,r = input_data.shape 
    ids = (num_bins*((input_data - s0)/(s1-s0))).astype(int).clip(max=num_bins-1) 
    offset = num_bins*(r*np.arange(n)[:,None] + np.arange(r)) 
    ids3D = ids + offset 
    count3D = np.bincount(ids3D.ravel(), minlength=n*r*num_bins).reshape(n,r,-1) 
    bins3D = create_ranges_nd(s0, s1, num_bins+1)[...,:-1] 

    out = np.empty((n,r,num_bins,2)) 
    out[...,0] = bins3D 
    out[...,1] = count3D 
    return out 

辅助功能（s） -

# https://stackoverflow.com/a/46694364/ @Divakar 
def create_ranges_nd(start, stop, N, endpoint=True): 
    if endpoint==1: 
     divisor = N-1 
    else: 
     divisor = N 
    steps = (1.0/divisor) * (stop - start) 
    return start[...,None] + steps[...,None]*np.arange(N) 

运行测试

原始的方法 -

def org_app(input_data, num_bins): 
    q,m,n = input_data.shape 
    plot_data = np.zeros((m,n,num_bins,2)) 
    for i in range(m): 
     for j in range(n): 
      data = input_data[:, i, j] 
      hist, bins = np.histogram(data, bins=num_bins) 
      plot_data[i][j] = np.stack((bins[:-1], hist), axis=1) 
    return plot_data 

时序和验证 -

让我们测试出形状(100, 100, 100)的大型数据阵列上，并与框的数目为10：

In [967]: # Setup input 
    ...: num_bins = 10 
    ...: m = 100 
    ...: n = 100 
    ...: q = 100 
    ...: input_data = np.random.rand(q,m,n) 
    ...: 
    ...: out1 = org_app(input_data, num_bins) 
    ...: out2 = vectorized_app(input_data, num_bins) 
    ...: print np.allclose(out1, out2) 
    ...: 
True 

In [968]: %timeit org_app(input_data, num_bins) 
1 loop, best of 3: 748 ms per loop 

In [969]: %timeit vectorized_app(input_data, num_bins) 
100 loops, best of 3: 12.7 ms per loop 

In [970]: 748/12.7 # speedup with vectorized one over original 
Out[970]: 58.89763779527559