优雅的numpy阵列移位和NaN填充？

Question

我在这里有一个特定的性能问题。我正在与气象预报时间序列，这是我编译成numpy的2D阵列，使得

• DIM0 =时间在该预测一系列工作开始
• DIM1 =预测范围，例如。 0至120小时

现在，我想dim0每小时的间隔，但有些来源只产出预测每N小时。例如，假设N = 3，dim1中的时间步长为M = 1小时。然后我得到像

12:00 11.2 12.2 14.0 15.0 11.3 12.0 
13:00 nan nan nan nan nan nan 
14:00 nan nan nan nan nan nan 
15:00 14.7 11.5 12.2 13.0 14.3 15.1 

但当然也有信息在13:00和14:00以及，因为它可以从12:00预测运行填充。所以我想有这样的事情结束了：

12:00 11.2 12.2 14.0 15.0 11.3 12.0 
13:00 12.2 14.0 15.0 11.3 12.0 nan 
14:00 14.0 15.0 11.3 12.0 nan nan 
15:00 14.7 11.5 12.2 13.0 14.3 15.1 

什么是到那里最快的方式，假设DIM0在1E4和DIM1在1E2的订单的订单？现在我一直在做它，但这是非常缓慢的：

nRows, nCols = dat.shape 
if N >= M: 
    assert(N % M == 0) # must have whole numbers 
    for i in range(1, nRows): 
     k = np.array(np.where(np.isnan(self.dat[i, :]))) 
     k = k[k < nCols - N] # do not overstep 
     self.dat[i, k] = self.dat[i-1, k+N] 

我敢肯定，必须有一个更优雅的方式来做到这一点？任何提示将不胜感激。

Answer 1

使用a=yourdata[:,1:]切片您的数据。

def shift_time(dat): 

    #Find number of required iterations 
    check=np.where(np.isnan(dat[:,0])==False)[0] 
    maxiters=np.max(np.diff(check))-1 

    #No sense in iterations where it just updates nans 
    cols=dat.shape[1] 
    if cols<maxiters: maxiters=cols-1 

    for iters in range(maxiters): 
     #Find nans 
     col_loc,row_loc=np.where(np.isnan(dat[:,:-1])) 

     dat[(col_loc,row_loc)]=dat[(col_loc-1,row_loc+1)] 


a=np.array([[11.2,12.2,14.0,15.0,11.3,12.0], 
[np.nan,np.nan,np.nan,np.nan,np.nan,np.nan], 
[np.nan,np.nan,np.nan,np.nan,np.nan,np.nan], 
[14.7,11.5,12.2,13.0,14.3,15.]]) 

shift_time(a) 
print a 

[[ 11.2 12.2 14. 15. 11.3 12. ] 
[ 12.2 14. 15. 11.3 12. nan] 
[ 14. 15. 11.3 12. nan nan] 
[ 14.7 11.5 12.2 13. 14.3 15. ]] 

要使用你的数据是，也可以稍微改变直接拿去，但是这似乎表明这一条明路：

shift_time(yourdata[:,1:]) #Updates in place, no need to return anything. 

使用蒂亚戈的测试：

tmp = np.random.uniform(-10, 20, (1e4, 1e2)) 
nan_idx = np.random.randint(30, 1e4 - 1,1e4) 
tmp[nan_idx] = np.nan 

t=time.time() 
shift_time(tmp,maxiter=1E5) 
print time.time()-t 

0.364198923111 (seconds) 

如果你真的很聪明，你应该能够逃脱一个np.where。

Answer 2

本垫，辊，基本组合的每次迭代做了你在找什么：

import numpy as np 
from numpy import nan as nan 

# Startup array 
A = np.array([[11.2, 12.2, 14.0, 15.0, 11.3, 12.0], 
       [nan, nan, nan, nan, nan, nan], 
       [nan, nan, nan, nan, nan, nan], 
       [14.7, 11.5, 12.2, 13.0, 14.3, 15.1]]) 

def pad_nan(v, pad_width, iaxis, kwargs): 
    v[:pad_width[0]] = nan 
    v[-pad_width[1]:] = nan 
    return v 

def roll_data(A): 
    idx = np.isnan(A) 
    A[idx] = np.roll(np.roll(np.pad(A,1, pad_nan),1,0), -1, 1)[1:-1,1:-1][idx] 
    return A 

print A 
print roll_data(A) 
print roll_data(A) 

输出给：

[[ 11.2 12.2 14. 15. 11.3 12. ] 
[ nan nan nan nan nan nan] 
[ nan nan nan nan nan nan] 
[ 14.7 11.5 12.2 13. 14.3 15.1]] 

[[ 11.2 12.2 14. 15. 11.3 12. ] 
[ 12.2 14. 15. 11.3 12. nan] 
[ nan nan nan nan nan nan] 
[ 14.7 11.5 12.2 13. 14.3 15.1]] 

[[ 11.2 12.2 14. 15. 11.3 12. ] 
[ 12.2 14. 15. 11.3 12. nan] 
[ 14. 15. 11.3 12. nan nan] 
[ 14.7 11.5 12.2 13. 14.3 15.1]] 

一切都是纯numpy的，所以应该是非常快每次迭代。但是，我不确定创建填充数组和运行多次迭代的成本，如果您尝试使用它，请让我知道结果！

Answer 3

这似乎这样的伎俩：用一些测试数据

import numpy as np 

def shift_time(dat): 
    NX, NY = dat.shape 
    for i in range(NY): 
     x, y = np.where(np.isnan(dat)) 
     xr = x - 1 
     yr = y + 1 
     idx = (xr >= 0) & (yr < NY) 
     dat[x[idx], y[idx]] = dat[xr[idx], yr[idx]] 
    return 

现在：

In [1]: test_data = array([[ 11.2, 12.2, 14. , 15. , 11.3, 12. ], 
          [ nan, nan, nan, nan, nan, nan], 
          [ nan, nan, nan, nan, nan, nan], 
          [ 14.7, 11.5, 12.2, 13. , 14.3, 15.1], 
          [ nan, nan, nan, nan, nan, nan], 
          [ 15.7, 16.5, 17.2, 18. , 14. , 12. ]]) 
In [2]: shift_time(test_data) 
In [3]: print test_data 
Out [3]: 
array([[ 11.2, 12.2, 14. , 15. , 11.3, 12. ], 
     [ 12.2, 14. , 15. , 11.3, 12. , nan], 
     [ 14. , 15. , 11.3, 12. , nan, nan], 
     [ 14.7, 11.5, 12.2, 13. , 14.3, 15.1], 
     [ 11.5, 12.2, 13. , 14.3, 15.1, nan], 
     [ 15.7, 16.5, 17.2, 18. , 14. , 12. ]]) 

并测试了（1E4，1E2）阵列：

In [1]: tmp = np.random.uniform(-10, 20, (1e4, 1e2)) 
In [2]: nan_idx = np.random.randint(30, 1e4 - 1,1e4) 
In [3]: tmp[nan_idx] = nan 
In [4]: time test3(tmp) 
CPU times: user 1.53 s, sys: 0.06 s, total: 1.59 s 
Wall time: 1.59 s 

Answer 4

看哪，布尔索引的力量！

def shift_nans(arr) : 
    while True: 
     nan_mask = np.isnan(arr) 
     write_mask = nan_mask[1:, :-1] 
     read_mask = nan_mask[:-1, 1:] 
     write_mask &= ~read_mask 
     if not np.any(write_mask): 
      return arr 
     arr[1:, :-1][write_mask] = arr[:-1, 1:][write_mask] 

我认为命名是自我解释发生了什么。获取切片右边是一个痛苦，但它似乎是工作：

In [214]: shift_nans_bis(test_data) 
Out[214]: 
array([[ 11.2, 12.2, 14. , 15. , 11.3, 12. ], 
     [ 12.2, 14. , 15. , 11.3, 12. , nan], 
     [ 14. , 15. , 11.3, 12. , nan, nan], 
     [ 14.7, 11.5, 12.2, 13. , 14.3, 15.1], 
     [ 11.5, 12.2, 13. , 14.3, 15.1, nan], 
     [ 15.7, 16.5, 17.2, 18. , 14. , 12. ]]) 

而对于计时：

tmp1 = np.random.uniform(-10, 20, (1e4, 1e2)) 
nan_idx = np.random.randint(30, 1e4 - 1,1e4) 
tmp1[nan_idx] = np.nan 
tmp1 = tmp.copy() 

import timeit 

t1 = timeit.timeit(stmt='shift_nans(tmp)', 
        setup='from __main__ import tmp, shift_nans', 
        number=1) 
t2 = timeit.timeit(stmt='shift_time(tmp1)', # Ophion's code 
        setup='from __main__ import tmp1, shift_time', 
        number=1) 

In [242]: t1, t2 
Out[242]: (0.12696346416487359, 0.3427293070417363)