python熊猫与numpy数组的搜索性能

Question

我想在熊猫列中搜索一个字符串。我已经读过，它应该是最快排序的第一列和搜索字符串使用搜索排序值。我发现这比在同一个numpy数组上搜索蛮力（array == string）要慢。要知道为什么，我已经进行了以下试验：

import timeit 

setup4 = ''' 
import numpy as np, string, random 

d =  np.array([ 
      u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)) 
      for _ in range(20000) 
      ],dtype=np.object) 
''' 



setup5 = ''' 
import numpy as np, pandas as pd, string, random 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
df_sorted = df.sort(['B','C']) 
e = df_sorted['B'].values 
''' 

setup6 = ''' 
import numpy as np, pandas as pd, string, random 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
f = df['B'].values 
''' 

print(timeit.timeit("index = d == u'ASDASD123ASADKHX'", setup=setup4,number=1000)) 
print(timeit.timeit("index = e == u'ASDASD123ASADKHX'", setup=setup5,number=1000)) 
print(timeit.timeit("index = f == u'ASDASD123ASADKHX'", setup=setup6,number=1000)) 

结果如下：

print(timeit.timeit("index = d == u'ASDASD123ASADKHX'", setup=setup4,number=1000)) 
0.808505267014 

print(timeit.timeit("index = e == u'ASDASD123ASADKHX'", setup=setup5,number=1000)) 

3.06733738226 

print(timeit.timeit("index = f == u'ASDASD123ASADKHX'", setup=setup6,number=1000)) 
1.64207848896 

我的问题在这里：为什么是纯numpy的阵列上的表现好多了？我如何使用从熊猫表中提取的数据实现相同的性能？

非常感谢。

Answer 1

DataFrame中的每个字符串都是一个对象，您从df['B'].values得到的是对象数组。但是，当您通过np.array()创建字符串数组时，它会返回一个数组，每个字符串使用相同的字节数。

下面是一个示例，a是S10 dtype的数组，b是一个带有对象dtype的数组。

import numpy as np 
import random 
import string 
words = ["".join(random.choice(string.ascii_uppercase) for _ in range(10)) for _ in range(10000)] 
a = np.array(words) 
b = a.astype("O") 
%timeit a == "123" 
%timeit b == "123" 

输出：

10000 loops, best of 3: 122 µs per loop 
10000 loops, best of 3: 164 µs per loop 

Answer 2

我测试你的代码在IPython中，并得到了几乎所有相同的性能未排序的数据帧变种分开：

In [85]: 

import numpy as np, string, random 

d =  np.array([ 
      u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)) 
      for _ in range(20000) 
      ],dtype=np.object) 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
df_sorted = df.sort(['B','C']) 
e = df_sorted['B'].values 
f = df['B'].values 
%timeit index = d == u'ASDASD123ASADKHX' 
%timeit index = e == u'ASDASD123ASADKHX' 
%timeit index = f == u'ASDASD123ASADKHX' 
1000 loops, best of 3: 536 µs per loop 
1000 loops, best of 3: 568 µs per loop 
1000 loops, best of 3: 538 µs per loop