为什么这个脚本需要这么长时间才能运行？

我有一个包含200,000行的csv文件。我已加载到这个数据帧和想用摊贩与下面的脚本匿名保护：为什么这个脚本需要这么长时间才能运行？

for i in range(MasterDE1.FirstName.size): 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Female'), ['FirstName','LastName']] = fake.first_name_female(),fake.last_name_female() 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Male'), ['FirstName','LastName']] = fake.first_name_male(),fake.last_name_male() 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Unknown'), ['FirstName','LastName']] = fake.first_name(),fake.last_name() 
    MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName'] 
    MasterDE1['EmailAddress'] = 'smithandthunder' + str(i+1) + '@gmail.com'

它已经从过去的20几分钟内运行（我不认为内核是死的）。

来源

2017-04-25 Data_Kid

不介意负面的问题，但会很感激评论，所以我可以改善未来的问题 –

而是在每次迭代更新数据框，你可以首先生成的名称，然后分配：

df = pd.DataFrame({'Gender': np.random.choice(['Female', 'Male', 'Unknown'], p=[0.45, 0.45, 0.1], size=2*10**5), 
        'First Name': np.nan, 'Last Name': np.nan}) 


df.head() 
Out: 
    First Name Gender Last Name 
0   NaN Female  NaN 
1   NaN Male  NaN 
2   NaN Female  NaN 
3   NaN Male  NaN 
4   NaN Male  NaN 

df.shape 
Out: (200000, 3)

现在下面应完成在几分钟内：

df.loc[df['Gender']=='Female', ('First Name', 'Last Name')] = [(fake.first_name_female(), fake.last_name_female()) for _ in range(df[df['Gender']=='Female'].shape[0])] 

df.loc[df['Gender']=='Male', ('First Name', 'Last Name')] = [(fake.first_name_male(), fake.last_name_male()) for _ in range(df[df['Gender']=='Male'].shape[0])] 

df.loc[df['Gender']=='Unknown', ('First Name', 'Last Name')] = [(fake.first_name(), fake.last_name()) for _ in range(df[df['Gender']=='Unknown'].shape[0])] 

df.head() 
Out: 
    First Name Gender Last Name 
0  Ruth Female  Moore 
1 Christina Female  Jones 
2 Lindsey Female  Davis 
3  Aaron Unknown Watkins 
4  Joshua  Male  Henry

之后，像df['Name'] = df['First Name'] + ' ' + df['Last Name']这样的东西应该很快。

来源

2017-04-25 08:46:59 ayhan

像魅力一样工作！非常感谢！ –

@Data_Kid欢迎您。 :) – ayhan

您可以省略循环：

MasterDE1 = pd.DataFrame({'Gender__pc':['Female','Male','Unknown'], 
         'FirstName':['s','d','f'], 
         'LastName': ['d','f','r']}) 
MasterDE1 = pd.concat([MasterDE1]*3).reset_index(drop=True) 
print (MasterDE1) 
    FirstName Gender__pc LastName 
0   s  Female  d 
1   d  Male  f 
2   f Unknown  r 
3   s  Female  d 
4   d  Male  f 
5   f Unknown  r 
6   s  Female  d 
7   d  Male  f 
8   f Unknown  r 

def f1(): 
    return 'first_name_female' + str(np.random.randint(100)) 
def f2(): 
    return 'last_name_female' + str(np.random.randint(100)) 

maskfem = (MasterDE1["Gender__pc"] == 'Female') 
a = pd.Series(((np.arange(len(MasterDE1.index))) + 1).astype(str)) 

MasterDE1.loc[maskfem, 'FirstName'] = [f1() for x in np.arange(maskfem.sum())] 
MasterDE1.loc[maskfem, 'LastName'] = [f2() for x in np.arange(maskfem.sum())] 

MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName'] 
MasterDE1['EmailAddress'] = 'smithandthunder' + a + '@gmail.com'

print (MasterDE1) 
      FirstName Gender__pc   LastName \ 
0 first_name_female70  Female last_name_female64 
1     d  Male     f 
2     f Unknown     r 
3 first_name_female6  Female last_name_female67 
4     d  Male     f 
5     f Unknown     r 
6 first_name_female59  Female last_name_female99 
7     d  Male     f 
8     f Unknown     r 

            Name    EmailAddress 
0 first_name_female70 last_name_female64 [email protected] 
1          d f [email protected] 
2          f r [email protected] 
3 first_name_female6 last_name_female67 [email protected] 
4          d f [email protected] 
5          f r [email protected] 
6 first_name_female59 last_name_female99 [email protected] 
7          d f [email protected] 
8          f r [email protected]

来源

2017-04-25 07:06:53 jezrael

谢谢。当我尝试这样做时，我得到这个错误：TypeError：ufunc'add'不包含与签名匹配类型的循环dtype（'

我相信' fake.first_name_female（）'（和其他人）每次调用时都会生成新名称。因此，循环或应用是必要的。 – ayhan

是@ayhan。我已经尝试过这种方式，它给了整个表的相同名称。我希望所有的名字都有所不同。 –

我不知道确切地告诉你它为什么采取这一长，但它可能是因为该文件的大小。

但是，你能找到一种方法来监视循环知道它是否仍在工作：

signal = 0 

for i in range(0,200000): 
    .... 
    # something going on in the loop 
    .... 
    # signal the loop 
    signal += 1 
    if signal == 50000 or signal == 100000 or signal == 150000: 
     print('It\'s still going!') 
    elif signal > 200000: 
     print('It\'s over 200000 already!') 
     break # or you can raise an error instead of break (raise RuntimeError)

来源

2017-04-25 07:28:46 Alex

感谢您的支持。对未来很有用 –

为什么这个脚本需要这么长时间才能运行？

回答

相关问题