Python的熊猫 - 构造多元枢轴表来显示NaN和非NaN的

Question

我具有基于不同气象站几个变量（温度，压力等）的数据集，

stationID | Time | Temperature | Pressure |... 
----------+------+-------------+----------+ 
123  | 1 |  30  | 1010.5 | 
123  | 2 |  31  | 1009.0 | 
202  | 1 |  24  | NaN  | 
202  | 2 |  24.3 | NaN  | 
202  | 3 |  NaN  | 1000.3 | 
... 

和我想的计数想创建一个数据透视表，将显示NaN，并且每气象站非NaN的，这样的数字：

stationID | nanStatus | Temperature | Pressure |... 
----------+-----------+-------------+----------+ 
123  | NaN  |  0  |  0 |  
      | nonNaN |  2  |  2 | 
202  | NaN  |  1  |  2 | 
      | nonNaN |  2  |  1 | 
... 

下面我展示一下我迄今所做的，它的工作原理（以繁琐的方式）的温度。但是，如何获得两个变量的相同，如上所示？

import pandas as pd 
import bumpy as np 
df = pd.DataFrame({'stationID':[123,123,202,202,202], 'Time':[1,2,1,2,3],'Temperature':[30,31,24,24.3,np.nan],'Pressure':[1010.5,1009.0,np.nan,np.nan,1000.3]}) 

dfnull = df.isnull() 
dfnull['stationID'] = df['stationID'] 
dfnull['tempValue'] = df['Temperature'] 
dfnull.pivot_table(values=["tempValue"], index=["stationID","Temperature"], aggfunc=len,fill_value=0) 

输出是：

---------------------------------- 
         tempValue 
stationID | Temperature   
123  | False    2 
202  | False    2 
      | True     1 

Answer 1

UPDATE：感谢@root：

In [16]: df.groupby('stationID')[['Temperature','Pressure']].agg([nans, notnans]).astype(int).stack(level=1) 
Out[16]: 
        Temperature Pressure 
stationID 
123  nans    0   0 
      notnans   2   2 
202  nans    1   2 
      notnans   2   1 

原来的答复：

In [12]: %paste 
def nans(s): 
    return s.isnull().sum() 

def notnans(s): 
    return s.notnull().sum() 
## -- End pasted text -- 

In [37]: df.groupby('stationID')[['Temperature','Pressure']].agg([nans, notnans]).astype(np.int8) 
Out[37]: 
      Temperature   Pressure 
       nans notnans  nans notnans 
stationID 
123     0  2  0  2 
202     1  2  2  1 

Answer 2

我承认这不是最漂亮的解决方案，但它有效。首先定义两个临时列TempNaN和PresNaN：

df['TempNaN'] = df['Temperature'].apply(lambda x: 'NaN' if x!=x else 'NonNaN') 
df['PresNaN'] = df['Pressure'].apply(lambda x: 'NaN' if x!=x else 'NonNaN') 

然后定义使用多指标的结果数据框：

Results['Temperature'] = df.groupby(['stationID','TempNaN'])['Temperature'].apply(lambda x: x.shape[0]) 
Results['Pressure'] = df.groupby(['stationID','PresNaN'])['Pressure'].apply(lambda x: x.shape[0]) 

，并填补了：

Results = pd.DataFrame(index=pd.MultiIndex.from_tuples(list(zip(*[sorted(list(df['stationID'].unique())*2),['NaN','NonNaN']*df['stationID'].nunique()])),names=['stationID','NaNStatus'])) 

Store中的结果数据框计算零值空白值：

Results.fillna(value=0,inplace=True) 

如果更容易，您可以遍历列。例如：

Results = pd.DataFrame(index=pd.MultiIndex.from_tuples(list(zip(*[sorted(list(df['stationID'].unique())*2),['NaN','NonNaN']*df['stationID'].nunique()])),names=['stationID','NaNStatus'])) 
for col in ['Temperature','Pressure']: 
    df[col + 'NaN'] = df[col].apply(lambda x: 'NaN' if x!=x else 'NonNaN') 
    Results[col] = df.groupby(['stationID',col + 'NaN'])[col].apply(lambda x: x.shape[0]) 
    df.drop([col + 'NaN'],axis=1,inplace=True) 
Results.fillna(value=0,inplace=True) 

Answer 3

d = {'stationID':[], 'nanStatus':[], 'Temperature':[], 'Pressure':[]} 

for station_id, data in df.groupby(['stationID']): 

    temp_nans = data.isnull().Temperature.mean()*data.isnull().Temperature.count() 
    pres_nans = data.isnull().Pressure.mean()*data.isnull().Pressure.count() 

    d['stationID'].append(station_id) 
    d['nanStatus'].append('NaN') 
    d['Temperature'].append(temp_nans) 
    d['Pressure'].append(pres_nans) 

    d['stationID'].append(station_id) 
    d['nanStatus'].append('nonNaN') 
    d['Temperature'].append(data.isnull().Temperature.count() - temp_nans) 
    d['Pressure'].append(data.isnull().Pressure.count() - pres_nans) 

df2 = pd.DataFrame.from_dict(d) 
print(df2) 

结果是：

Pressure Temperature nanStatus stationID 
0  0.0   0.0  NaN  123 
1  2.0   2.0 nonNaN  123 
2  2.0   1.0  NaN  202 
3  1.0   2.0 nonNaN  202