两个矩阵特征的高效配对相关

Question

在Python中，我需要找到矩阵A中的所有要素与矩阵B中的所有要素之间的成对相关性。特别是，我发现在A中给定的特征具有B中的所有特征时，找到最强的Pearson相关性很有趣。我不关心最强的关联是正面的还是负面的。

我做了一个低效的实现，使用两个循环和下面的scipy。不过，我想用np.corrcoef或其他类似的方法来有效地计算它。矩阵A具有形状40000x400和B具有形状40000x1440。我试图有效地做到这一点可以在下面看到，方法find_max_absolute_corr(A,B)。但是，它失败并出现以下错误：

ValueError: all the input array dimensions except for the concatenation axis must match exactly。

import numpy as np 
from scipy.stats import pearsonr 


def find_max_absolute_corr(A, B): 
    """ Finds for each feature in `A` the highest Pearson 
     correlation across all features in `B`. """ 

    max_corr_A = np.zeros((A.shape[1]))  

    for A_col in range(A.shape[1]): 
     print "Calculating {}/{}.".format(A_col+1, A.shape[1]) 

     metric = A[:,A_col] 
     pearson = np.corrcoef(B, metric, rowvar=0) 

     # takes negative correlations into account as well 
     min_p = min(pearson) 
     max_p = max(pearson) 
     max_corr_A[A_col] = max_absolute(min_p, max_p) 

    return max_corr_A 


def max_absolute(min_p, max_p): 
    if np.isnan(min_p) or np.isnan(max_p): 
     raise ValueError("NaN correlation.") 
    if abs(max_p) > abs(min_p): 
     return max_p 
    else: 
     return min_p 


if __name__ == '__main__': 

    A = np.array(
     [[10, 8.04, 9.14, 7.46], 
     [8, 6.95, 8.14, 6.77], 
     [13, 7.58, 8.74, 12.74], 
     [9, 8.81, 8.77, 7.11], 
     [11, 8.33, 9.26, 7.81]]) 

    B = np.array(
     [[-14, -9.96, 8.10, 8.84, 8, 7.04], 
     [-6, -7.24, 6.13, 6.08, 5, 5.25], 
     [-4, -4.26, 3.10, 5.39, 8, 5.56], 
     [-12, -10.84, 9.13, 8.15, 5, 7.91], 
     [-7, -4.82, 7.26, 6.42, 8, 6.89]]) 

    # simple, inefficient method 
    for A_col in range(A.shape[1]): 
     high_corr = 0 
     for B_col in range(B.shape[1]): 
      corr,_ = pearsonr(A[:,A_col], B[:,B_col]) 
      high_corr = max_absolute(high_corr, corr) 
     print high_corr 

    # -0.161314601631 
    # 0.956781516149 
    # 0.621071009239 
    # -0.421539304112   

    # efficient method 
    max_corr_A = find_max_absolute_corr(A, B) 
    print max_corr_A 

    # [-0.161314601631, 
    # 0.956781516149, 
    # 0.621071009239, 
    # -0.421539304112] 

Answer 1

似乎scipy.stats.pearsonr如下从A & B施加在列方向对Pearson相关系数式这个定义 -

基于该公式，可以容易地向量化作为成对计算来自A和B的列是彼此独立的。下面是使用broadcasting一个矢量化溶液 -

# Get number of rows in either A or B 
N = B.shape[0] 

# Store columnw-wise in A and B, as they would be used at few places 
sA = A.sum(0) 
sB = B.sum(0) 

# Basically there are four parts in the formula. We would compute them one-by-one 
p1 = N*np.einsum('ij,ik->kj',A,B) 
p2 = sA*sB[:,None] 
p3 = N*((B**2).sum(0)) - (sB**2) 
p4 = N*((A**2).sum(0)) - (sA**2) 

# Finally compute Pearson Correlation Coefficient as 2D array 
pcorr = ((p1 - p2)/np.sqrt(p4*p3[:,None])) 

# Get the element corresponding to absolute argmax along the columns 
out = pcorr[np.nanargmax(np.abs(pcorr),axis=0),np.arange(pcorr.shape[1])] 

样品运行 -

1）输入：

In [12]: A 
Out[12]: 
array([[ 10. , 8.04, 9.14, 7.46], 
     [ 8. , 6.95, 8.14, 6.77], 
     [ 13. , 7.58, 8.74, 12.74], 
     [ 9. , 8.81, 8.77, 7.11], 
     [ 11. , 8.33, 9.26, 7.81]]) 

In [13]: B 
Out[13]: 
array([[-14. , -9.96, 8.1 , 8.84, 8. , 7.04], 
     [ -6. , -7.24, 6.13, 6.08, 5. , 5.25], 
     [ -4. , -4.26, 3.1 , 5.39, 8. , 5.56], 
     [-12. , -10.84, 9.13, 8.15, 5. , 7.91], 
     [ -7. , -4.82, 7.26, 6.42, 8. , 6.89]]) 

2）原始的多圈的代码运行 -

In [14]: high_corr_out = np.zeros(A.shape[1]) 
    ...: for A_col in range(A.shape[1]): 
    ...:  high_corr = 0 
    ...:  for B_col in range(B.shape[1]): 
    ...:   corr,_ = pearsonr(A[:,A_col], B[:,B_col]) 
    ...:   high_corr = max_absolute(high_corr, corr) 
    ...:  high_corr_out[A_col] = high_corr 
    ...:  

In [15]: high_corr_out 
Out[15]: array([ 0.8067843 , 0.95678152, 0.74016181, -0.85127779]) 

3）提出的码运行 -

个

In [16]: N = B.shape[0] 
    ...: sA = A.sum(0) 
    ...: sB = B.sum(0) 
    ...: p1 = N*np.einsum('ij,ik->kj',A,B) 
    ...: p2 = sA*sB[:,None] 
    ...: p3 = N*((B**2).sum(0)) - (sB**2) 
    ...: p4 = N*((A**2).sum(0)) - (sA**2) 
    ...: pcorr = ((p1 - p2)/np.sqrt(p4*p3[:,None])) 
    ...: out = pcorr[np.nanargmax(np.abs(pcorr),axis=0),np.arange(pcorr.shape[1])] 
    ...: 

In [17]: pcorr # Pearson Correlation Coefficient array 
Out[17]: 
array([[ 0.41895565, -0.5910935 , -0.40465987, 0.5818286 ], 
     [ 0.66609445, -0.41950457, 0.02450215, 0.64028344], 
     [-0.64953314, 0.65669916, 0.30836196, -0.85127779], 
     [-0.41917583, 0.59043266, 0.40364532, -0.58144102], 
     [ 0.8067843 , 0.07947386, 0.74016181, 0.53165395], 
     [-0.1613146 , 0.95678152, 0.62107101, -0.4215393 ]]) 

In [18]: out # elements corresponding to absolute argmax along columns 
Out[18]: array([ 0.8067843 , 0.95678152, 0.74016181, -0.85127779]) 

运行测试 -

In [36]: A = np.random.rand(4000,40) 

In [37]: B = np.random.rand(4000,144) 

In [38]: np.allclose(org_app(A,B),proposed_app(A,B)) 
Out[38]: True 

In [39]: %timeit org_app(A,B) # Original approach 
1 loops, best of 3: 1.35 s per loop 

In [40]: %timeit proposed_app(A,B) # Proposed vectorized approach 
10 loops, best of 3: 39.1 ms per loop 

Answer 2

添加到从个人的经验上面的回答，

p1 = N*np.dot(B.T,A) 

相比

p1 = N*np.einsum('ij,ik->kj',A,B) 

当我的方式更快的工作这是特别当A和B是大维矩阵时为真。