scipy/sklearn用于文档分类的稀疏矩阵分解

Question

我试图在使用标准scikit-learn方法时在大型语料库（4 mil文档）上进行文档分类并且遇到内存错误。在清理/填充我的数据后，我有一个非常稀疏的矩阵，大约有1个字。我的第一个想法是使用sklearn.decomposition.TruncatedSVD，但由于内存错误，我无法执行足够大的k的.fit（）操作（我能做的最大操作只占数据方差的25％ ）。我试着按照sklearn分类here，但是在做KNN分类时仍然记忆犹新。 我想手动进行核心矩阵变换，将矩阵PCA/SVD应用于矩阵以降低维度，但需要先计算特征向量的方法。我希望能使用scipy.sparse.linalg.eigs 有没有一种方法来计算特征向量矩阵，以完成下面显示的代码？

from sklearn.feature_extraction.text import TfidfVectorizer 
import scipy.sparse as sp 
import numpy as np 
import cPickle as pkl 
from sklearn.neighbors import KNeighborsClassifier 

def pickleLoader(pklFile): 
    try: 
     while True: 
      yield pkl.load(pklFile) 
    except EOFError: 
     pass 

#sample docs 
docs = ['orange green','purple green','green chair apple fruit','raspberry pie banana yellow','green raspberry hat ball','test row green apple'] 
classes = [1,0,1,0,0,1] 
#first k eigenvectors to keep 
k = 3 

#returns sparse matrix 
tfidf = TfidfVectorizer() 
tfs = tfidf.fit_transform(docs) 

#write sparse matrix to file 
pkl.dump(tfs, open('pickleTest.p', 'wb')) 



#NEEDED - THIS LINE THAT CALCULATES top k eigenvectors 
del tfs 

x = np.empty([len(docs),k]) 

#iterate over sparse matrix 
with open('D:\\GitHub\\Avitro-Classification\\pickleTest.p') as f: 
    rowCounter = 0 
    for dataRow in pickleLoader(f): 
     colCounter = 0 
     for col in k: 
      x[rowCounter, col] = np.sum(dataRow * eingenvectors[:,col]) 
f.close() 

clf = KNeighborsClassifier(n_neighbors=10) 
clf.fit(x, k_class) 

任何援助或指导将不胜感激！如果有更好的方法来做到这一点，我很乐意尝试一种不同的方法，但我想尝试KNN这个大的稀疏数据集，最好使用一些降维（这在我运行的小测试数据集上表现得非常好 - 我讨厌失去我的性能，因为愚蠢的内存限制）

编辑：这是我第一次尝试运行代码，导致我下来做我自己外的核心稀疏PCA实现的路径。修复这个内存错误的任何帮助都会让这个更容易！

from sklearn.decomposition import TruncatedSVD 
import pickle 

dataFolder = 'D:\\GitHub\\project\\' 

# in the form of a list: [word sample test word, big sample test word test, green apple test word] 
descWords = pickle.load(open(dataFolder +'descriptionWords.p')) 

vectorizer = TfidfVectorizer() 
X_words = vectorizer.fit_transform(descWords) 

print np.shape(X_words) 

del descWords 
del vectorizer 

svd = TruncatedSVD(algorithm='randomized', n_components=50000, random_state=42) 
output = svd.fit_transform(X_words) 

与输出：

(3995803, 923633) 
--------------------------------------------------------------------------- 
MemoryError        Traceback (most recent call last) 
<ipython-input-27-c0db86bd3830> in <module>() 
    16 
    17 svd = TruncatedSVD(algorithm='randomized', n_components=50000, random_state=42) 
---> 18 output = svd.fit_transform(X_words) 

C:\Python27\lib\site-packages\sklearn\decomposition\truncated_svd.pyc in fit_transform(self, X, y) 
    173    U, Sigma, VT = randomized_svd(X, self.n_components, 
    174           n_iter=self.n_iter, 
--> 175           random_state=random_state) 
    176   else: 
    177    raise ValueError("unknown algorithm %r" % self.algorithm) 

C:\Python27\lib\site-packages\sklearn\utils\extmath.pyc in randomized_svd(M, n_components, n_oversamples, n_iter, transpose, flip_sign, random_state, n_iterations) 
    297   M = M.T 
    298 
--> 299  Q = randomized_range_finder(M, n_random, n_iter, random_state) 
    300 
    301  # project M to the (k + p) dimensional space using the basis vectors 

C:\Python27\lib\site-packages\sklearn\utils\extmath.pyc in randomized_range_finder(A, size, n_iter, random_state) 
    212 
    213  # generating random gaussian vectors r with shape: (A.shape[1], size) 
--> 214  R = random_state.normal(size=(A.shape[1], size)) 
    215 
    216  # sampling the range of A using by linear projection of r 

C:\Python27\lib\site-packages\numpy\random\mtrand.pyd in mtrand.RandomState.normal (numpy\random\mtrand\mtrand.c:9968)() 

C:\Python27\lib\site-packages\numpy\random\mtrand.pyd in mtrand.cont2_array_sc (numpy\random\mtrand\mtrand.c:2370)() 

MemoryError: 

Answer 1

稀疏数据外的核心SVD或PCA 不执行scikit学习0.15.2。您可能需要尝试gensim。

编辑：我忘了在我的第一个回复中指定“关于稀疏数据”。