0
我想在包含许多行的文件上使用TfidfVectorizer(),每个文本都包含一个短语。然后我想用一小部分短语做一个测试文件,做TfidfVectorizer(),然后取原始文件和测试文件之间的余弦相似度,这样对于测试文件中的给定短语,我可以检索出前N个匹配原始文件。这里是我的尝试:Python:比较两个不同尺寸的tfidf矩阵内的项目
corpus = tuple(open("original.txt").read().split('\n'))
test = tuple(open("test.txt").read().split('\n'))
from sklearn.feature_extraction.text import TfidfVectorizer
tf = TfidfVectorizer(analyzer='word', ngram_range=(1,3), min_df = 0, stop_words = 'english')
tfidf_matrix = tf.fit_transform(corpus)
tfidf_matrix2 = tf.fit_transform(test)
from sklearn.metrics.pairwise import linear_kernel
def new_find_similar(tfidf_matrix2, index, tfidf_matrix, top_n = 5):
cosine_similarities = linear_kernel(tfidf_matrix2[index:index+1], tfidf_matrix).flatten()
related_docs_indices = [i for i in cosine_similarities.argsort()[::-1] if i != index]
return [(index, cosine_similarities[index]) for index in related_docs_indices][0:top_n]
for index, score in find_similar(tfidf_matrix, 1234567):
print score, corpus[index]
但是我得到:
for index, score in new_find_similar(tfidf_matrix2, 1000, tfidf_matrix):
print score, test[index]
Traceback (most recent call last):
File "<ipython-input-53-2bf1cd465991>", line 1, in <module>
for index, score in new_find_similar(tfidf_matrix2, 1000, tfidf_matrix):
File "<ipython-input-51-da874b8d3076>", line 2, in new_find_similar
cosine_similarities = linear_kernel(tfidf_matrix2[index:index+1], tfidf_matrix).flatten()
File "C:\Users\arron\AppData\Local\Continuum\Anaconda2\lib\site-packages\sklearn\metrics\pairwise.py", line 734, in linear_kernel
X, Y = check_pairwise_arrays(X, Y)
File "C:\Users\arron\AppData\Local\Continuum\Anaconda2\lib\site-packages\sklearn\metrics\pairwise.py", line 122, in check_pairwise_arrays
X.shape[1], Y.shape[1]))
ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 66662 while Y.shape[1] == 3332088
我不会介意组合这两个文件,然后转化,但我想给b确保我不会从任何比较的短语测试文件中的其他词组的测试文件。
任何指针?
优秀,非常感谢。 – brucezepplin