没有矢量化,我可以重新创建矩阵,或多或少,这个顺序操作:
In [703]: documents = ("The sky is blue", "The sun is bright", "The sun in the sky is bright", "We can see the shining sun the bright sun")
得到的话(全部小写)列出的清单:
In [704]: alist = [l.lower().split() for l in documents]
通过
alist
和c
In [705]: aset = set()
In [706]: [aset.update(l) for l in alist]
Out[706]: [None, None, None, None]
In [707]: unq = sorted(list(aset))
In [708]: unq
Out[708]:
['blue',
'bright',
'can',
'in',
'is',
'see',
'shining',
'sky',
'sun',
'the',
'we']
转到:
得到词的排序列表(唯一) ollect字数。 rows
将语句编号,cols
将是唯一字索引
In [709]: rows, cols, data = [],[],[]
In [710]: for i,row in enumerate(alist):
...: for c in row:
...: rows.append(i)
...: cols.append(unq.index(c))
...: data.append(1)
...:
建立从这个数据稀疏矩阵:
In [711]: M = sparse.csr_matrix((data,(rows,cols)))
In [712]: M
Out[712]:
<4x11 sparse matrix of type '<class 'numpy.int32'>'
with 21 stored elements in Compressed Sparse Row format>
In [713]: print(M)
(0, 0) 1
(0, 4) 1
(0, 7) 1
(0, 9) 1
(1, 1) 1
....
(3, 9) 2
(3, 10) 1
In [714]: M.A # viewed as 2d array
Out[714]:
array([[1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0],
[0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0],
[0, 1, 0, 1, 1, 0, 0, 1, 1, 2, 0],
[0, 1, 1, 0, 0, 1, 1, 0, 2, 2, 1]], dtype=int32)
由于这是使用sklearn
,我可以重现你的矩阵:
In [717]: from sklearn import feature_extraction
In [718]: tf = feature_extraction.text.TfidfVectorizer()
In [719]: tfM = tf.fit_transform(documents)
In [720]: tfM
Out[720]:
<4x11 sparse matrix of type '<class 'numpy.float64'>'
with 21 stored elements in Compressed Sparse Row format>
In [721]: print(tfM)
(0, 9) 0.34399327143
(0, 7) 0.519713848879
(0, 4) 0.420753151645
....
(3, 5) 0.374599471224
(3, 6) 0.374599471224
In [722]: tfM.A
Out[722]:
array([[ 0.65919112, 0. , 0. , 0. , 0.42075315,
0. , 0. , 0.51971385, 0. , 0.34399327,
0. ],....
[ 0. , 0.23910199, 0.37459947, 0. , 0. ,
0.37459947, 0.37459947, 0. , 0.47820398, 0.39096309,
0.37459947]])
的实际数据被存储为3个属性数组:
In [723]: tfM.indices
Out[723]:
array([ 9, 7, 4, 0, 9, 4, 8, 1, 9, 7, 4, 8, 1, 3, 9, 8, 1,
10, 2, 5, 6], dtype=int32)
In [724]: tfM.data
Out[724]:
array([ 0.34399327, 0.51971385, 0.42075315, 0.65919112, 0.42685801,
...
0.37459947])
In [725]: tfM.indptr
Out[725]: array([ 0, 4, 8, 14, 21], dtype=int32)
对各行的indices
值告诉我们哪些词出现在了那句话:
In [726]: np.array(unq)[M[0,].indices]
Out[726]:
array(['blue', 'is', 'sky', 'the'],
dtype='<U7')
In [727]: np.array(unq)[M[3,].indices]
Out[727]:
array(['bright', 'can', 'see', 'shining', 'sun', 'the', 'we'],
dtype='<U7')
这看起来像一个收集某种关于句子统计在列表中的矩阵(其中4)和独特的字(11?)。例如,第一行有4个矩阵项,4个字。 'tfidt_matrix.A'应该以传统的矩阵形式显示它。 – hpaulj
@hpaulj:你能帮我写下更详细的矩阵吗? –