NLTK FreqDist，绘制标准化计数？

Question

在NLTK，你可以很容易地计算的话计数文本，比方说，通过做

from nltk.probability import FreqDist 
fd = FreqDist([word for word in text.split()]) 

，其中文本是一个字符串。 现在，你可以绘制分布

fd.plot() 

，这将让你与计数每个字一个很好的线图。在docs中没有提到绘制实际频率的方法，您可以在fd.freq(x)中看到。

绘制标准化计数的任何直接方法，不需要将数据转化为其他数据结构，分别标准化和绘图？

Answer 1

您可以更新FD [文字]与FD [文字] /总

from nltk.probability import FreqDist 

text = "This is an example . This is test . example is for freq dist ." 
fd = FreqDist([word for word in text.split()]) 

total = fd.N() 
for word in fd: 
    fd[word] /= float(total) 

fd.plot() 

注意：您将失去原始FreqDist值。

Answer 2

请原谅缺少文件。在nltk,FreqDist为您提供文本中的原始计数（即单词的频率），但ProbDist为您提供给定文本的单词的概率。

欲了解更多信息，你必须做一些读码：https://github.com/nltk/nltk/blob/develop/nltk/probability.py

的具体线路是做正常化自带形式https://github.com/nltk/nltk/blob/develop/nltk/probability.py#L598

因此，要获得一个标准化ProbDist，你可以做到以下几点：

>>> from nltk.corpus import brown 
>>> from nltk.probability import FreqDist 
>>> from nltk.probability import DictionaryProbDist 
>>> brown_freqdist = FreqDist(brown.words()) 
# Cast the frequency distribution into probabilities 
>>> brown_probdist = DictionaryProbDist(brown_freqdist) 
# Something strange in NLTK to note though 
# When asking for probabilities in a ProbDist without 
# normalization, it looks it returns the count instead... 
>>> brown_freqdist['said'] 
1943 
>>> brown_probdist.prob('said') 
1943 
>>> brown_probdist.logprob('said') 
10.924070185585345 
>>> brown_probdist = DictionaryProbDist(brown_freqdist, normalize=True) 
>>> brown_probdist.logprob('said') 
-9.223104921442907 
>>> brown_probdist.prob('said') 
0.0016732805599763002