NLTK Sentence Tokenizer错误

Question

我注意到NLTK sent_tokenizer在某些日期时会出错。有什么办法来调整，以便它能够正确标记化如下：

valid any day after january 1. not valid on federal holidays, including february 14, 
or with other in-house events, specials, or happy hour. 

目前运行sent_tokenize的结果：

['valid any day after january 1. not valid on federal holidays, including february 14, 
or with other in-house events, specials, or happy hour.'] 

但它不应导致：

['valid any day after january 1.', 'not valid on federal holidays, including february 14, 
    or with other in-house events, specials, or happy hour.'] 

为'1月1日'之后的时期是合法的句子终止字符。

Answer 1

首先，sent_tokenize函数使用用于标记格式良好的英语句子的punkt标记器。因此，通过包括正确的大小写必须解决您的问题：

>>> from nltk import sent_tokenize 
>>> s = 'valid any day after january 1. not valid on federal holidays, including february 14, or with other in-house events, specials, or happy hour.' 
>>> sent_tokenize(s) 
['valid any day after january 1. not valid on federal holidays, including february 14, or with other in-house events, specials, or happy hour.'] 
>>>> 
>>> s2 = 'Valid any day after january 1. Not valid on federal holidays, including february 14, or with other in-house events, specials, or happy hour.' 
>>> sent_tokenize(s2) 
['Valid any day after january 1.', 'Not valid on federal holidays, including february 14, or with other in-house events, specials, or happy hour.'] 

现在，让我们深入挖掘，该PUNKT分词器是Kiss and Strunk (2005)的算法，见https://github.com/nltk/nltk/blob/develop/nltk/tokenize/punkt.py的实施。

此标记生成器把一个文本句子的列表，通过使用 监督的算法来构建该启动句子缩写词， 搭配，和词的典范。其必须在 上训练目标语言的大量明文，然后才能使用 。

所以在sent_tokenize的情况下，我敢肯定这是一个结构良好的英语语料库因此事实句号后市值句子边界的一个强烈的信号列车。并且fullstop本身可能不是，因为我们有像i.e. , e.g.

而且在某些情况下，语料库可能有类似01. put pasta in pot \n02. fill the pot with water的东西。在训练数据中存在这样的句子/文档时，算法很可能认为跟随未捕获的单词的整个句子不是句子边界。

因此，要解决这个问题，我建议如下：

1. 手动细分句子的10-20％，并重新训练标记生成器
2. 特定语料库将您的语料库为形成良好正字使用前sent_tokenize

参见：training data format for nltk punkt