在Prolog中计算字词频率

Question

我搜索了Google无济于事。这里的任何人都可以告诉我如何在Prolog中对术语频率进行编码？越“逻辑”越好。 给定一个文本文件，忽略非字母数字字符，检测单词，删除停用词（可能在外部文件中给出），计算文件中每个单词的出现次数并输出按照降低频率排序的“word：freq”。

谢谢！

（这不是功课，顺便说一句，这是我做的一个项目）

Answer 1

这里的东西，让你开始：

它假设的话是每个输入的单独的行文件

无法处理 “停止词”

没有实现那种

; Print the term frequency for words in file Path 
print_freq_from_file(Path) :- 
    open(Path, read, Stream), 
    read_file(Stream, Words), 
    calc_freq(Words, Freq), 
    sort_freq(Freq, SortedFreq), 
    print_freq(SortedFreq). 

; calc_freq(Words,Freq) Freq is the term frequency of the words in list Words 
calc_freq([], []). 
calc_freq([Word|Words], Freq) :- 
    calc_freq(Words, FreqRest), 
    add_word(Word, FreqRest, Freq). 

; add_word(Word, Freq1, Freq2) Freq2 is Freq1 with Word added 
add_word(Word, [], [(Word,1)]). 
add_word(Word, [(Word,N)|Rest], [(Word,N1)|Rest]) :- 
    N1 is N+1. 
add_word(Word, [Term|Rest], [Term|NewRest]) :- 
    add_word(Word, Rest, NewRest). 

; Print the given term frequency 
print_freq([]). 
print_freq([(W,N)|Rest]) :- 
    write(W), write(' : '), write(N), nl, 
    print_freq(Rest). 

Answer 2

对于过滤停止的话，一个好的办法是通过assert：

read_sw(Path) :- 
    open(Path, read, Stream), 
    read_words_file(Stream, Words), 
    add_sw(Words). 

add_sw([]). 
add_sw([Word|Rest]) :- 
    assertz(stop_word(Word)), 
    add_sw(Rest). 

一旦做到这一点，你可以用stop_word/1检查一个词是停用词，并忽略它。举例来说，如果你正在实施斯科特·亨特的回答，您可以在calc_freq/2增加一个附加条款，以筛选出那些话：

calc_freq([], []). 
calc_freq([Word|Words], Freq) :- 
    stop_word(Word), !, 
    calc_freq(Words, Freq). 
calc_freq([Word|Words], Freq) :- 
    calc_freq(Words, FreqRest), 
    add_word(Word, FreqRest, Freq). 

Answer 3

我用这个答案SWI-Prolog的丰富库支持，因此可能是不合适的案件。

当然，你是如何解决任何编程任务强力地什么在你选择的语言，从你使用它的能力和可用的影响。

这里我使用library(assoc)，如果你的系统错过了它，那么你可以使用列表模拟，或者使用assert/retract。

这个片段只是显示了一个（惯用）的方式计数单词，计算频率可以很容易地与库（聚合）或一些算术，可能你想尝试记下自己在锻炼语言...

/* File: frequency_of_words.pl 
    Author: Carlo,,, 
    Created: May 23 2012 
    Purpose: http://stackoverflow.com/questions/10711483/calculating-term-frequency-in-prolog 
*/ 

:- module(frequency_of_words, [count_words/2, count_words/1]). 
:- [library(assoc)]. 

count_words(File, Assoc) :- 
    empty_assoc(Empty), 
    open(File, read, Stream), 
    frequency_of_words(Stream, Empty, Assoc, ""), 
    close(Stream). 
count_words(File) :- 
    count_words(File, Assoc), 
    assoc_to_list(Assoc, List), 
    maplist(writeln, List). 

frequency_of_words(Stream, SoFar, Words, CurrWord) :- 
    get_code(Stream, Code), 
    ( Code == -1 
    -> update_dictionary(SoFar, Words, CurrWord) 
    ; use_character(Code, SoFar, Updated, CurrWord, NextWord), 
     frequency_of_words(Stream, Updated, Words, NextWord) 
    ). 

update_dictionary(SoFar, SoFar, Word) :- 
    skip_word(Word). 
update_dictionary(SoFar, Updated, Codes) :- 
    atom_codes(Word, Codes), 
    (get_assoc(Word, SoFar, CountSoFar) ; CountSoFar = 0), 
    WordCount is CountSoFar + 1, 
    put_assoc(Word, SoFar, WordCount, Updated). 

use_character(Code, SoFar, Updated, CurrWord, NextWord) :- 
    ( word_character(Code) 
    -> Updated = SoFar, 
     NextWord = [Code|CurrWord] 
    ; reverse(CurrWord, Forward), 
     update_dictionary(SoFar, Updated, Forward), 
     NextWord = "" 
    ). 

word_character(Code) :- 
    [Code] @>= "A", [Code] @=< "Z" ; 
    [Code] @>= "a", [Code] @=< "z" ; 
    [Code] @>= "0", [Code] @=< "9" ; 
    [Code] == "_". 

skip_word(""). % a trick on EOF or consecutive blanks: not really a skipword 
skip_word("is"). 

测试：

?- count_words('frequency_of_words.pl'). 
0-2 
1-3 
2012-1 
23-1 
9-1 
A-1 
Author-1 
Carlo-1 
Code-14 
Codes-2 
CountSoFar-3 
Created-1 
CurrWord-6 
EOF-1 
Empty-2 
File-3 
Forward-2 
Full-2 
List-2 
May-1 
NextWord-5 
Purpose-1 
SoFar-11 
StackOverflow-1 
Stream-6 
Updated-7 
Word-5 
WordCount-2 
Words-3 
Z-1 
_-1 
a-3 
answer-1 
assoc-1 
assoc_to_list-1 
atom_codes-1 
blanks-1 
close-1 
consecutive-1 
count_words-2 
empty_assoc-1 
frequency_of_words-5 
get_assoc-1 
get_code-1 
library-1 
maplist-1 
module-1 
not-1 
on-1 
open-1 
or-1 
pl-1 
put_assoc-1 
read-1 
really-1 
reverse-1 
skip_word-3 
skipword-1 
trick-1 
update_dictionary-4 
use_character-2 
word_character-2 
writeln-1 
z-1