在nutch 1.3中重新抓取网址

Question

我设置了re_crawler来每天抓取一个网站。但它取得这个网站3次。 我应该在nutch中设置什么属性？谢谢。

Answer 1

我想你在过去的几个月里已经找到了自己的解决方案，但这是社区的答案。所述的nutch-default.xml中定义了3个属性：

<property> 
<name>db.default.fetch.interval</name> 
<value>30</value> 
<description>(DEPRECATED) The default number of days between re-fetches of a page. 
</description> 
</property> 

<property> 
<name>db.fetch.interval.default</name> 
<value>2592000</value> 
<description>The default number of seconds between re-fetches of a page (30 days). 
</description> 
</property> 

<property> 
<name>db.fetch.interval.max</name> 
<value>7776000</value> 
<description>The maximum number of seconds between re-fetches of a page 
(90 days). After this period every page in the db will be re-tried, no 
matter what is its status. 
</description> 

可在的nutch-site.xml中被覆盖。