我必须提取一些由Nutch在默认情况下不提供的Apache Nutch 2.3.1抓取数据的元数据信息。为此我必须编写一个插件。为了学习的目的,我以Nutch tutorial为起点。我知道这个教程是为1.x版本。我已经改变了所有必要的分类,并成功地建立了它。以下是我所遵循的步骤。Apache Nutch 2.3.1插件不工作
- 创建类似$ NUTCH_HOME一个的/ src /插件/为myplugin
- 复制指数的元数据,以我的plugina并创建一个文件myField.java CP -r索引元/ *为myplugin/
- 目录列表应该像
myPlugin/plugin.xml build.xml ivy.xml src/java/org/apache/nutch/indexer/AddField.java
- 插件/ myplgin/plugin.xml中应该是这样的
- 变化的build.xml像
- Then
编辑您./conf/nutch-site.xml
<property> <name>plugin.includes</name> <value>plugin-1|plugin-2|myPlugin</value> <description>Added myPlugin</description> </property>
添加下列行分别在schema.xml和solrindex-mapping.xml中
<field name="pageLength" type="long" stored="true" indexed="true"/> <field dest="pageLength" source="pageLength"/>
然后我编我编写的代码(类似于网址给定的例子)
<?xml version="1.0" encoding="UTF-8"?> <plugin id="myPlugin" name="Add Field to Index" version="1.0.0" provider-name="your name"> <runtime> <library name="myPlugin.jar"> <export name="*"/> </library> </runtime> <extension id="org.apache.nutch.indexer.myPlugin" name="Add Field to Index" point="org.apache.nutch.indexer.IndexingFilter"> <implementation id="myPlugin" class="org.apache.nutch.indexer.AddField"/> </extension> </plugin>
<?xml version="1.0" encoding="UTF-8"?> <project name="myPlugin" default="jar"> <import file="../build-plugin.xml"/> </project>
<ant dir="myPlugin" target="deploy" />
当我以本地模式运行的Nutch,以下是索引到Solr步日志信息
Active IndexWriters :
SOLRIndexWriter
solr.server.url : URL of the SOLR instance (mandatory)
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : username for authentication
solr.auth.password : password for authentication
IndexingJob: done.
我在solr schema中也添加了field pageLength。根据我的预期,应该有一个新的字段pageLength具有适当的值,但solr中没有字段。
问题在哪里?它是一个简单的玩具例子。 这是nutch日志文件(hadoop。日志)输出索引步骤
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: content dest: content
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: title dest: title
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: host dest: host
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: batchId dest: batchId
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: boost dest: boost
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: digest dest: digest
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
2016-07-26 16:53:25,649 INFO solr.SolrMappingReader - source: pageLength dest: pageLength
2016-07-26 16:53:26,140 INFO solr.SolrIndexWriter - Total 1 document is added.
2016-07-26 16:53:26,140 INFO indexer.IndexingJob - IndexingJob: done.
如何确认插件是由nutch加载的? 其次,有没有什么办法可以测试Nutch插件,然后将其配置为无法抓取?