Flume - HiveSink：java.lang.NoClassDefFoundError：org/apache/hive/hcatalog/streaming/RecordWriter

Question

我想在我的水槽配置中创建一个配置单元接收器，当我运行flume-ng时，出现如下所示的一些依赖性问题。你能告诉我如何克服这种依赖性？它看起来有一些运行时依赖。我已经正确安装了配置单元，并将所需的环境变量设置指向了HIVE_HOME。任何帮助都是有好处的。谢谢。

2016-01-15 14:41:37,757 (conf-file-poller-0) 
[INFO -org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] 
Creating instance of sink: hiveSink, type: hive 
2016-01-15 14:41:37,763 (conf-file-poller-0) 
[ERROR -  org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunn able.run(PollingPropertiesFileConfigurationProvider.java:145)] 
Failed to start agent because dependencies were not found in classpath. Error follows. 

java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter 
at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:223) 
at org.apache.flume.sink.hive.HiveSink.configure(HiveSink.java:203) 
at org.apache.flume.conf.Configurables.configure(Configurables.java:41) 
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413) 
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98) 
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) 
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter 
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
... 13 more 

Answer 1

你应该要么flume-env.sh同时设置HIVE_HOME和HCAT_HOME环境变量或将其提供给用户配置文件。

Answer 2

我正在学习CDH 5.7发行中的Flume。我确实面临同样的问题，flume-env.sh脚本似乎不适合我，所以我利用--classpath命令参数来引用与hive和hive-hcatalog相关的所有库。遵循尝试和错误的方法来添加多个lib文件夹引用。没有找到关于这个论点的很多文档。

例如flume-ng agent --conf /home/cloudera/flume/ --conf-file /home/cloudera/flume/netcat_memchannel_hivesink.conf --name agent1 --classpath "/usr/lib/hive-hcatalog/share/hcatalog/*":"/usr/lib/hive/lib/*"