在亚马逊EMR上用纱线运行sqoop2服务器时出错

Question

我正在尝试在Amazon EMR集群上安装sqoop 2（版本1.99.3）（AMI版本3.2.0/Hadoop版本2.4.0）。当我开始sqoop服务器，我看到localhost.log此错误：

Sep 10, 2014 4:55:56 PM org.apache.catalina.core.StandardContext listenerStart 
SEVERE: Exception sending context initialized event to listener instance of class org.apache.sqoop.server.ServerInitializer 
java.lang.RuntimeException: Failure in server initialization 
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57) 
    at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36) 
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206) 
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705) 
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) 
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) 
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) 
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) 
    at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) 
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) 
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) 
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) 
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) 
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) 
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) 
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) 
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) 
    at org.apache.catalina.core.StandardService.start(StandardService.java:525) 
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) 
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) 
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) 
Caused by: org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization 
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:115) 
    at org.apache.sqoop.framework.JobManager.initialize(JobManager.java:215) 
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53) 
    ... 25 more 
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. 
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) 
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) 
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) 
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470) 
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449) 
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:113) 
    ... 27 more 

这就是我所做的，每installation instructions。请注意，使用EMR时，$ HADOOP_HOME是/ home/hadoop。

• 我下载sqoop-1.99.3彬hadoop200.tar.gz从Apache并提取它到$ HADOOP_HOME/sqoop。
• 将以下内容添加到catalina.properties中的common.loader属性中： /home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar， /home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar
• 在sqoop.properties：
  • 我换成@ LOGDIR @以/ home/Hadoop的/ sqoop /日志
  • 我换成@ BASEDIR @与财产org.apache.sqoop.submission.engine.mapreduce.configuration.directory /家/的Hadoop/sqoop
  • ，我取代的/ etc/Hadoop的/ conf目录/ with/home/hadoop/conf/
• 于是，我开始在服务器：斌/ sqoop.sh服务器启动

据我可以从错误中知道的，问题的根源是这一行：

Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. 

，并且此属性在mapred-site.xml中：

<property><name>mapreduce.framework.name</name><value>yarn</value></property> 

，这就是我希望它是。

我觉得我在安装/配置sqoop时缺少一些参数设置，并且我尝试了其他一些东西（比如尝试使用AMI 3.0.4/Hadoop 2.2.0），但一直未能启动sqoop服务器。

fyi。我已阅读Kyle Mulka's blog上的这篇文章，但它引用了不同版本的Hadoop和sqoop，但似乎没有提供对我的配置的深入了解。我已经阅读了这个网站上的其他几页，但还没有找到一个引用我正在使用的Hadoop和sqoop版本的网页。我已经看到这个配置与Cloudera一起运行（sqoop2和Hadoop 2与纱线），但无法弄清楚这将如何转化为EMR安装。

Answer 1

2009年，由于this post反馈，我改变了catalina.properties的common.loader财产从我面前：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar 

这样：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/share/hadoop/yarn/lib/*.jar 

之后，sqoop服务器成功启动。