增加额外arguements到HadoopJarStepConfig失败

我试图通过AWS SDK得到这个命令：增加额外arguements到HadoopJarStepConfig失败

hadoop jar /home/hadoop/contrib/streaming/hadoop-streaming.jar -input hdfs:///logs/ -output hdfs:///no_dups -mapper dedup_mapper.py -reducer dedup_reducer.py -file deduplication.py dedup_mapper.py dedup_reducer.py timber.py signature_v4.py

我的Java代码：

HadoopJarStepConfig config = new StreamingStep() 
     .withInputs("hdfs:///logs") 
     .withOutput("hdfs:///no_dups") 
     .withMapper("dedup_mapper.py") 
     .withReducer("dedup_reducer.py") 
     .toHadoopJarStepConfig(); 

Collection<String> aggs = config.getArgs(); 
aggs.add("-file deduplication.py timber.py dedup_mapper.py dedup_reducer.py signature_v4.py"); 
config.setArgs(aggs);

将会产生以下AddJobFlowStepsRequest（的ToString（）时被称为）：

{JobFlowId: j-3TDECOMCOO8HE, Steps: [{Name: DeDup, ActionOnFailure: CONTINUE, HadoopJarStep: {Properties: [], Jar: /home/hadoop/contrib/streaming/hadoop-streaming.jar, Args: [-input, hdfs:///logs, -output, hdfs:///no_dups, -mapper, dedup_mapper.py, -reducer, dedup_reducer.py, -file deduplication.py timber.py dedup_mapper.py dedup_reducer.py signature_v4.py], }, }], }

最后，我在主节点上看到的错误：

2013-04-26 16:43:48,116 ERROR org.apache.hadoop.streaming.StreamJob (main): Unrecognized option: -file deduplication.py timber.py dedup_mapper.py dedup_reducer.py signature_v4.p

奇怪的是，错误日志列出了可用的选项，并且其中包含-file。有其他人看过这个问题吗？

更多的日志：

2013-04-26T16:43:46.105Z INFO Fetching jar file. 

2013-04-26T16:43:47.609Z INFO Working dir /mnt/var/lib/hadoop/steps/9 

2013-04-26T16:43:47.609Z INFO Executing /usr/lib/jvm/java-6-sun/bin/java -cp /home/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/hadoop-tools.jar:/home/hadoop/hadoop-tools-1.0.3.jar:/home/hadoop/hadoop-core.jar:/home/hadoop/lib/*:/home/hadoop/lib/jetty-ext/* -Xmx1000m -Dhadoop.log.dir=/mnt/var/log/hadoop/steps/9 -Dhadoop.log.file=syslog -Dhadoop.home.dir=/home/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA -Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/9/tmp -Djava.library.path=/home/hadoop/native/Linux-amd64-64 org.apache.hadoop.util.RunJar /home/hadoop/contrib/streaming/hadoop-streaming.jar -input hdfs:///logs -output hdfs:///no_dups -mapper dedup_mapper.py -reducer dedup_reducer.py -file deduplication.py timber.py dedup_mapper.py dedup_reducer.py signature_v4.py 

2013-04-26T16:43:48.611Z INFO Execution ended with ret val 1 

2013-04-26T16:43:48.612Z WARN Step failed with bad retval

来源

2013-04-26 Shane

该错误出现的原因是因为整个命令被解释为单个命令选项。

解决的办法是添加的命令选项，然后像这样的论点：

args.add("-file"); 
args.add("myfile.txt");

如果要添加多个文件，那你就去做这样的：

args.add("-file"); 
args.add("myfile.txt"); 
args.add("-file"); 
args.add("myfile2.txt");

如果您只需在一个参数中将文件作为列表提供，那么整行将被解释为文件名，并且可能会抛出错误。

来源

2013-04-29 09:41:13 Shane

增加额外arguements到HadoopJarStepConfig失败

回答

相关问题