EMR 5.0 + Spark无尽循环堆栈

我试图部署Spark 2.0 StreamingAmazon EMR 5.0。似乎应用陷在无限循环INFO客户端日志 “的无限循环‘：申请报告application_14111979683_1111（状态：接受）。’ 然后退出EMR 5.0 + Spark无尽循环堆栈

这里是我如何努力deploy- -

AWS EMR添加步骤--cluster-ID --steps 类型=星火，名称= “星火计划 ”，ActionOnFailure =继续，参数数量= [：通过命令行提交mode，cluster， - class ,, s3：//.jar]

有什么想法？

感谢，叶兰

16/08/30 15:43:27 INFO SecurityManager: Changing view acls to: hadoop 
16/08/30 15:43:27 INFO SecurityManager: Changing modify acls to: hadoop 
16/08/30 15:43:27 INFO SecurityManager: Changing view acls groups to: 
16/08/30 15:43:27 INFO SecurityManager: Changing modify acls groups to: 
16/08/30 15:43:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set() 
16/08/30 15:43:27 INFO Client: Submitting application application_14111979683_1111 to ResourceManager 
16/08/30 15:43:27 INFO YarnClientImpl: Submitted application application_14111979683_1111 
16/08/30 15:43:28 INFO Client: Application report for application_14111979683_1111 (state: ACCEPTED) 
16/08/30 15:43:28 INFO Client: 
    client token: N/A 
    diagnostics: N/A 
    ApplicationMaster host: N/A 
    ApplicationMaster RPC port: -1 
    queue: default 
    start time: 1472571807467 
    final status: UNDEFINED 
    tracking URL: http://xxxxxx:20888/proxy/application_14111979683_1111/ 
    user: hadoop 
16/08/30 15:43:29 INFO Client: Application report for application_14111979683_1111 (state: ACCEPTED)

，这将引发的异常：

16/08/31 08:14:48 INFO Client: 
    client token: N/A 
    diagnostics: Application application_1472630652740_0001 failed 2 times due to AM Container for appattempt_1472630652740_0001_000002 exited with exitCode: 13 
For more detailed output, check application tracking page:http://ip-10-0-0-8.eu-west-1.compute.internal:8088/cluster/app/application_1472630652740_0001Then, click on links to logs of each attempt. 
Diagnostics: Exception from container-launch. 
Container id: container_1472630652740_0001_02_000001 
Exit code: 13 
Stack trace: ExitCodeException exitCode=13: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) 
    at org.apache.hadoop.util.Shell.run(Shell.java:456) 
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) 
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) 
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) 
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)

来源

2016-08-30 EranM

是那组参数是否正确？ '[--deploy-mode，cluster， - class ,, s3：//.jar]'s3：//.jar当然不是一个真正的路径。 – Kristian

不是不完整的路径。当我写这个问题时，它似乎被删除了。 – EranM

这是抛出的真正异常： – EranM

EMR实际上是一个包装纱。因此，我们需要在部署命令行中添加“--master yarn”作为参数。示例： aws emr add-steps --cluster-id j-XXXXXXXXX --steps Type = Spark，Name =“Spark Program”，ActionOnFailure = CONTINUE，Args = [ - deploy-mode，cluster， - master， yarn， - class，com.xxx.MyMainClass，s3：//]

需要的另一件事是从spark的初始化中删除'sparkConf.setMaster（“local [*]”）'，。

来源

2016-09-05 07:50:27 EranM

EMR 5.0 + Spark无尽循环堆栈

回答

相关问题