使用java代码将火花作业提交给AWS EMR，并等待执行并获得最终状态

Question

我试图通过AWS EMR SDK API向AWS EMR提交火花作业。 我希望过程提交作业，然后等待作业完成/失败并获取相应的状态。

代码：

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); 
AmazonElasticMapReduce emr = 
     AmazonElasticMapReduceClientBuilder 
       .standard() 
       .withCredentials(new AWSStaticCredentialsProvider(credentials)) 
       .build(); 

HadoopJarStepConfig sparkStepConf = 
     new HadoopJarStepConfig() 
       .withJar("command-runner.jar") 
       .withArgs("spark-submit") 
       .withArgs("--master", "yarn") 
       .withArgs(sparkJarPath) 
       .withArgs(args); 

StepConfig sparkStep = 
     new StepConfig().withName("Spark Step").withActionOnFailure(ActionOnFailure.CONTINUE).withHadoopJarStep(
       sparkStepConf); 

AddJobFlowStepsRequest req = 
     new AddJobFlowStepsRequest().withJobFlowId(clusterId).withSteps(Collections.singletonList(sparkStep)); 
emr.addJobFlowSteps(req); 

找不到东西来获取提交的作业的状态

Answer 1

下面是一个例子（请您在代码的某些领域空虚）：

ListStepsResult stepsResult = emr.listSteps(new ListStepsRequest().withClusterId(clusterId).withStepIds(req.getStepIds())); 
List<StepSummary> stepsList = stepsResult.getSteps(); 
StepSummary stepSummary = stepsList.get(0); 
StepStatus stepSummaryStatus = stepSummary.getStatus(); 
String stepStatus = stepSummaryStatus.getState(); 
StepExecutionState stepState = StepExecutionState.valueOf(stepStatus); 

stepState将有你想要的。