未能在法兰克福启动AWS-EMR群集，但在北弗吉尼亚州成功

Question

我试图通过适用于AWS的Java SDK启动小型EMR群集。我试图在法兰克福（eu-central-1）发射，但在弗吉尼亚北部（美国东部1号）发射时发射成功。

我的配置：

• MACOS 10.12.1
• 的Java 1.8.102
• AWS SDK的Java 60年1月11日
• 的Hadoop 2.7.3
• 的IntelliJ 2016年2月4日

我已经验证以下内容：

1. 我请求的实例类型（M1Medium）存在于两个区域中。
2. 我正在请求群集（2.7.3）的Hadoop版本是存在于EMR版本（5.2.0）中的版本。
3. 我有适当的IAM Roles来支持群集（默认的 - EMR_EC2_DefaultRole & EMR_DefaultRole），并且它们显然工作正常，因为它们用于在弗吉尼亚北部启动群集。
4. 我有两个区域的EC2密钥对。
5. 我已验证EMR在两个地区均可作为服务提供。
6. 我已经通过我的网络浏览器通过EC2仪表板验证了我正在为两个区域使用正确的可用区域，并且这些区域是健康的。
7. 对于每个集群尝试，我在同一个区域使用一个S3存储桶来输入，输出和EMR日志。

这是启动群集在法兰克福代码：

public static void main(String[] args) throws Exception { 
    parseArgs(args); 

    if (environment.equals("local")) { 
     // Local machine, single node setup. Used in order to debug the M-R logic. 
     String[] p1args = {"input", "output", environment}; 
     Phase1.main(p1args); 
    } else { 
     // EMR setup. This is the main intent of this app. 
     AWSCredentials credentials = null; 
     try { 
      credentials = new ProfileCredentialsProvider().getCredentials(); 
     } catch (Exception e) { 
      throw new AmazonClientException(
        "Cannot load the credentials from the credential profiles file. " + 
          "Please make sure that your credentials file is at the correct " + 
          "location (~/.aws/credentials), and is in valid format.", 
        e); 
     } 

     AmazonElasticMapReduce mapReduce = new AmazonElasticMapReduceClient(credentials); 

     HadoopJarStepConfig jarStep1 = new HadoopJarStepConfig() 
       .withJar("s3n://skill-finder-eu-central-1/jars/SkillFinder.jar") 
       .withMainClass("Phase1") 
       .withArgs("s3n://skill-finder-eu-central-1/input-10K", "s3n://skill-finder-eu-central-1/output-eu-central-1", environment); 

     StepConfig step1Config = new StepConfig() 
       .withName("Phase 1") 
       .withHadoopJarStep(jarStep1) 
       .withActionOnFailure("TERMINATE_JOB_FLOW"); 

     JobFlowInstancesConfig instances = new JobFlowInstancesConfig() 
       .withInstanceCount(5) 
       .withMasterInstanceType(InstanceType.M1Medium.toString()) 
       .withSlaveInstanceType(InstanceType.M1Medium.toString()) 
       .withHadoopVersion("2.7.3") 
       .withEc2KeyName("AWS-EU-CENTRAL-1") 
       .withKeepJobFlowAliveWhenNoSteps(false) 
       .withPlacement(new PlacementType("eu-central-1a")); 

     RunJobFlowRequest runFlowRequest = new RunJobFlowRequest() 
       .withName("skill-finder") 
       .withInstances(instances) 
       .withSteps(step1Config) 
       .withJobFlowRole("EMR_EC2_DefaultRole") 
       .withServiceRole("EMR_DefaultRole") 
       .withReleaseLabel("emr-5.2.0") 
       .withLogUri("s3n://skill-finder-eu-central-1/logs/") 
       .withBootstrapActions(); 

     System.out.println("Submitting the JobFlow Request to Amazon EMR and running it..."); 
     RunJobFlowResult runJobFlowResult = mapReduce.runJobFlow(runFlowRequest); 
     String jobFlowId = runJobFlowResult.getJobFlowId(); 
     System.out.println("Ran job flow with id: " + jobFlowId); 
    } 

} 

当N.弗吉尼亚州发射，我只是us-east-1取代eu-central-1。

这是例外：

Exception in thread "main" com.amazonaws.services.elasticmapreduce.model.AmazonElasticMapReduceException: Specified Availability Zone is not supported. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: 578db9ad-b3bf-11e6-9a57-5179acb16d3f) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1545) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1183) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) 
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) 
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) 
at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.doInvoke(AmazonElasticMapReduceClient.java:1469) 
at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:1445) 
at com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.runJobFlow(AmazonElasticMapReduceClient.java:1255) 
at MRTaskLauncher.main(MRTaskLauncher.java:97) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 

Answer 1

OK，找到了解决办法：我公司推出的集群M3Xlarge实例，代替M2Medium。奇迹般有效！

我如何得到这个：

1. 因为我设法推出集群N.弗吉尼亚州与EMR默认的IAM角色，我开始想，我可以有身份验证问题。当我通过CLI在法兰克福启动了一个集群时，这得到了进一步的支持（在Create and Use IAM Roles with the AWS CLI下找到了示例here）。
2. 我接下来做的是试图通过SDK重新启动集群。群集失败了，但我复制了启动命令，所以我可以通过CLI启动。为此，我点击EMR群集列表（Web界面）中的群集，点击View cluster details，然后点击顶部行AWS CLI export上的按钮。
3. 令人惊讶的是，CLI提供了更多特定的错误消息（与Web界面相比，列出了验证错误），这表明罪魁祸首是实例类型！然后我检查了here以查明法兰克福有哪些实例可用，并选择了一个不需要VPC（M4需要它）的实例，因为我没有精力开始搞这些东西。
4. 有点前奏 - 列出的验证错误导致我找到this。正是这个问题导致我研究了默认IAM角色的问题，并试图使用CLI。