使用Whirr启动EC2群集

Question

我正在通过Jeffery Breen的教程进行工作。当我想用Whirr启动一个ec2集群时，我遇到了一些麻烦。我使用了cloudera demo vm cdh3u4。

我下载了whirr的0.8.1版本。

这里是我遇到的所有命令：

$ wget http://mirror.switch.ch/mirror/apache/dist/whirr/whirr-0.8.1/whirr-0.8.1.tar.gz 
$ tar zxf whirr-0.8.0.tar.gz 
$ export PATH="~/whirr-0.8.0/bin:$PATH" 

$ export AWS_ACCESS_KEY_ID=MY ACCESS KEY 
$ export AWS_SECRET_ACCESS_KEY=MY SECRET ACCESS KEY 

$ ssh-keygen -t rsa -P hadoop-ec2 

然后有人问我在哪个文件的关键应该萨法德，我打字：Hadoop的EC2

$ whirr launch-cluster --config hadoop-ec2.properties 

...这里是问题：没有发生实例！我得到了以下信息：

Exception in thread "main" org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source hadoop-ec2.properties 
     at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:249) 
     at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229) 
     at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:149) 
     at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:252) 
     at org.apache.whirr.command.AbstractClusterCommand.getClusterSpec(AbstractClusterCommand.java:122) 
     at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:57) 
     at org.apache.whirr.cli.Main.run(Main.java:69) 
     at org.apache.whirr.cli.Main.main(Main.java:102) 

下一个我所做的是直接链接的属性文件，以杰弗里·布林发表在他的教程，然后我得到了如下的东西文件：

[cloudera@localhost ~]$ whirr launch-cluster --config /home/cloudera/TutorialBreen/config/whirr-ec2/hadoop-ec2.properties 
Running on provider aws-ec2 using identity ${env:AKIAIXPYW6EBNWSZWMTQ} 
Bootstrapping cluster 
Configuring template for bootstrap-hadoop-datanode_hadoop-tasktracker 
Unable to start the cluster. Terminating all nodes. 
org.jclouds.rest.AuthorizationException: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 -> HTTP/1.1 401 Unauthorized 
     at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:123) 
     at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:92) 
     at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69) 
     at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197) 
     at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167) . . . 

当时这是迈向正确方向的一步，如果是的话，我必须做些什么才能发挥作用？

我是一个非常初学者，所以我非常感谢你的帮助，如果可能的话，我会尽可能“清楚”，因为我是 - 正如我所说 - 是一个初学者。

下一个步骤将是运行此命令：

$ whirr run-script --script install-r+packages.sh --config hadoop-ec2.properties 

我真希望在这里找到一些帮助，这样我可以与教程继续。

呼呼-配置文件中：

whirr.cluster-name=hadoop-ec2 

# Change the number of machines in the cluster here 
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5 hadoop-datanode+hadoop-tasktracker 
# whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1 hadoop-datanode+hadoop-tasktracker 

# Uncomment out these lines to run CDH 
# You need cdh3 because of the streaming combiner backport 
whirr.hadoop.install-function=install_cdh_hadoop 
whirr.hadoop.configure-function=configure_cdh_hadoop 
# make sure java is set up correctly, requires Whirr >= 0.7.1 
whirr.java.install-function=install_oab_java 

# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. 
whirr.provider=aws-ec2 
whirr.identity=${env:DFD...(mycode)..DFDSDF} 
whirr.credential=${env:df342.(mycode)..3434324} 

# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/ 
whirr.hardware-id=m1.large 
# whirr.hardware-id=c1.xlarge 

# select recent, 64-bit CentOS 5.6 AMI from RightScale 
whirr.image-id=us-east-1/ami-49e32320 

# here's what Cloudera recommends: 
# whirr.image-id=us-east-1/ami-ccb35ea5 

# If you choose a different location, make sure whirr.image-id is updated too 
whirr.location-id=us-east-1 

# You can also specify the spot instance price 
# http://aws.amazon.com/ec2/spot-instances/ 
# whirr.aws-ec2-spot-price=0.109 

# By default use the user system SSH keys. Override them here. 
# whirr.private-key-file=${sys:user.home}/.ssh/id_rsa 
# whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub 

# Expert: override Hadoop properties by setting properties with the prefix 
# hadoop-common, hadoop-hdfs, hadoop-mapreduce to set Common, HDFS, MapReduce 
# site properties, respectively. The prefix is removed by Whirr, so that for 
# example, setting 
# hadoop-common.fs.trash.interval=1440 
# will result in fs.trash.interval being set to 1440 in core-site.xml. 

# Expert: specify the version of Hadoop to install. 
#whirr.hadoop.version=0.20.2 
#whirr.hadoop.tarball.url=http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz 

Answer 1

是去除#

样品在这里：

whirr.private−key−file=${sys:user.home}/.ssh/whirr_id_rsa 
whirr.public−key−file=${sys:user.home}/.ssh/whirr_id_rsa.pub 

repalce whirr_id_rsa您的文件名。更多详细信息，请访问：

http://www.xmsxmx.com/apache-whirr-create-hadoop-cluster-automatically/