0
我在Amazon AWS上使用Elastic Mapreduce基础结构。流量自动终止。根据亚马逊控制台的最后状态更改原因是:“作业流程中的所有奴隶都已终止”。AWS EMR错误:作业流程中的所有从属都已终止
创建jobflow命令:
elastic-mapreduce --create --name MyCluster --alive --instance-group master --instance-type m1.xlarge --instance-count 1 --bid-price 2.0 --instance-group core --instance-type m1.xlarge --instance-count 10 --bid-price 2.0 --hive-interactive --enable-debugging
详情约jobflow:日志
最后几行...
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201310231204_0099, Tracking URL = http://ip-10-197-16-105.us-west-1.compute.internal:9100/jobdetails.jsp?jobid=job_201310231204_0099
Kill Command = /home/hadoop/bin/hadoop job -Dmapred.job.tracker=10.197.16.105:9001 -kill job_201310231204_0099
2013-10-23 14:11:38,618 Stage-1 map = 0%, reduce = 0%
2013-10-23 14:11:48,741 Stage-1 map = 100%, reduce = 0%
正如你可以在上面记录看到,没有错误如此引发。
我想原因是
我认为,这是因为发生在价格OD现场情况突然增加。更多细节在我的答案下面。
避免这种问题的策略之一是将主控和核心实例保持为按需和任务实例。这将防止集群发生价格跳跃,但会导致成本上升。 AWS Datapipeline使用此策略。 –