我需要使用所有reduce任务的结果执行聚合。基本上减少任务找到总数和计数和一个值。我需要添加所有的总和和数量,并找到最终的平均值。在master和reduce之间共享数据
我试着用conf.setInt
中的reduce。但是,当我试图从main函数访问失败
class Main {
public static class MyReducer
extends Reducer<Text, Text,Text,IntWritable> {
public void reduce(Text key, Iterable<Text> values,
Context context
) throws IOException, InterruptedException {
int i = 0;
int fd = 0, fc = 0;
fd = context.getConfiguration().getInt("fd", -1);
fc = context.getConfiguration().getInt("fc", -1);
//when I check the value of fd, fc here they are fine. fc fd is shared across all reduce tasks and the updated value is seen by all reduce task. Only main function doesnt seem to have access to it.
}
}
public static void main(String[] args) throws Exception{
Configuration conf = new Configuration();
conf.setInt("fc", 5);
Job job = new Job(conf, "Flight Data");
job.setJarByClass(FlightData.class);
job.setMapperClass(TokenizerMapper.class);
job.setReducerClass(MyReducer.class);
job.setPartitionerClass(FirstPartitioner.class);
job.setGroupingComparatorClass(GroupComparator.class);
job.setSortComparatorClass(KeyComparator.class);
job.setNumReduceTasks(10);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
flightCount = job.getConfiguration().getInt("fc", -1);
flightDelay = job.getConfiguration().getInt("fd", -1);
//here when I access fc, fd, I get back 5 & 5
System.out.println("Final " + flightCount +" " + flightDelay+ " " + flightDelay/flightCount);
}
你得到什么错误?你也可以添加你使用的语言作为标签吗? – 2013-02-24 02:43:00