无法在群集上运行MR

我有一个在独立（Ecllipse）模式下成功运行的Map Reduce程序，但试图通过在群集中导出该jar来运行相同的MR。它显示这样的空指针异常，无法在群集上运行MR

13/06/26 05:46:22 ERROR mypackage.HHDriver: Error while configuring run method. 
    java.lang.NullPointerException

我对run方法使用了下面的代码。

public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException { 
    Configuration configuration = new Configuration(); 
    Tool headOfHouseHold = new HHDriver(); 

    try { 
     ToolRunner.run(configuration,headOfHouseHold,args); 
    } catch (Exception exception) { 
     exception.printStackTrace(); 
     LOGGER.error("Error while configuring run method", exception); 
     // System.exit(1); 
    } 
}

run方法：

if (args != null && args.length == 8) { 
    // Setting the Configurations 
    GenericOptionsParser genericOptionsParser=new GenericOptionsParser(args); 
    Configuration configuration=genericOptionsParser.getConfiguration(); 

    //Configuration configuration = new Configuration(); 

    configuration.set("fs.default.name", args[0]); 
    configuration.set("mapred.job.tracker", args[1]); 
    configuration.set("deltaFlag",args[2]);         
    configuration.set("keyPrefix",args[3]); 
    configuration.set("outfileName",args[4]); 
    configuration.set("Inpath",args[5]); 
    String outputPath=args[6]; 

    configuration.set("mapred.map.tasks.speculative.execution", "false"); 
    configuration.set("mapred.reduce.tasks.speculative.execution", "false"); 

    // To avoid the creation of _LOG and _SUCCESS files 
    configuration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false"); 
    configuration.set("hadoop.job.history.user.location", "none"); 
    configuration.set(Constants.MAX_NUM_REDUCERS,args[7]); 

    // Configuration of the MR-Job 
    Job job = new Job(configuration, "HH Job"); 
    job.setJarByClass(HHDriver.class); 
    job.setOutputKeyClass(Text.class); 
    job.setOutputValueClass(Text.class); 

    job.setNumReduceTasks(HouseHoldingHelper.numReducer(configuration)); 
    job.setInputFormatClass(TextInputFormat.class); 
    job.setOutputFormatClass(TextOutputFormat.class); 

    MultipleOutputs.addNamedOutput(job,configuration.get("outfileName"), 
            TextOutputFormat.class,Text.class,Text.class); 

    // Deletion of OutputTemp folder (if exists) 
    FileSystem fileSystem = FileSystem.get(configuration); 
    Path path = new Path(outputPath); 

    if (path != null /*&& path.depth() >= 5*/) { 
     fileSystem.delete(path, true); 
    } 

    // Deletion of empty files in the output (if exists) 
    FileStatus[] fileStatus = fileSystem.listStatus(new Path(outputPath)); 
    for (FileStatus file : fileStatus) { 
     if (file.getLen() == 0) { 
      fileSystem.delete(file.getPath(), true); 
     } 
    } 
    // Setting the Input/Output paths 
    FileInputFormat.setInputPaths(job, new Path(configuration.get("Inpath"))); 
    FileOutputFormat.setOutputPath(job, new Path(outputPath)); 

    job.setMapperClass(HHMapper.class); 
    job.setReducerClass(HHReducer.class); 

    job.waitForCompletion(true); 

    return job.waitForCompletion(true) ? 0 : 1;

我双重检查的run方法参数那些不为空，它是在独立模式下运行以及..

来源

2013-06-26 RAVITEJA SATYAVADA

您是否正确创建了hadoop配置？更重要的是你的MR工作依赖于外部罐子？ –

@JitB是的。我检查了它 –

问题可能是因为Hadoop配置没有正确传递给你的程序。你可以尝试把这个在你的驱动程序类的开头：

GenericOptionsParser genericOptionsParser=new GenericOptionsParser(args[]); 
Configuration hadoopConfiguration=genericOptionsParser.getConfiguration();

初始化对象时，然后使用hadoopConfiguration对象。

例如

public int run(String[] args) throws Exception {   
    GenericOptionsParser genericOptionsParser=new GenericOptionsParser(args[]); 
    Configuration hadoopConfiguration=genericOptionsParser.getConfiguration(); 

    Job job = new Job(hadoopConfiguration); 
    //set other stuff 
}

来源

2013-06-26 10:09:09

你是要求放置这些行而不是使用run方法？ –

不，这些进入run方法，使用它来创建配置对象，然后创建hadoop fs对象。 –

请让我知道它是否工作 –

无法在群集上运行MR

回答

相关问题