2017-07-03 26 views
2

我想实现Mapside加入使用CompositeTextInoutFormat。不过,我在Map reduce工作中遇到了以下错误,我无法解决这个问题。 1.在下面的代码中,我在使用Compose方法时出现错误,并在设置inputformat Class时出现错误。错误如下所示。无法使用CompositetextinputFormat在Mapside加入

在 类型CompositeInputFormat方法撰写(字符串,类,路径...)不适用于参数 (字符串,类,路径[])

能有人帮

package Hadoop.MR.Practice; 

import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.LongWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapred.TextInputFormat; 
import org.apache.hadoop.mapred.join.CompositeInputFormat; 
import org.apache.hadoop.mapreduce.Job; 
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat; 
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 
import org.apache.hadoop.util.Tool; 
import org.apache.hadoop.util.ToolRunner; 
//import org.apache.hadoop.mapred.join.CompositeInputFormat; 


public class MapJoinJob implements Tool{ 

private Configuration conf;  

public Configuration getConf() { 
    return conf; 
} 
public void setConf(Configuration conf) { 
    this.conf = conf; 
} 
@Override 
public int run(String[] args) throws Exception { 
    Job job = Job.getInstance(getConf(), "MapSideJoinJob"); 
    job.setJarByClass(this.getClass()); 

    Path[] inputs = new Path[] { new Path(args[0]), new Path(args[1])}; 
    String join = CompositeInputFormat.compose("inner", KeyValueTextInputFormat.class, inputs); 
    job.getConfiguration().set("mapreduce.join.expr", join); 

    job.setInputFormatClass(CompositeInputFormat.class); 

    job.setMapperClass(MapJoinMapper.class); 
    job.setMapOutputKeyClass(Text.class); 
    job.setMapOutputValueClass(LongWritable.class); 

    //Configuring reducer 
    job.setReducerClass(WCReducer.class); 
    job.setOutputKeyClass(Text.class); 
    job.setOutputValueClass(LongWritable.class); 
    job.setNumReduceTasks(0); 

    FileOutputFormat.setOutputPath(job, new Path(args[2])); 

    job.waitForCompletion(true); 
    return 0; 
} 

public static void main(String[] args) throws Exception { 
    Configuration conf = new Configuration(); 
    MapJoinJob mjJob = new MapJoinJob(); 
    ToolRunner.run(conf, mjJob, args); 
} 

回答

1

我会说你的问题可能与混合hadoop API有关。您可以看到您的进口产品混合了mapredmapreduce

例如,您尝试使用org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormatorg.apache.hadoop.mapred.join.CompositeInputFormat这不太可能工作。

你应该选择一个(可能mapreduce我会说),并确保一切都使用相同的API。

+0

谢谢很多..会尽力解决这个.. –