星火saveAsNewAPIHadoopFile产生java.io.IOException：找不到值类

Question

我试图存储一个java对RDD作为一个Hadoop序列文件如下串行：

JavaPairRDD<ImmutableBytesWritable, Put> putRdd = ... 
config.set("io.serializations","org.apache.hadoop.io.serializer.JavaSerialization,org.apache.hadoop.io.serializer.WritableSerialization"); 
putRdd.saveAsNewAPIHadoopFile(outputPath, ImmutableBytesWritable.class, Put.class, SequenceFileOutputFormat.class, config); 

但我得到的异常即使我设置io.serializations：

2017-04-06 14:39:32,623 ERROR [Executor task launch worker-0] executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) 
java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Put'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1192) 
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) 
    at org.apache.spark.scheduler.Task.run(Task.scala:88) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
2017-04-06 14:39:32,669 ERROR [task-result-getter-0] scheduler.TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job 

我如何能解决这个任何想法？

Answer 1

我发现修复，显然Put（和所有的HBase突变）都有一个特定的序列号MutationSerialization。

下面这行修复该问题：

config.setStrings("io.serializations", 
    config.get("io.serializations"), 
    MutationSerialization.class.getName(), 
    ResultSerialization.class.getName());