0
我们正在运行Spark流作业从Kafka读取，转换为csv，然后写入Hbase。我正在使用API CSVBulkLoad来运行批量加载作业。火花作业开始罚款，转换为CSV但csvBulkLoad.run（）开始一个新的MR作业失败与此异常Apache Phoenix 4.7 csvBulkLoad.run（）抛出ClassNotFoundException
Error: java.lang.ClassNotFoundException: org.apache.commons.csv.CSVFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.mapreduce.CsvToKeyValueMapper$CsvLineParser.<init>(CsvToKeyValueMapper.java:96)
at org.apache.phoenix.mapreduce.CsvToKeyValueMapper.setup(CsvToKeyValueMapper.java:69)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
这是我的火花作业命令
spark-submit --jars $(echo lib/*.jar | tr ' ' ',') --class "com.lam.app.Driver" --name "MyBulkLoader" --master yarn --deploy-mode cluster --driver-class-path $(echo lib/*.jar | tr ' ' ',') --driver-memory 4g --executor-memory 1g --num-executors 2 --executor-cores 2 --files conf.properties PipeLine-1.0.jar
早些时候我曾经得到另一个关于丢失类（java.lang.NoClassDefFoundError：com/yammer/metrics/core/MetricsRegistry）的错误，但在包含“--driver-class-path $（echo lib/*。jar | tr' ''，'）“。现在我们得到了关于缺少CSVFormat类的错误。
我的lib目录中包含所有需要的jar文件，包括commons-csv jar，但仍然出现此错误。我使用maven来构建项目。 下面是运行csvbulkload
CsvBulkLoadTool csvBulkLoadTool = new CsvBulkLoadTool();
final org.apache.hadoop.conf.Configuration conf = HBaseConfiguration.create();
System.out.println("ZKQuorum " + this.ZKQUORUM);
conf.set(HConstants.ZOOKEEPER_QUORUM, this.ZKQUORUM);
conf.set("hbase.zookeeper.property.clientPort", String.valueOf(2181));
conf.set("zookeeper.znode.parent", "/hbase-unsecure");
csvBulkLoadTool.setConf(conf);
int exitCode = csvBulkLoadTool.run(new String[] { "--input", "\"" + this.hdfsInputFile + "\"",
"--table", this.TABLENAME, "--zookeeper", this.ZKQUORUM + ":2181:/hbase-unsecure" });
System.out.println("Return code of WDL bulk load execution " + exitCode);
我退出码的代码段为-1始终。请让我知道如果我失去了我的课程路径