2016-12-18 72 views
0

我的代码是:不能在代码提交申请火花集群

SparkSession spark = SparkSession.builder().appName("plzzzz").master("local[*]").getOrCreate(); 
Dataset<Row> dataset = spark.read().format("libsvm").load("/home/ceny/tools/123.txt"); 
KMeans kmeans = new KMeans().setK(2).setSeed(1L); 
KMeansModel model = kmeans.fit(dataset);//this line is 20 

的错误是:

16/12/19 00:08:49 WARN KMeans: The input data was not directly cached, which may hurt performance if its parent RDDs are also uncached. 
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror; 
    at org.apache.spark.ml.clustering.KMeansModel.transform(KMeans.scala:124) 
    at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:326) 
    at Test.main(Test.java:20) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) 

我的build.gradle是:

compile group: 'org.scala-lang', name: 'scala-reflect', version: '2.11.8' 
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.0.2' 
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.0.2' 
compile group: 'org.apache.spark', name: 'spark-mllib_2.10', version: '2.0.2' 

我的环境是:

火花版本:2.0.2

斯卡拉版本:2.11.8

说明:

我尝试执行以下代码IDEA,出现错误。 如果我将它打包为JAR,然后提交它的一切,一切都很好。

虽然如果我用SparkSubmit.main()通过代码提交jar,错误仍然出现。

现在该怎么办?

回答

3

您需要所有工件的相同Scala版本。它应该是:

compile group: 'org.apache.spark', name: 'spark-mllib_2.11', version: '2.0.2' 
+0

是的,谢谢!我在maven仓库中搜索** spark **,并且**广告**让我认为结果的计数仅为7。 – ceny