build.sbt：如何添加火花依赖

Question

你好我想下载spark-core，spark-streaming，twitter4j和spark-streaming-twitter在下面的build.sbt文件：

name := "hello" 

version := "1.0" 

scalaVersion := "2.11.8" 

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" 
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

libraryDependencies ++= Seq(
    "org.twitter4j" % "twitter4j-core" % "3.0.3", 
    "org.twitter4j" % "twitter4j-stream" % "3.0.3" 
) 

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating" 

我只是把这个libraryDependencies在线等我不确定要使用哪些版本等。

有人可以请向我解释我应该如何解决这个.sbt文件。我花了几个小时试图弄清楚，但没有任何建议起作用。我安装scala通过自制软件，我的版本2.11.8

我所有的错误都是关于：

Modules were resolved with conflicting cross-version suffixes. 

Answer 1

的问题是，你混合斯卡拉2.11和2.10的文物。您有：

scalaVersion := "2.11.8" 

然后：

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

当被要求2.10神器。您还混合星火版本，而不是使用一个一致的版本：

// spark 1.6.1 
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" 

// spark 1.4.1 
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1" 

// spark 0.9.0-incubating 
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating" 

这里是一个build.sbt能解决两个问题：

name := "hello" 

version := "1.0" 

scalaVersion := "2.11.8" 

val sparkVersion = "1.6.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" %% "spark-streaming" % sparkVersion, 
    "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion 
) 

你也不需要手动添加twitter4j的依赖，因为它们是由spark-streaming-twitter传递添加。