使用Scala TTL选项保存到Cassandra的火花

Question

我在Scala中使用Cassandra Spark连接器。 这是我的示例代码，它正在将数据保存到Cassandra。

val data = rdd.map 
(f => new CassandraRow(IndexedSeq("pk", "count"), IndexedSeq(f._1, f._2.toString()))) 

data.saveToCassandra("keyspace", "table") 

然后，我尝试使用TTL。这里是我添加TTL的示例代码。

增加进口

import com.datastax.spark.connector.writer.{TTLOption, WriteConf} 

和增加的TTL saveToCassandra

data.saveToCassandra 
("keyspace", "table", writeConf = WriteConf(ttl = TTLOption.constant(604800)) 

然而，当我编译它，它就会错误。

错误消息

[error] bad symbolic reference. A signature in TTLOption.class refers to term streaming 
[error] in package org.apache.spark which is not available. 
[error] It may be completely missing from the current classpath, or the version on 
[error] the classpath might be incompatible with the version used when compiling TTLOption.class. 
[error] one error found 
[error] (compile:compile) Compilation failed 
[error] Total time: 4 s, completed Mar 31, 2015 11:55:14 AM 

build.sbt

name := "Cassandra" 

version := "1.0" 

scalaVersion := "2.10.4" 

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.1" 

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.2.1" 

libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.10" % "1.2.0-rc2" 

有我的代码的问题？

Answer 1

这是当前连接器中有关依赖于流式火花库的已知错误。

https://datastax-oss.atlassian.net/browse/SPARKC-113

如果这是阻塞，你需要一种解决方法，你可以只包括火花流库作为构建文件的依赖性。

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.2.1"