我试图连接到IBM的星火作为Bluemix从RStudio我的台式机上运行的运行的服务。无法解析主网址:“spark.bluemix.net”
我抄config.yml
从IBM的数据科学实验运行自动配置RStudio环境:
default:
method: "shell"
CS-DSX:
method: "bluemix"
spark.master: "spark.bluemix.net"
spark.instance.id: "myinstanceid"
tenant.id: "mytenantid"
tenant.secret: "mytenantsecret"
hsui.url: "https://cdsx.ng.bluemix.net"
我试图连接,像这样:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2") # installed spark to '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE, config = "CSX-DSX")
sc <- spark_connect(spark_home = spark_home, config = config)
错误:
17/03/07 09:36:19 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'spark.bluemix.net'
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2735)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:522)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
stackover上还有其他一些问题流类似的错误消息,但它们不尝试连接到在Bluemix上运行的Spark服务。
更新1
我已经改变了我config.yml看起来像这样:
default:
method: "bluemix"
spark.master: "spark://spark.bluemix.net:7070"
spark.instance.id: "7a4089bf-3594-4fdf-8dd1-7e9fd7607be5"
tenant.id: "sdd1-7e9fd7607be53e-39ca506ba762"
tenant.secret: "6146a713-949f-4d4e-84c3-9913d2165b9e"
hsui.url: "https://cdsx.ng.bluemix.net"
...和我的连接代码看起来像这样:
install.packages("sparklyr")
library(sparklyr)
spark_install(version = "1.6.2")
spark_home = '~/Library/Caches/spark/spark-1.6.2-bin-hadoop2.6'
config = spark_config(file = "./config.yml", use_default = FALSE)
sc <- spark_connect(spark_home = spark_home, config = config)
但是,错误是现在:
Error in force(code) :
Failed during initialize_connection: java.lang.NullPointerException
at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sparklyr.Invoke$.invoke(invoke.scala:94)
...
与URL同样的错误。我猜它也需要一个端口。不知道应该是什么。 –
根据这个答案,架构应该是火花:而不是http:http://stackoverflow.com/a/27250746/5629418 –