如何激发提交到ZooKeeper管理的Mesos集群（给出java.net.UnknownHostException：zk for mesos：// zk：// master URL）？

Question

我正在运行Spark 2.0.2和Mesos 0.28.2。

我试图提出申请，星火，使用的ZooKeeper管理Mesos集群作为主：

$SPARK_HOME/bin/spark-submit --verbose \ 
--conf spark.mesos.executor.docker.image=$DOCKER_IMAGE \ 
--conf spark.mesos.executor.home=$SPARK_HOME \ 
--conf spark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \ 
--deploy-mode cluster \ 
--master mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos \ 
--class $APP_MAIN_CLASS \ 
file://$APP_JAR_PATH 

（<ip 1>，<ip 2>，并且<ip 3>都在10.0.0.0/8 IPv4地址块）

按照documentation，我似乎对主正确的格式：

大师网址˚F或Mesos的格式为mesos：// host：单主Master Mesos集群的主机：5050或多主Mesos集群的Mesos：// zk：// host1：2181，host2：2181，host3：2181/mesos使用ZooKeeper。

然而，似乎火花读mesos://zk://...字符串，然后尝试连接到zk：

17/04/07 20:10:06 INFO RestSubmissionClient: Submitting a request to launch an application in mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos. 
Exception in thread "main" java.net.UnknownHostException: zk 
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) 
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
    at java.net.Socket.connect(Socket.java:589) 
    at java.net.Socket.connect(Socket.java:538) 
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) 
    at sun.net.www.http.HttpClient.New(HttpClient.java:308) 
    at sun.net.www.http.HttpClient.New(HttpClient.java:326) 
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) 
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) 
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) 
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) 
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) 
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.org$apache$spark$deploy$rest$RestSubmissionClient$$postJson(RestSubmissionClient.scala:214) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:89) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:85) 
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) 
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) 
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) 
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.createSubmission(RestSubmissionClient.scala:85) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$.run(RestSubmissionClient.scala:417) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$.main(RestSubmissionClient.scala:430) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.main(RestSubmissionClient.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

如何获得星火认识到，应该使用三个ZooKeeper的节点，而不是试图连接到不存在的zk主机？

Answer 1

TL;博士它不会工作，除非你要么改变--deploy-mode到client或使用主URL与单个Mesos主机，例如mesos://host:port。

以下行提供了提示在何处查找相关代码。

17/04/07 20:10:06 INFO RestSubmissionClient：在mesos：// zk：//：2181，：2181，：2181/mesos中提交启动应用程序的请求。

看起来这条消息只是用Spark Standalone和Apache Mesos打印出来的--deploy-mode cluster。将其更改为默认client，并且部署路径将更改并希望接受主URL。

查看自己负责群集部署的代码 - RestSubmissionClient。

HereRestSubmissionClient说：

private val supportedMasterPrefixes = Seq("spark://", "mesos://") 

这证明mesos://网址覆盖，但here你看到下列情况：

private val masters: Array[String] = if (master.startsWith("spark://")) { 
    Utils.parseStandaloneMasterUrls(master) 
} else { 
    Array(master) 
} 

即打印出来here与上述INFO消息，显示URL只能是单个Mesos主人。