无法使用pyspark连接到EC2的Spark群集

我按照Spark的网站上的说明进行操作，我在亚马逊上运行了1个主站和1个从站。但是，我无法使用pyspark连接到主节点。我可以使用SSH连接到主节点，而不会出现任何问题。我的命令 spark-ec2 --key-pair =东-1A推出图形集群无法使用pyspark连接到EC2的Spark群集

我可以去http://ec2-54-152-xx-xxx.compute-1.amazonaws.com:8080/

，看到火花运行起来我也看到这个星火法师在

spark://ec2-54-152-xx-xxx.compute-1.amazonaws.com:7077

然而，当我运行命令

MASTER=spark://ec2-54-152-xx-xx.compute-1.amazonaws.com:7077 pyspark

我得到这个错误

2015-09-16 15:39:31,800 ERROR actor.OneForOneStrategy (Slf4jLogger.scala:apply$mcV$sp(66)) - 
java.lang.NullPointerException 
    at org.apache.spark.deploy.client.AppClient$ClientActor$$anonfun$receiveWithLogging$1.applyOrElse(AppClient.scala:160) 
    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) 
    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) 
    at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) 
    at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:59) 
    at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42) 
    at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) 
    at org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42) 
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465) 
    at org.apache.spark.deploy.client.AppClient$ClientActor.aroundReceive(AppClient.scala:61) 
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) 
    at akka.actor.ActorCell.invoke(ActorCell.scala:487) 
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) 
    at akka.dispatch.Mailbox.run(Mailbox.scala:220) 
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) 
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 
2015-09-16 15:39:31,804 INFO client.AppClient$ClientActor (Logging.scala:logInfo(59)) - Connecting to master akka.tcp://[email protected]:7077/user/Master... 
2015-09-16 15:39:31,955 INFO util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52333. 
2015-09-16 15:39:31,956 INFO netty.NettyBlockTransferService (Logging.scala:logInfo(59)) - Server created on 52333 
2015-09-16 15:39:31,959 INFO storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Trying to register BlockManager 
2015-09-16 15:39:31,964 INFO storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(59)) - Registering block manager xxx:52333 with 265.1 MB RAM, BlockManagerId(driver, xxx, 52333) 
2015-09-16 15:39:31,969 INFO storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Registered BlockManager 
2015-09-16 15:39:32,458 ERROR spark.SparkContext (Logging.scala:logError(96)) - Error initializing SparkContext. 
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext 
    at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:103) 
    at org.apache.spark.SparkContext.getSchedulingMode(SparkContext.scala:1503) 
    at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:2007) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:543) 
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) 
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) 
    at py4j.Gateway.invoke(Gateway.java:214) 
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) 
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) 
    at py4j.GatewayConnection.run(GatewayConnection.java:207) 
    at java.lang.Thread.run(Thread.java:745) 
2015-09-16 15:39:32,460 INFO spark.SparkContext (Logging.scala:logInfo(59)) - SparkContext already stopped. 
Traceback (most recent call last): 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/pyspark/shell.py", line 43, in <module> 
    sc = SparkContext(appName="PySparkShell", pyFiles=add_files) 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/pyspark/context.py", line 113, in __init__ 
    conf, jsc, profiler_cls) 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/pyspark/context.py", line 165, in _do_init 
    self._jsc = jsc or self._initialize_context(self._conf._jconf) 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/pyspark/context.py", line 219, in _initialize_context 
    return self._jvm.JavaSparkContext(jconf) 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 701, in __call__ 
    File "/usr/local/Cellar/apache-spark/1.4.1/libexec/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value 
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. 
: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext 
    at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:103) 
    at org.apache.spark.SparkContext.getSchedulingMode(SparkContext.scala:1503) 
    at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:2007) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:543) 
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) 
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) 
    at py4j.Gateway.invoke(Gateway.java:214) 
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) 
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) 
    at py4j.GatewayConnection.run(GatewayConnection.java:207) 
    at java.lang.Thread.run(Thread.java:745)

来源

2015-09-16 toy

Spark_ec2不会不打开端口7077的群集之外的传入连接主节点上。

您可以检查AWS控制台/ EC2 /网络&安全/安全组和检查图形集群主控安全组的入站选项卡。

您可以添加规则以打开端口7077.

入站连接，但它建议在EC2集群运行从主计算机pyspark（主要是星火的App驱动程序），并避免网络之外运行的驱动程序。造成这种情况的原因 - 增加延迟和防火墙连接设置问题 - 您需要打开一些端口，以便执行程序可以连接到计算机上的驱动程序。

所以要走的路是登录到SSH集群与此命令：

spark-ec2 --key-pair=graph-cluster --identity-file=/Users/.ssh/pem.pem --region=us-east-1 --zone=us-east-1a login graph-cluster

并运行主服务器的命令：

cd spark 
bin/pyspark

你需要转移相关文件（你的脚本和数据）掌握。我通常在S3上保存数据并用vim编辑脚本文件或者启动ipython笔记本。

顺便说一句，后者非常简单 - 您需要将传入连接的规则从计算机IP添加到EC2控制台主服务器安全组中的端口18888。然后在集群上运行命令：

IPYTHON_OPTS = “笔记本--pylab直列--port = 18888 --ip = '*'” pyspark

然后你就可以用http://ec2-54-152-xx-xxx.compute-1.amazonaws.com:18888/

访问

来源

2015-09-16 20:16:42 vvladymyrov

无法使用pyspark连接到EC2的Spark群集

回答

相关问题