我试图在访问Hive表的Cloud 4.2 Enterprise上的BigInsights上运行pyspark脚本。Spark Hive报告pyspark.sql.utils.AnalysisException:u'Table not found:XXX'在纱线群集上运行时
首先,我创建的蜂巢表:
[[email protected] ~]$ hive
hive> CREATE TABLE pokes (foo INT, bar STRING);
OK
Time taken: 2.147 seconds
hive> LOAD DATA LOCAL INPATH '/usr/iop/4.2.0.0/hive/doc/examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
Loading data to table default.pokes
Table default.pokes stats: [numFiles=1, numRows=0, totalSize=5812, rawDataSize=0]
OK
Time taken: 0.49 seconds
hive>
然后,我创建一个简单的pyspark脚本:
[[email protected] ~]$ cat test_pokes.py
from pyspark import SparkContext
sc = SparkContext()
from pyspark.sql import HiveContext
hc = HiveContext(sc)
pokesRdd = hc.sql('select * from pokes')
print(pokesRdd.collect())
我尝试与执行:
[[email protected] ~]$ spark-submit \
--master yarn-cluster \
--deploy-mode cluster \
--jars /usr/iop/4.2.0.0/hive/lib/datanucleus-api-jdo-3.2.6.jar, \
/usr/iop/4.2.0.0/hive/lib/datanucleus-core-3.2.10.jar, \
/usr/iop/4.2.0.0/hive/lib/datanucleus-rdbms-3.2.9.jar \
test_pokes.py
不过,我遇到错误:
Traceback (most recent call last):
File "test_pokes.py", line 8, in <module>
pokesRdd = hc.sql('select * from pokes')
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/pyspark.zip/pyspark/sql/context.py", line 580, in sql
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
File "/disk6/local/usercache/biadmin/appcache/application_1477084339086_0481/container_e09_1477084339086_0481_01_000001/pyspark.zip/pyspark/sql/utils.py", line 51, in deco
pyspark.sql.utils.AnalysisException: u'Table not found: pokes; line 1 pos 14'
End of LogType:stdout
如果我运行火花提交独立的,我可以看到表中存在确定:
[[email protected] ~]$ spark-submit test_pokes.py
…
…
16/12/21 13:09:13 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 18962 bytes result sent to driver
16/12/21 13:09:13 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 168 ms on localhost (1/1)
16/12/21 13:09:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/12/21 13:09:13 INFO DAGScheduler: ResultStage 0 (collect at /home/biadmin/test_pokes.py:9) finished in 0.179 s
16/12/21 13:09:13 INFO DAGScheduler: Job 0 finished: collect at /home/biadmin/test_pokes.py:9, took 0.236558 s
[Row(foo=238, bar=u'val_238'), Row(foo=86, bar=u'val_86'), Row(foo=311, bar=u'val_311')
…
…
见与此相关的问题我刚才的问题:hive spark yarn-cluster job fails with: "ClassNotFoundException: org.datanucleus.api.jdo.JDOPersistenceManagerFactory"
这个问题是类似于此的其他问题: Spark can access Hive table from pyspark but not from spark-submit。但是,不像那个问题我使用HiveContext。
更新:在这里看到最终的解决方案https://stackoverflow.com/a/41272260/1033422
这并不能解释为什么它在独立模式下工作 –
这让我更进一步。我现在收到错误:'MetaException(message:Failed to instance listener named:com.ibm.biginsights.bigsql.sync.BIEventListener,reason:java.lang.ClassNotFoundException:com.ibm.biginsights.bigsql.sync.BIEventListener)' –
对不起,我应该解释一下。如果您独立运行,则驱动程序会在机器上运行。因此它会从本地类路径中选取'hive-site.xml'。如果你运行在'cluster-mode'上,这个xml文件不会被传输到集群上的容器,所以你必须手工指定它,Spark会把它放到你的类路径中。 –