如何spark-submit由IDE编写的hiveContext？

我想部署我的代码，其中包含hiveContext在Spark集群上。如何spark-submit由IDE编写的hiveContext？

./spark-submit --class com.dt.sparkSQL.DataFrameToHive --master spark://SparkMaster:7077 /root/Documents/DataFrameToHive.jar 但现在的问题是

17/08/13 10:29:46 INFO hive.metastore: Trying to connect to metastore with URI thrift://SparkMaster:9083 
17/08/13 10:29:46 WARN hive.metastore: Failed to connect to the MetaStore Server... 
17/08/13 10:29:46 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

当我做了火花外壳

./spark-shell --master spark://SparkMaster:7077

我可以SparkMaster连接：9083 successfully.Here是我的火花/ conf目录/ hive- site.xml

<configuration> 
<property> 
        <name>hive.metastore.uris</name> 
        <value>thrift://SparkMaster:9083</value> 
        <description>thrift URI for the remote metastore.Used by metastore client to connect to remote metastore. </description> 
</property> 
</configuration>

我的问题是为什么它会连接w ith SparkMaster：9083当我进行spark-submit时，SparkMaster：9083有什么问题？这里是IDE上的代码

package com.dt.sparkSQL 

import org.apache.spark.{SparkConf, SparkContext} 
import org.apache.spark.sql.SQLContext 
import org.apache.spark.sql.hive.HiveContext 
object DataFrameToHive { 
    def main(args: Array[String]): Unit = { 
    val conf = new SparkConf() 
    conf.setAppName("DataFrameToHive").setMaster("spark://SparkMaster:7077") 
    val sc = new SparkContext(conf) 
    val hiveContext = new HiveContext(sc) 
    hiveContext.sql("use userdb") 
    hiveContext.sql("DROP TABLE IF EXISTS people") 
    hiveContext.sql("CREATE TABLE IF NOT EXISTS people(name STRING, age INT)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'") 
    hiveContext.sql("LOAD DATA LOCAL INPATH '/root/Documents/people.txt' INTO TABLE people") 
    hiveContext.sql("use userdb") 
    hiveContext.sql("DROP TABLE IF EXISTS peopleScores") 
    hiveContext.sql("CREATE TABLE IF NOT EXISTS peopleScores(name STRING, score INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'") 
    hiveContext.sql("LOAD DATA LOCAL INPATH '/root/Documents/peopleScore.txt' INTO TABLE peopleScores") 
    val resultDF = hiveContext.sql("select pi.name,pi.age,ps.score " 
     +" from people pi join peopleScores ps on pi.name=ps.name" 
     +" where ps.score>90") 
    hiveContext.sql("drop table if exists peopleResult") 
    resultDF.saveAsTable("peopleResult") 
    val dataframeHive = hiveContext.table("peopleResult") 
    dataframeHive.show() 
    } 
} 
`

来源

2017-08-13 Jason Shu

我已经成功地解决了这个问题。部署hiveContext与普通的jar有所不同。

./spark-submit --class com.dt.sparkSQL.DataFrameToHive --files /usr/local/hive/apache-hive-1.2.1-bin/conf/hive-site.xml --master spark://SparkMaster:7077 /root/Documents/DataFrameToHive.jar

来源

2017-08-14 01:57:41

如何spark-submit由IDE编写的hiveContext？

回答

相关问题