使用配置单元创建表失败，出现parseexception

y3bcpkx1 于 2021-06-26 发布在 Hive

关注(0)|答案(2)|浏览(440)

尝试使用配置单元从spark在python中创建表失败，出现parseexception

Cannot create hive serde table

在hortonworks hdp 2.6上运行
代码是

warehouse_location = abspath('spark-warehouse') 

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("spark.sql.warehouse.dir", warehouse_location) \
    .enableHiveSupport() \
    .getOrCreate()

# spark is an existing SparkSession

spark.sql("CREATE TABLE IF NOT EXISTS tom (key INT, value STRING) USING hive")

生成的错误为：

INFO SparkSqlParser: Parsing command: CREATE TABLE IF NOT EXISTS tom (key INT, value STRING) USING hive
Traceback (most recent call last):
  File "/usr/repos/dataconnect/model/create_model.py", line 17, in <module>
    spark.sql("CREATE TABLE IF NOT EXISTS tom (key INT, value STRING) USING hive")
  File "/usr/hdp/2.6.1.0-129/spark2/python/lib/pyspark.zip/pyspark/sql/session.py", line 545, in sql
  File "/usr/hdp/2.6.1.0-129/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/hdp/2.6.1.0-129/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 73, in deco
pyspark.sql.utils.ParseException: u'\nCannot create hive serde table with CREATE TABLE USING\n== SQL ==\nCREATE TABLE IF NOT EXISTS tom (key INT, value STRING) USING hive'

Hive apache-spark

来源：https://stackoverflow.com/questions/47533719/spark-python-create-table-using-hive-fails-with-parseexception

2条答案

按热度按时间

hk8txs481#

只需删除“使用Hive”。所以命令变成了- spark.sql("CREATE TABLE IF NOT EXISTS spark_hive_table (key INT, value STRING)") 此命令在配置单元中创建表(我将表名更改为spark\u hive\u table，您可以使用自己的）

赞(0）回复(0）举报 2021-06-26

j13ufse22#

scala> hiveContext.sql("CREATE TABLE IF NOT EXISTS tom (key INT, value STRING) row format delimited fields terminated by ','")

res157: org.apache.spark.sql.DataFrame = []

scala> hiveContext.sql("select * from tom");

res158: org.apache.spark.sql.DataFrame = [key: int, value: string]

scala> hiveContext.sql("select * from tom").show()
+---+-----+
|key|value|
+---+-----+

赞(0）回复(0）举报 2021-06-26

我来回答

使用配置单元创建表失败，出现parseexception

2条答案

相关问题

热门标签

最新问答