无法通过spark sql从支持配置单元事务的表中获取结果

enxuqcxy  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(402)

background:-
我将hdp与spark1.6.0和hive1.2.1一起使用
步骤followed:-
创建配置单元table:-

hive>
CREATE TABLE orctest(PROD_ID bigint, CUST_ID bigint, TIME_ID timestamp, CHANNEL_ID bigint, PROMO_ID bigint, QUANTITY_SOLD decimal(10,0), AMOUNT_SOLD decimal(10,0) ) CLUSTERED BY (PROD_ID) INTO 32 BUCKETS STORED AS ORC TBLPROPERTIES ( "orc.compress"="SNAPPY", "transactional"="true" );

将记录插入测试

hive>
insert into orctest values(1, 1, '2016-08-02 21:36:54.000000000', 1, 1, 10, 10000);

尝试从spark shell访问测试表

scala>
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

val s = hiveContext.table("orctest")*

例外情况thrown:-

16/08/02 22:06:54 INFO OrcRelation: Listing hdfs://hadoop03:8020/apps/hive/warehouse/orctest on driver
16/08/02 22:06:54 
INFO OrcRelation: Listing hdfs://hadoop03:8020/apps/hive/warehouse/orctest/delta_0000005_0000005 on driver

**java.lang.AssertionError: assertion failed**

at scala.Predef$.assert(Predef.scala:165)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:39)
at org.apache.spark.sql.execution.datasources.LogicalRelation$$anonfun$1.apply(LogicalRelation.scala:38)
at scala.Option.map(Option.scala:145)
at org.apache.spark.sql.execution.datasources.LogicalRelation.<init>(LogicalRelation.scala:38)
at org.apache.spark.sql.execution.datasources.LogicalRelation.copy(LogicalRelation.scala:31)
at org.apache.spark.sql.hive.HiveMetastoreCatalog.org$apache$spark$sql$hive$HiveMetastoreCatalog$$convertToOrcRelation(HiveMetastoreCatalog.scala:588)

任何帮助都将不胜感激。

to94eoyn

to94eoyn1#

尝试设置: hiveContext.setConf("spark.sql.hive.convertMetastoreOrc", "false")

相关问题