如何对配置单元进行基准测试(azure交互式查询hdi 4.0)

wtlkbnrh  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(382)

到2020年,是否有人为使用Hadoop3.x+的azure interactive QueryHDI4.0集群制定并测试了TPCDS或tpc-h基准测试?
我用的是https://github.com/hortonworks/hive-testbench 但是我在为tpc-h和tpc-ds生成数据时遇到了一个错误。
交互式查询hdi 4.0(hadoop 3.1.1)。这个错误可能是什么?失败的步骤是当它运行jar文件时。

Generating data at scale factor 100.
Exception in thread "main" java.lang.IllegalAccessError: 
class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
...
ls: `/tmp/tpch-generate/100/lineitem': No such file or directory
Data generation failed, exiting.

然后第二个问题是,对于tpcds,每当我运行“大”规模因子时,故障就出现在优化步骤中。在表17或表18中,它通常不起作用。你知道这是什么吗?

INFO  : Loading data to table tpcds_bin_partitioned_orc_100.store_sales partition (ss_sold_date_sk=null) from wasb://asdasd-2020-04-16t02-32-03-034z@asdasd.blob.core.windows.net/hive/warehouse/managed/tpcds_bin_partitioned_orc_100.db/store_sales/.hive-staging_hive_2020-04-16_06-47-19_242_1371829803314907581-47/-ext-10000
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Exception updating metastore for acid table tpcds_bin_partitioned_orc_100.store_sales with partitions [store_sales
...
INFO  : Completed executing command(queryId=hive_20200416064719_4aa11ffb-31c0-411f-a7ca-954c9741891d); Time taken: 1280.036 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Exception updating metastore for acid table tpcds_bin_partitioned_orc_100.store_sales with partitions
tcbh2hod

tcbh2hod1#

movetask错误是由于内部sql数据库限制造成的。在azuresql数据库中,传入参数只能有2100个参数,基准测试生成的分区太多。

hive.direct.sql.max.elements.values.clause=200
hive.direct.sql.max.elements.in.clause=200

相关问题