到2020年,是否有人为使用Hadoop3.x+的azure interactive QueryHDI4.0集群制定并测试了TPCDS或tpc-h基准测试?
我用的是https://github.com/hortonworks/hive-testbench 但是我在为tpc-h和tpc-ds生成数据时遇到了一个错误。
交互式查询hdi 4.0(hadoop 3.1.1)。这个错误可能是什么?失败的步骤是当它运行jar文件时。
Generating data at scale factor 100.
Exception in thread "main" java.lang.IllegalAccessError:
class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
...
ls: `/tmp/tpch-generate/100/lineitem': No such file or directory
Data generation failed, exiting.
然后第二个问题是,对于tpcds,每当我运行“大”规模因子时,故障就出现在优化步骤中。在表17或表18中,它通常不起作用。你知道这是什么吗?
INFO : Loading data to table tpcds_bin_partitioned_orc_100.store_sales partition (ss_sold_date_sk=null) from wasb://asdasd-2020-04-16t02-32-03-034z@asdasd.blob.core.windows.net/hive/warehouse/managed/tpcds_bin_partitioned_orc_100.db/store_sales/.hive-staging_hive_2020-04-16_06-47-19_242_1371829803314907581-47/-ext-10000
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Exception updating metastore for acid table tpcds_bin_partitioned_orc_100.store_sales with partitions [store_sales
...
INFO : Completed executing command(queryId=hive_20200416064719_4aa11ffb-31c0-411f-a7ca-954c9741891d); Time taken: 1280.036 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Exception updating metastore for acid table tpcds_bin_partitioned_orc_100.store_sales with partitions
1条答案
按热度按时间tcbh2hod1#
movetask错误是由于内部sql数据库限制造成的。在azuresql数据库中,传入参数只能有2100个参数,基准测试生成的分区太多。