分区在Hive2.3.0中不起作用

monwx1rj  于 2021-06-26  发布在  Hive
关注(0)|答案(1)|浏览(384)

我创建了如下表:

  1. create table emp (
  2. > eid int,
  3. > fname string,
  4. > lname string,
  5. > salary double,
  6. > city string,
  7. > dept string )
  8. > row format delimited fields terminated by ',';

然后,为了启用分区,我设置了以下属性:

  1. set hive.exec.dynamic.partition=true;
  2. set hive.exec.dynamic.partition.mode=nonstrict;

我创建了如下分区表:

  1. create table part_emp (
  2. > eid int,
  3. > fname string,
  4. > lname string,
  5. > salary double,
  6. > dept string )
  7. > partitioned by ( city string )
  8. > row format delimited fields terminated by ',';

在创建表之后,我发出了insert查询

  1. insert into table part_emp partition(city)
  2. select eid,fname,lname,salary,dept,city from emp;

但它不起作用。。

  1. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  2. Query ID = max_20180311015337_5a67813d-dcc5-46c0-ac4b-a54c11ffb912
  3. Total jobs = 3
  4. Launching Job 1 out of 3
  5. Number of reduce tasks is set to 0 since there's no reduce operator
  6. Starting Job = job_1520757649534_0004, Tracking URL = http://ubuntu:8088/proxy/application_1520757649534_0004/
  7. Kill Command = /home/max/bigdata/hadoop-3.0.0/bin/hadoop job -kill job_1520757649534_0004
  8. Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
  9. 2018-03-11 01:53:44,996 Stage-1 map = 0%, reduce = 0%
  10. Ended Job = job_1520757649534_0004 with errors
  11. Error during job, obtaining debugging information...
  12. FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
  13. MapReduce Jobs Launched:
  14. Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL
  15. Total MapReduce CPU Time Spent: 0 msec

在Hive1.x上同样成功

uubf1zoe

uubf1zoe1#

我也有同样的问题 set hive.exec.max.dynamic.partitions.pernode=1000; (默认值100)解决了我的问题。你可以试试。
ps:此设置意味着:允许在每个mapper/reducer节点中创建的最大动态分区数。

相关问题