hadoop—为什么没有order by/sort by子句的配置单元查询会以单个缩减器结束?

42fyovps  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(176)

我有一个与流式处理作业关联的简单查询,其中没有order by语句。

set hive.exec.max.dynamic.partitions.pernode=100;
set hive.exec.max.dynamic.partitions=100;
set hive.exec.max.created.files=100;
set hive.exec.dynamic.partition.mode=nonstrict;
set mapred.reduce.tasks=20;
add file /home/devo/c1166313/pafvalid.py ;
add file /home/devo/c1166313/paf-rules.properties ;
from
 (from  
   (select * from mz_paf_errors_dummy_v) p
select transform (p.*)  row format delimited fields terminated by '|' 
using 'pafvalid.py paf-rules.properties 10'
as (<column list>)
row format delimited fields terminated by '|' )  b
insert overwrite table mytab partition (passfail, batch_sk) select <col list>;

这是一个中等规模的集群(几十台机器),Map器的数量超过2000。为什么需要一个减速机?

Number of reduce tasks determined at compile time: 1

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题