mapreduce作业将在map=0%、reduce=0%的情况下继续运行数小时

5sxhfpxr  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(670)

我正在运行一个配置单元查询

create table table1 as select split(comments,' ') as words from table2;

comments列具有以空格分隔的字符串形式的审阅注解。
当我运行此查询时,mapreduce作业将启动,并在map 0%的情况下继续运行数小时。在这个过程中它不会给出任何错误。

hive> create table jw_1 as select split(comments,' ') from removed_null_values;
Query ID = xxx-190418201314_7781cf59-6afb-4e82-ab75-c7e343c4985e
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555607912038_0013, Tracking URL = http://xxx-VirtualBox:8088/proxy/application_1555607912038_0013/
Kill Command = /usr/local/bin/hadoop-3.2.0/bin/mapred job  -kill job_1555607912038_0013
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-18 20:13:30,568 Stage-1 map = 0%,  reduce = 0%
2019-04-18 20:14:31,140 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 39.6 sec
2019-04-18 20:15:31,311 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 101.64 sec
2019-04-18 20:16:31,451 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 146.5 sec
2019-04-18 20:17:31,684 Stage-1 map = 0%,  reduce = 0%, Cumulative CPU 212.08 sec

但是当我尝试

select split(comments,' ') from table2;

我可以在shell中看到数组形式的注解。

["\"Lauren","was","promptly","responsive","in","advance","of","our","booking.","providing","a","lot","of","helpful","info.","And","she","stayed","in","contact","and","was","readily","available","prior","to","and","during","our","stay.","which","was","awesome.","The","location.","price","and","privacy","were","the","real","benefits."]

我还运行了其他一些查询,在这些查询中mapreduce作业将完成并生成所需的结果
我目前正在使用Hive3.1.1
基本上,我想用一个包含单词的数组创建一个新表,然后对该列进行标记化
我是新的Hive和我的感情分析工作的数据文件大小35mb。

qnzebej0

qnzebej01#

在第一种情况下,当转换为mapreduce时,您很可能没有完成配置单元查询所需的资源。您必须查看yarn或mr1来确定是否有足够的计算资源来运行mapreduce作业。
在第二个查询中,有些配置单元查询不会触发mapreduce作业,这就是它返回的原因。查看hive如何决定何时使用map reduce以及何时不使用?更多信息。

相关问题