hadoop:分割元数据大小超过10000000

y3bcpkx1  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(510)

运行级联作业时,出现错误:

Split metadata size exceeded 10000000

我试图通过将以下内容传递给命令行来增加每个职务级别的限制

xxx.jar -D mapreduce.job​.split.metainfo.maxsi‌​ze=30000000

我也试过了

xxx.jar -D mapreduce.jobtracker​.split.metainfo.maxsi‌​ze=30000000

但这两个都不起作用,我仍然得到相同的错误,所以没有选择参数。我正在使用hadoop2.5。谁能指出我做错了什么?

lrl1mhuk

lrl1mhuk1#

你能试着在中设置下列属性吗 conf/mapred-site.xml :

<!-- No limits if set to -1 -->
<property>
    <name>mapreduce.jobtracker.split.metainfo.maxsize</name>
    <value>-1</value>
</property>

不确定以下内容是否有用(试一试)

xxx.jar -D mapreduce.jobtracker​.split.metainfo.maxsi‌​ze=-1

参考文献:https://archive.cloudera.com/cdh/3/hadoop/mapred-default.html

| Name                                        | Default Value | Description                                                                                                                                                                                                                   |
|---------------------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| mapred.jobtracker.job.history.block.size    | 3145728       | The block size of the job history file. Since the job recovery uses job,history, its important to dump job history to disk as soon as possible.,Note that this is an expert level parameter. The default value is set to,3 MB |
| mapreduce.jobtracker.split.metainfo.maxsize | 10000000      | The maximum permissible size of the split metainfo file. The JobTracker,won't attempt to read split metainfo files bigger than the configured,value. No limits if set to -1.                                                  |

相关问题