cloudera hadoop mapreduce作业gc开销限制超出错误

6tr1vspr  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(336)

我正在clouderacdh4上运行canopy集群作业（使用mahout）。要聚集的内容有大约1m条记录（每条记录的大小小于1k）。整个hadoop环境（包括所有节点）都运行在一个具有4g内存的vm中。默认情况下安装cdh4。运行作业时出现以下异常。
根据异常情况，作业客户端似乎需要更高的jvm堆大小。然而，在cloudera管理器中，jvm堆大小有很多配置选项。我将“客户端java堆大小（字节）”从256mib更改为512mib。然而，情况并没有改善。
关于设置这些堆大小选项有什么提示/提示吗？

13/07/03 17:12:45 INFO input.FileInputFormat: Total input paths to process : 1
13/07/03 17:12:46 INFO mapred.JobClient: Running job: job_201307031710_0001
13/07/03 17:12:47 INFO mapred.JobClient:  map 0% reduce 0%
13/07/03 17:13:06 INFO mapred.JobClient:  map 1% reduce 0%
13/07/03 17:13:27 INFO mapred.JobClient:  map 2% reduce 0%
13/07/03 17:14:01 INFO mapred.JobClient:  map 3% reduce 0%
13/07/03 17:14:50 INFO mapred.JobClient:  map 4% reduce 0%
13/07/03 17:15:50 INFO mapred.JobClient:  map 5% reduce 0%
13/07/03 17:17:06 INFO mapred.JobClient:  map 6% reduce 0%
13/07/03 17:18:44 INFO mapred.JobClient:  map 7% reduce 0%
13/07/03 17:20:24 INFO mapred.JobClient:  map 8% reduce 0%
13/07/03 17:22:20 INFO mapred.JobClient:  map 9% reduce 0%
13/07/03 17:25:00 INFO mapred.JobClient:  map 10% reduce 0%
13/07/03 17:28:08 INFO mapred.JobClient:  map 11% reduce 0%
13/07/03 17:31:46 INFO mapred.JobClient:  map 12% reduce 0%
13/07/03 17:35:57 INFO mapred.JobClient:  map 13% reduce 0%
13/07/03 17:40:52 INFO mapred.JobClient:  map 14% reduce 0%
13/07/03 17:46:55 INFO mapred.JobClient:  map 15% reduce 0%
13/07/03 17:55:02 INFO mapred.JobClient:  map 16% reduce 0%
13/07/03 18:08:42 INFO mapred.JobClient:  map 17% reduce 0%
13/07/03 18:59:11 INFO mapred.JobClient:  map 8% reduce 0%
13/07/03 18:59:13 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000001_0, Status : FAILED
Error: GC overhead limit exceeded
13/07/03 18:59:23 INFO mapred.JobClient:  map 9% reduce 0%
13/07/03 19:00:09 INFO mapred.JobClient:  map 10% reduce 0%
13/07/03 19:01:49 INFO mapred.JobClient:  map 11% reduce 0%
13/07/03 19:04:25 INFO mapred.JobClient:  map 12% reduce 0%
13/07/03 19:07:48 INFO mapred.JobClient:  map 13% reduce 0%
13/07/03 19:12:48 INFO mapred.JobClient:  map 14% reduce 0%
13/07/03 19:19:46 INFO mapred.JobClient:  map 15% reduce 0%
13/07/03 19:29:05 INFO mapred.JobClient:  map 16% reduce 0%
13/07/03 19:43:43 INFO mapred.JobClient:  map 17% reduce 0%
13/07/03 20:49:36 INFO mapred.JobClient:  map 8% reduce 0%
13/07/03 20:49:38 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000001_1, Status : FAILED
Error: GC overhead limit exceeded
13/07/03 20:49:48 INFO mapred.JobClient:  map 9% reduce 0%
13/07/03 20:50:31 INFO mapred.JobClient:  map 10% reduce 0%
13/07/03 20:52:08 INFO mapred.JobClient:  map 11% reduce 0%
13/07/03 20:54:38 INFO mapred.JobClient:  map 12% reduce 0%
13/07/03 20:58:01 INFO mapred.JobClient:  map 13% reduce 0%
13/07/03 21:03:01 INFO mapred.JobClient:  map 14% reduce 0%
13/07/03 21:10:10 INFO mapred.JobClient:  map 15% reduce 0%
13/07/03 21:19:54 INFO mapred.JobClient:  map 16% reduce 0%
13/07/03 21:31:35 INFO mapred.JobClient:  map 8% reduce 0%
13/07/03 21:31:37 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000000_0, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 65.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

13/07/03 21:32:09 INFO mapred.JobClient:  map 9% reduce 0%
13/07/03 21:33:31 INFO mapred.JobClient:  map 10% reduce 0%
13/07/03 21:35:42 INFO mapred.JobClient:  map 11% reduce 0%
13/07/03 21:38:41 INFO mapred.JobClient:  map 12% reduce 0%
13/07/03 21:42:27 INFO mapred.JobClient:  map 13% reduce 0%
13/07/03 21:48:20 INFO mapred.JobClient:  map 14% reduce 0%
13/07/03 21:56:12 INFO mapred.JobClient:  map 15% reduce 0%
13/07/03 22:07:20 INFO mapred.JobClient:  map 16% reduce 0%
13/07/03 22:26:36 INFO mapred.JobClient:  map 17% reduce 0%
13/07/03 23:35:30 INFO mapred.JobClient:  map 8% reduce 0%
13/07/03 23:35:32 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000000_1, Status : FAILED
Error: GC overhead limit exceeded
13/07/03 23:35:42 INFO mapred.JobClient:  map 9% reduce 0%
13/07/03 23:36:16 INFO mapred.JobClient:  map 10% reduce 0%
13/07/03 23:38:01 INFO mapred.JobClient:  map 11% reduce 0%
13/07/03 23:40:47 INFO mapred.JobClient:  map 12% reduce 0%
13/07/03 23:44:44 INFO mapred.JobClient:  map 13% reduce 0%
13/07/03 23:50:42 INFO mapred.JobClient:  map 14% reduce 0%
13/07/03 23:58:58 INFO mapred.JobClient:  map 15% reduce 0%
13/07/04 00:10:22 INFO mapred.JobClient:  map 16% reduce 0%
13/07/04 00:21:38 INFO mapred.JobClient:  map 7% reduce 0%
13/07/04 00:21:40 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000001_2, Status : FAILED
java.lang.Throwable: Child Error
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 65.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

13/07/04 00:21:50 INFO mapred.JobClient:  map 8% reduce 0%
13/07/04 00:22:27 INFO mapred.JobClient:  map 9% reduce 0%
13/07/04 00:23:52 INFO mapred.JobClient:  map 10% reduce 0%
13/07/04 00:26:00 INFO mapred.JobClient:  map 11% reduce 0%
13/07/04 00:28:47 INFO mapred.JobClient:  map 12% reduce 0%
13/07/04 00:32:17 INFO mapred.JobClient:  map 13% reduce 0%
13/07/04 00:37:34 INFO mapred.JobClient:  map 14% reduce 0%
13/07/04 00:44:30 INFO mapred.JobClient:  map 15% reduce 0%
13/07/04 00:54:28 INFO mapred.JobClient:  map 16% reduce 0%
13/07/04 01:16:30 INFO mapred.JobClient:  map 17% reduce 0%
13/07/04 01:32:05 INFO mapred.JobClient:  map 8% reduce 0%
13/07/04 01:32:08 INFO mapred.JobClient: Task Id : attempt_201307031710_0001_m_000000_2, Status : FAILED
Error: GC overhead limit exceeded
13/07/04 01:32:21 INFO mapred.JobClient:  map 9% reduce 0%
13/07/04 01:33:26 INFO mapred.JobClient:  map 10% reduce 0%
13/07/04 01:35:37 INFO mapred.JobClient:  map 11% reduce 0%
13/07/04 01:38:48 INFO mapred.JobClient:  map 12% reduce 0%
13/07/04 01:43:06 INFO mapred.JobClient:  map 13% reduce 0%
13/07/04 01:49:58 INFO mapred.JobClient:  map 14% reduce 0%
13/07/04 01:59:07 INFO mapred.JobClient:  map 15% reduce 0%
13/07/04 02:12:00 INFO mapred.JobClient:  map 16% reduce 0%
13/07/04 02:37:56 INFO mapred.JobClient:  map 17% reduce 0%
13/07/04 03:31:55 INFO mapred.JobClient:  map 8% reduce 0%
13/07/04 03:32:00 INFO mapred.JobClient: Job complete: job_201307031710_0001
13/07/04 03:32:00 INFO mapred.JobClient: Counters: 7
13/07/04 03:32:00 INFO mapred.JobClient:   Job Counters 
13/07/04 03:32:00 INFO mapred.JobClient:     Failed map tasks=1
13/07/04 03:32:00 INFO mapred.JobClient:     Launched map tasks=8
13/07/04 03:32:00 INFO mapred.JobClient:     Data-local map tasks=8
13/07/04 03:32:00 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=11443502
13/07/04 03:32:00 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
13/07/04 03:32:00 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/07/04 03:32:00 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
Exception in thread "main" java.lang.RuntimeException: java.lang.InterruptedException: Canopy Job failed processing vector
hadoopmapreduceclouderaheap

2条答案

按热度按时间
dldeef67

dldeef671#

驯兽师的工作是记忆密集型的。我不知道是Map器还是还原器是罪魁祸首，但是，无论哪种方法，你都必须告诉hadoop给它们更多的ram。”gc overhead limit exceeded”只是表示“内存不足”的一种方式——意味着jvm放弃了回收最后0.01%的可用ram的尝试。
如何设置这一点确实有点复杂，因为hadoop2中有几个属性，它们都发生了变化。cdh4可以支持Hadoop1或2—您使用哪一种？
如果非要我猜的话：设定 mapreduce.child.java.opts 至 -Xmx1g . 但正确的答案实际上取决于你的版本和数据。

赞(0）分享  回复(0）举报  2021-06-04
35g0bw71

35g0bw712#

您需要更改hadoop的内存设置，因为为hadoop分配的内存不足以满足您正在运行的作业要求，请尝试增加堆内存并验证，由于内存的过度使用，操作系统可能会终止作业失败所导致的进程。

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com