hadoop mapreduce运行非常慢

uxh89sit  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(404)

我使用的是一个4datanode/1namenode hadoop集群，其版本为1.1.2，作为vms安装在xenserver中。我有一个1gb的文本文件，并试图做字数。Map花了2个小时，减速器就挂了。一个普通的perl脚本在10分钟内完成了这项工作。我的设置好像少了点什么。
即使对于kbs中的小文件，也需要花费很少的时间。

  1. hadoop@master ~]$ hadoop jar /usr/share/hadoop/hadoop-examples-1.1.2.jar  wordcount huge out
  2. 13/05/29 10:45:09 INFO input.FileInputFormat: Total input paths to process : 1
  3. 13/05/29 10:45:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  4. 13/05/29 10:45:09 WARN snappy.LoadSnappy: Snappy native library not loaded
  5. 13/05/29 10:45:11 INFO mapred.JobClient: Running job: job_201305290801_0002
  6. 13/05/29 10:45:12 INFO mapred.JobClient:  map 0% reduce 0%
  7. 13/05/29 10:57:14 INFO mapred.JobClient:  map 2% reduce 0%
  8. 13/05/29 10:58:01 INFO mapred.JobClient:  map 3% reduce 0%
  9. 13/05/29 10:58:53 INFO mapred.JobClient:  map 4% reduce 0%
  10. 13/05/29 10:58:54 INFO mapred.JobClient:  map 5% reduce 0%
  11. 13/05/29 10:59:33 INFO mapred.JobClient:  map 6% reduce 0%
  12. 13/05/29 11:01:52 INFO mapred.JobClient:  map 7% reduce 0%
  13. 13/05/29 11:03:02 INFO mapred.JobClient:  map 8% reduce 0%
  14. 13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED
  15. Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing!
  16. 13/05/29 11:03:28 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000003_0, Status : FAILED
  17. Task attempt_201305290801_0002_m_000003_0 failed to report status for 604 seconds. Killing!
  18. 13/05/29 11:03:29 INFO mapred.JobClient:  map 9% reduce 0%
  19. 13/05/29 11:04:07 INFO mapred.JobClient:  map 10% reduce 0%
  20. 13/05/29 11:05:13 INFO mapred.JobClient:  map 11% reduce 0%
  21. 13/05/29 11:06:34 INFO mapred.JobClient:  map 12% reduce 0%
  22. 13/05/29 11:06:59 INFO mapred.JobClient:  map 13% reduce 0%
  23. 13/05/29 11:08:14 INFO mapred.JobClient:  map 14% reduce 0%
  24. 13/05/29 11:08:39 INFO mapred.JobClient:  map 15% reduce 0%
  25. 13/05/29 11:09:35 INFO mapred.JobClient:  map 16% reduce 0%
  26. 13/05/29 11:10:03 INFO mapred.JobClient:  map 17% reduce 0%
  27. 13/05/29 11:10:55 INFO mapred.JobClient:  map 18% reduce 0%
  28. 13/05/29 11:11:47 INFO mapred.JobClient:  map 19% reduce 0%
  29. 13/05/29 11:14:05 INFO mapred.JobClient:  map 20% reduce 0%
  30. 13/05/29 11:15:22 INFO mapred.JobClient:  map 21% reduce 0%
  31. 13/05/29 11:15:49 INFO mapred.JobClient:  map 22% reduce 0%
  32. 13/05/29 11:17:09 INFO mapred.JobClient:  map 23% reduce 0%
  33. 13/05/29 11:18:06 INFO mapred.JobClient:  map 24% reduce 0%
  34. 13/05/29 11:18:29 INFO mapred.JobClient:  map 25% reduce 0%
  35. 13/05/29 11:18:53 INFO mapred.JobClient:  map 26% reduce 0%
  36. 13/05/29 11:20:05 INFO mapred.JobClient:  map 27% reduce 0%
  37. 13/05/29 11:21:09 INFO mapred.JobClient:  map 28% reduce 0%
  38. 13/05/29 11:21:45 INFO mapred.JobClient:  map 29% reduce 0%
  39. 13/05/29 11:22:14 INFO mapred.JobClient:  map 30% reduce 0%
  40. 13/05/29 11:22:31 INFO mapred.JobClient:  map 31% reduce 0%
  41. 13/05/29 11:22:32 INFO mapred.JobClient:  map 32% reduce 0%
  42. 13/05/29 11:23:01 INFO mapred.JobClient:  map 33% reduce 0%
  43. 13/05/29 11:23:41 INFO mapred.JobClient:  map 34% reduce 0%
  44. 13/05/29 11:24:29 INFO mapred.JobClient:  map 35% reduce 0%
  45. 13/05/29 11:25:16 INFO mapred.JobClient:  map 36% reduce 0%
  46. 13/05/29 11:25:58 INFO mapred.JobClient:  map 37% reduce 0%
  47. 13/05/29 11:27:09 INFO mapred.JobClient:  map 38% reduce 0%
  48. 13/05/29 11:27:55 INFO mapred.JobClient:  map 39% reduce 0%
  49. 13/05/29 11:28:33 INFO mapred.JobClient:  map 40% reduce 0%
  50. 13/05/29 11:29:50 INFO mapred.JobClient:  map 41% reduce 0%
  51. 13/05/29 11:30:29 INFO mapred.JobClient:  map 42% reduce 0%
  52. 13/05/29 11:31:37 INFO mapred.JobClient:  map 43% reduce 0%
  53. 13/05/29 11:32:10 INFO mapred.JobClient:  map 44% reduce 0%
  54. 13/05/29 11:32:34 INFO mapred.JobClient:  map 45% reduce 0%
  55. 13/05/29 11:34:08 INFO mapred.JobClient:  map 46% reduce 0%
  56. 13/05/29 11:36:01 INFO mapred.JobClient:  map 47% reduce 0%
  57. 13/05/29 11:36:57 INFO mapred.JobClient:  map 48% reduce 0%
  58. 13/05/29 11:37:53 INFO mapred.JobClient:  map 49% reduce 0%
  59. 13/05/29 11:39:50 INFO mapred.JobClient:  map 50% reduce 0%
  60. 13/05/29 11:42:17 INFO mapred.JobClient:  map 51% reduce 0%
  61. 13/05/29 11:43:26 INFO mapred.JobClient:  map 52% reduce 0%
  62. 13/05/29 11:47:55 INFO mapred.JobClient:  map 53% reduce 0%
  63. 13/05/29 11:48:25 INFO mapred.JobClient:  map 54% reduce 0%
  64. 13/05/29 11:49:28 INFO mapred.JobClient:  map 54% reduce 2%
  65. 13/05/29 11:49:31 INFO mapred.JobClient:  map 54% reduce 4%
  66. 13/05/29 11:50:03 INFO mapred.JobClient:  map 55% reduce 4%
  67. 13/05/29 11:50:49 INFO mapred.JobClient:  map 56% reduce 4%
  68. 13/05/29 11:50:54 INFO mapred.JobClient:  map 58% reduce 4%
  69. 13/05/29 11:51:21 INFO mapred.JobClient:  map 59% reduce 4%
  70. 13/05/29 11:51:46 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_1, Status : FAILED
  71. Task attempt_201305290801_0002_m_000002_1 failed to report status for 685 seconds. Killing!
  72. 13/05/29 11:52:09 INFO mapred.JobClient:  map 61% reduce 4%
  73. 13/05/29 11:52:27 INFO mapred.JobClient:  map 62% reduce 4%
  74. 13/05/29 11:52:53 INFO mapred.JobClient:  map 63% reduce 4%
  75. 13/05/29 11:53:36 INFO mapred.JobClient:  map 64% reduce 4%
  76. 13/05/29 11:53:57 INFO mapred.JobClient:  map 65% reduce 4%
  77. 13/05/29 11:54:41 INFO mapred.JobClient:  map 66% reduce 4%
  78. 13/05/29 11:55:51 INFO mapred.JobClient:  map 67% reduce 4%
  79. 13/05/29 11:57:00 INFO mapred.JobClient:  map 68% reduce 4%
  80. 13/05/29 11:57:04 INFO mapred.JobClient:  map 69% reduce 4%
  81. 13/05/29 11:57:11 INFO mapred.JobClient:  map 70% reduce 4%
  82. 13/05/29 11:57:41 INFO mapred.JobClient:  map 71% reduce 4%
  83. 13/05/29 11:58:13 INFO mapred.JobClient:  map 72% reduce 4%
  84. 13/05/29 11:58:45 INFO mapred.JobClient:  map 73% reduce 4%
  85. 13/05/29 11:59:05 INFO mapred.JobClient:  map 74% reduce 4%
  86. 13/05/29 11:59:08 INFO mapred.JobClient:  map 74% reduce 6%
  87. 13/05/29 11:59:42 INFO mapred.JobClient:  map 75% reduce 6%
  88. 13/05/29 11:59:52 INFO mapred.JobClient:  map 76% reduce 6%
  89. 13/05/29 12:00:33 INFO mapred.JobClient:  map 77% reduce 6%
  90. 13/05/29 12:00:53 INFO mapred.JobClient:  map 78% reduce 6%
  91. 13/05/29 12:01:06 INFO mapred.JobClient:  map 79% reduce 6%
  92. 13/05/29 12:01:51 INFO mapred.JobClient:  map 80% reduce 6%
  93. 13/05/29 12:02:29 INFO mapred.JobClient:  map 81% reduce 6%
  94. 13/05/29 12:02:39 INFO mapred.JobClient:  map 82% reduce 6%
  95. 13/05/29 12:02:56 INFO mapred.JobClient:  map 83% reduce 6%
  96. 13/05/29 12:03:36 INFO mapred.JobClient:  map 84% reduce 6%
  97. 13/05/29 12:04:05 INFO mapred.JobClient:  map 85% reduce 6%
  98. 13/05/29 12:04:59 INFO mapred.JobClient:  map 86% reduce 6%
  99. 13/05/29 12:05:47 INFO mapred.JobClient:  map 87% reduce 6%
  100. 13/05/29 12:07:04 INFO mapred.JobClient:  map 88% reduce 6%
  101. 13/05/29 12:08:00 INFO mapred.JobClient:  map 89% reduce 6%
  102. 13/05/29 12:08:32 INFO mapred.JobClient:  map 90% reduce 6%
  103. 13/05/29 12:09:41 INFO mapred.JobClient:  map 91% reduce 6%
  104. 13/05/29 12:10:04 INFO mapred.JobClient:  map 92% reduce 6%
  105. 13/05/29 12:10:17 INFO mapred.JobClient:  map 93% reduce 6%
  106. 13/05/29 12:10:45 INFO mapred.JobClient:  map 94% reduce 6%
  107. 13/05/29 12:10:49 INFO mapred.JobClient:  map 95% reduce 6%
  108. 13/05/29 12:11:00 INFO mapred.JobClient:  map 96% reduce 6%
  109. 13/05/29 12:11:03 INFO mapred.JobClient:  map 97% reduce 6%
  110. 13/05/29 12:11:12 INFO mapred.JobClient:  map 98% reduce 6%
  111. 13/05/29 12:11:17 INFO mapred.JobClient:  map 99% reduce 6%
  112. 13/05/29 12:12:02 INFO mapred.JobClient:  map 100% reduce 6%
  114. ^C[hadoop@master ~]$
hadoopmapreduceword-count

1条答案

按热度按时间
bvpmtnay

bvpmtnay1#

从您提供的有限信息（控制台输出）来看，集群似乎并不健康。

  1. 13/05/29 11:03:20 INFO mapred.JobClient: Task Id : attempt_201305290801_0002_m_000002_0, Status : FAILED
  2. Task attempt_201305290801_0002_m_000002_0 failed to report status for 604 seconds. Killing!

在某个节点上尝试了任务，但该节点在10分钟内未向jobtracker报告。这导致任务重新安排。深入查看更多日志，确定哪些特定节点无法完成分配的任务可能是您应该做的事情。

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com