简单hadoop map reduce出错

pdsfdshx  于 2021-06-04  发布在  Hadoop
关注(0)|答案(4)|浏览(354)

我试着运行一个hadoopMap来减少字数问题。据我所知，我正确地设置了hdfs和所有内容。
当我执行的时候，
我在尝试运行一个简单的字数计算问题时遇到这个错误。我是hadoop新手。任何帮助如何解决这将不胜感激。

  1. 13/06/13 20:21:17 INFO input.FileInputFormat: Total input paths to process : 4
  2. 13/06/13 20:21:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  3. 13/06/13 20:21:17 WARN snappy.LoadSnappy: Snappy native library not loaded
  4. 13/06/13 20:21:17 INFO mapred.JobClient: Running job: job_201306131940_0007
  5. 13/06/13 20:21:18 INFO mapred.JobClient:  map 0% reduce 0%
  6. 13/06/13 20:21:21 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_0, Status : FAILED
  7. java.lang.Throwable: Child Error
  8. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  9. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 failed.
  10. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  11. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  12. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  13. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  15. 13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stdout
  16. 13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stderr
  17. 13/06/13 20:21:24 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_1, Status : FAILED
  18. java.lang.Throwable: Child Error
  19. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  20. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 failed.
  21. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  22. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  23. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  24. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  26. 13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stdout
  27. 13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stderr
  28. 13/06/13 20:21:28 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_2, Status : FAILED
  29. java.lang.Throwable: Child Error
  30. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  31. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 failed.
  32. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  33. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  34. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  35. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  37. 13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stdout
  38. 13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stderr
  39. 13/06/13 20:21:34 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_0, Status : FAILED
  40. java.lang.Throwable: Child Error
  41. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  42. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 failed.
  43. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  44. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  45. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  46. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  48. 13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stdout
  49. 13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stderr
  50. 13/06/13 20:21:37 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_1, Status : FAILED
  51. java.lang.Throwable: Child Error
  52. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  53. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 failed.
  54. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  55. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  56. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  57. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  59. 13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stdout
  60. 13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stderr
  61. 13/06/13 20:21:40 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_2, Status : FAILED
  62. java.lang.Throwable: Child Error
  63. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  64. Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 failed.
  65. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110)
  66. at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
  67. at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
  68. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  69. 13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stdout
  70. 13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stderr
  71. 13/06/13 20:21:43 INFO mapred.JobClient: Job complete: job_201306131940_0007
  72. 13/06/13 20:21:43 INFO mapred.JobClient: Counters: 4
  73. 13/06/13 20:21:43 INFO mapred.JobClient:   Job Counters 
  74. 13/06/13 20:21:43 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=24100
  75. 13/06/13 20:21:43 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
  76. 13/06/13 20:21:43 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
  77. 13/06/13 20:21:43 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
hadoopubuntumapreduceword-count

4条答案

按热度按时间
z9gpfhce

z9gpfhce1#

我有这个问题，但它是一个包名问题。
我在用maven编译。在pom.xml中，我们有 <groupId>GROUPNAME</groupId> 我的所有包都必须在groupname中，例如groupname.newpackage.classname
然后，它就起作用了。

赞(0）分享  回复(0）举报  2021-06-04
hm2xizp9

hm2xizp92#

我得到了同样的错误，而试图做同样的事情-执行简单的文本字计数。在我的例子中，我解决了更改hadoop所依赖的jdk版本的问题。我正在使用hadoop1.2.1，我在网上发现它的首选java版本是1.6

赞(0）分享  回复(0）举报  2021-06-04
nzrxty8p

nzrxty8p3#

我花了一整天的时间来解决这个问题，直到我在网上找到一个解决方案，在virtualbox的共享目录中不支持创建symlink的操作。如果将hadoop目录移动到/usr/local可以解决您的问题，请尝试。

赞(0）分享  回复(0）举报  2021-06-04
vtwuwzda

vtwuwzda4#

当你得到

  1. java.lang.Throwable: Child Error
  2. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

似乎分配给任务跟踪器的内存多于节点的实际内存。
你可以在这里得到更多

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com