apache pig-illustrate命令错误

neekobn8  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(403)
  1. ]$ cat webccess.txt
  2. mark,yahoo.com,6
  3. sam,google.com,7
  4. john,yahoo.com,3
  5. patrick,cnn.com,8
  6. mary,facebook.com,1
  7. mark,yahoo.com,4
  8. john,bbc.com,10
  9. andrew,twitter.com,3
  10. patrick,twitter.com,9

我正在cloudera quick vm hue pig shell(咕噜)中运行低于任务

  1. grunt> stage1 = LOAD '/user/cloudera/webaccess.txt' USING PigStorage(',') AS (name:chararray, website:chararray, access:int);
  2. grunt> DUMP stage1;
  3. grunt> stage2 = FILTER stage1 by access >= 8;
  4. grunt> stage3 = GROUP stage1 by name;
  5. grunt> stage4 = FOREACH stage3 GENERATE group as GROUPS, MAX(stage1.access);
  6. grunt> DUMP stage4;

输出:

  1. (sam,7)
  2. (john,10)
  3. (mark,6)
  4. (mary,1)
  5. (andrew,3)
  6. (patrick,9)

直到这一切都好起来。
当我在关系stage4上应用illustrate命令复查时,我得到如下所示的错误,

  1. grunt> ILLUSTRATE stage4;
  2. 2014-10-07 04:02:43,639 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
  3. 2014-10-07 04:02:43,642 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost.localdomain:8020
  4. 2014-10-07 04:02:43,643 [main] WARN org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
  5. 2014-10-07 04:02:43,643 [main] WARN org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
  6. 2014-10-07 04:02:43,643 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost.localdomain:8021
  7. 2014-10-07 04:02:43,799 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
  8. 2014-10-07 04:02:43,800 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
  9. 2014-10-07 04:02:43,800 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
  10. 2014-10-07 04:02:43,804 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
  11. 2014-10-07 04:02:43,805 [main] ERROR org.apache.pig.pen.ExampleGenerator - Error reading data. Internal error creating job configuration.
  12. java.lang.RuntimeException: Internal error creating job configuration.
  13. at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:160)
  14. at org.apache.pig.PigServer.getExamples(PigServer.java:1182)
  15. at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:739)
  16. at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:626)
  17. at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:323)
  18. at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
  19. at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
  20. at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
  21. at org.apache.pig.Main.run(Main.java:538)
  22. at org.apache.pig.Main.main(Main.java:157)
  23. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  24. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  25. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  26. at java.lang.reflect.Method.invoke(Method.java:597)
  27. at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
  28. 2014-10-07 04:02:43,868 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. Exception
  29. Details at logfile: /dev/null

我正处于学习阶段,由于这个错误,我没能转到下一个主题。
同样在第一次打开hue pig shell(grunt)开始这个任务之前,我发现了以下警告。

  1. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
  2. Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
  3. at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
  4. at java.security.AccessController.doPrivileged(Native Method)
  5. at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
  6. at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
  7. at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
  8. at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
  9. Could not find the main class: org.apache.hadoop.util.PlatformName. Program will exit.
  10. which: no hadoop in ((null))
  11. which: no /usr/lib/hadoop/bin/hadoop in ((null))
  12. dirname: missing operand
  13. Try `dirname --help' for more information.
  14. 2014-10-07 03:18:27,802 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.0-cdh4.7.0 (rexported) compiled May 28 2014, 11:05:48
  15. 2014-10-07 03:18:27,803 [main] INFO org.apache.pig.Main - Logging error messages to: /dev/null
  16. 2014-10-07 03:18:28,758 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/cloudera/.pigbootup not found
  17. 2014-10-07 03:18:30,436 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
  18. 2014-10-07 03:18:30,444 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://localhost.localdomain:8020
  19. 2014-10-07 03:18:37,832 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: localhost.localdomain:8021
  20. 2014-10-07 03:18:37,842 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
mlnl4t2r

mlnl4t2r1#

我没有面对任何问题,说明指挥工作正常。你能先在本地模式下执行吗?

  1. $pig -x local
  2. grunt> stage1 = LOAD 'input.txt' USING PigStorage(',') AS (name:chararray, website:chararray, access:int);
  3. grunt> stage2 = FILTER stage1 by access >= 8;
  4. grunt> stage3 = GROUP stage1 by name;
  5. grunt> stage4 = FOREACH stage3 GENERATE group as GROUPS, MAX(stage1.access);
  6. grunt> DUMP stage4;
  7. (sam,7)
  8. (john,10)
  9. (mark,6)
  10. (mary,1)
  11. (andrew,3)
  12. (patrick,9)
  13. grunt> ILLUSTRATE stage4;
  14. ----------------------------------------------------------------------------
  15. | stage1 | name:chararray | website:chararray | access:int |
  16. ----------------------------------------------------------------------------
  17. | | john | yahoo.com | 3 |
  18. | | john | bbc.com | 10 |
  19. ----------------------------------------------------------------------------
  20. --------------------------------------------------------------------------------------------------------------------------
  21. | stage3 | group:chararray | stage1:bag{:tuple(name:chararray,website:chararray,access:int)} |
  22. --------------------------------------------------------------------------------------------------------------------------
  23. | | john | {(john, yahoo.com, 3), (john, bbc.com, 10)} |
  24. | | john | {(john, yahoo.com, 3), (john, bbc.com, 10)} |
  25. --------------------------------------------------------------------------------------------------------------------------
  26. ------------------------------------------------
  27. | stage4 | GROUPS:chararray | :int |
  28. ------------------------------------------------
  29. | | john | 10 |
  30. ------------------------------------------------
展开查看全部
cuxqih21

cuxqih212#

似乎是类路径问题。请检查是否在类路径中指定了所有必需的jar。请查看此线程以了解更多详细信息

相关问题