从eclipse导出hadoopjar

f1tvaqid  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(229)

我正在eclipse中构建mapreduce程序,并尝试将jar导出为具有我自己的运行配置的可运行jar。在run配置中,我指定了main类,获取导出的jar并将其放到hadoop/playground文件夹中。当我试图用这个命令执行它时:

  1. hadoop jar playground/MyProg.jar "packageinfo" "input" "output"

我总是得到一个privilegedaccessexception,说输入路径不存在。我已经检查了包结构和“输入”和“输出”路径。我怀疑这与出口jar的结构有关。
使用以下两个命令是非常不协调的:

  1. $ javac -classpath *:lib/* -d playground/classes playground/src/WordCount.java
  2. $ jar -cvf playground/WordCount.jar -C playground/classes/ .

尤其是当我的mapreduce程序变得越来越复杂时。
我想知道是否有一种特定的方法可以用正确的目录结构从eclipse导出jar,这样我就不会得到privilegedaccessexception。对我来说,虽然mapreduce本身的开发变得越来越容易,但它的调试、部署和测试却成了一个巨大的挑战,尤其是在linux环境中,所有的终端和文件系统命令都是如此的神秘。非常感谢所有帮助我理解这一过程的投入。
尝试运行名为myjob.jar的应用程序时出现的错误消息示例如下:

  1. hadoop jar playground/MyJob.jar src.main.myjob.MyJob /usr/root/MyJob/cite75_99.txt /usr/root/MyJob/cite75_99OUT.txt
  2. Warning: $HADOOP_HOME is deprecated.
  3. 13/09/28 15:20:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
  4. 13/09/28 15:20:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
  5. 13/09/28 15:20:14 WARN snappy.LoadSnappy: Snappy native library not loaded
  6. 13/09/28 15:20:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:9000/tmp/hadoop-root/mapred/staging/root/.staging/job_201309281443_0007
  7. 13/09/28 15:20:14 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/src.main.myjob.MyJob
  8. Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/src.main.myjob.MyJob
  9. at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
  10. at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
  11. at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
  12. at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
  13. at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
  14. at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
  15. at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
  16. at java.security.AccessController.doPrivileged(Native Method)
  17. at javax.security.auth.Subject.doAs(Subject.java:415)
  18. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  19. at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
  20. at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
  21. at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
  22. at src.main.myjob.MyJob.run(MyJob.java:87)
  23. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
  24. at src.main.myjob.MyJob.main(MyJob.java:94)
  25. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  26. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  27. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  28. at java.lang.reflect.Method.invoke(Method.java:606)
  29. at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题