library包不能与oozie一起使用

c9qzyr3d  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(217)

嗨,我在用shell脚本运行oozie。在这个shell脚本中,我使用的是sparkr作业。每当运行oozie作业时,我都会遇到库错误。
这是我的错误。

  1. Stdoutput Running /opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark/bin/spark-submit --class edu.berkeley.cs.amplab.sparkr.SparkRRunner --files pi.R --master yarn-client /SparkR-pkg/lib/SparkR/sparkr-assembly-0.1.jar pi.R yarn-client 4
  2. Stdoutput Error in library(SparkR) : there is no package called SparkR
  3. Stdoutput Execution halted
  4. Exit code of the Shell command 1
  5. <<< Invocation of Shell command completed <<<

我的job.properties文件

  1. nameNode=hdfs://ip-172-31-41-199.us-west-2.compute.internal:8020
  2. jobTracker=ip-172-31-41-199.us-west-2.compute.internal:8032
  3. queueName=default
  4. oozie.libpath=hdfs://ip-172-31-41-199.us-west- 2.compute.internal:8020/SparkR-pkg/lib/
  5. oozie.use.system.libpath=true
  6. oozie.wf.rerun.failnodes=true

oozieprojectroot=shell\u示例oozie.wf.application.path=${oozieprojectroot}/apps/shell
我的工作流.xml

  1. <workflow-app xmlns="uri:oozie:workflow:0.1" name="Test">
  2. <start to="shell-node"/>
  3. <action name="shell-node">
  4. <shell xmlns="uri:oozie:shell-action:0.1">
  5. <job-tracker>${jobTracker}</job-tracker>
  6. <name-node>${nameNode}</name-node>
  7. <configuration>
  8. <property>
  9. <name>mapred.job.queue.name</name>
  10. <value>${queueName}</value>
  11. </property>
  12. </configuration>
  13. <exec>script.sh</exec>
  14. <file>oozie-oozi/script.sh#script.sh</file>
  15. <file>/user/karun/examples/pi.R</file>
  16. <capture-output/>
  17. </shell>
  18. <ok to="end"/>
  19. <error to="fail"/>
  20. </action>
  21. <kill name="fail">
  22. <message>Incorrect output</message>
  23. </kill>
  24. <end name="end"/>
  25. </workflow-app>

我的shell脚本文件

  1. export SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark
  2. export YARN_CONF_DIR=/etc/hadoop/conf
  3. export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
  4. export HADOOP_CMD=/usr/bin/hadoop
  5. /SparkR-pkg/lib/SparkR/sparkR-submit --master yarn-client pi.R yarn-client 4

我不知道如何解决这个问题。任何帮助都将不胜感激。。。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题