无法使用oozie运行shell脚本

dsf9zpds  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(528)

嗨,我正在尝试通过oozie运行shell脚本。

  1. org.apache.oozie.action.hadoop.ShellMain], exit code [1]

我的job.properties文件

  1. nameNode=hdfs://ip-172-31-41-199.us-west-2.compute.internal:8020
  2. jobTracker=ip-172-31-41-199.us-west-2.compute.internal:8032
  3. queueName=default
  4. oozie.libpath=${nameNode}/user/oozie/share/lib/
  5. oozie.use.system.libpath=true
  6. oozie.wf.rerun.failnodes=true
  7. oozieProjectRoot=shell_example
  8. oozie.wf.application.path=${nameNode}/user/karun/${oozieProjectRoot}/apps/shell

我的工作流.xml

  1. <workflow-app xmlns="uri:oozie:workflow:0.1" name="pi.R example">
  2. <start to="shell-node"/>
  3. <action name="shell-node">
  4. <shell xmlns="uri:oozie:shell-action:0.1">
  5. <job-tracker>${jobTracker}</job-tracker>
  6. <name-node>${nameNode}</name-node>
  7. <configuration>
  8. <property>
  9. <name>mapred.job.queue.name</name>
  10. <value>${queueName}</value>
  11. </property>
  12. </configuration>
  13. <exec>script.sh</exec>
  14. <file>/user/karun/oozie-oozi/script.sh#script.sh</file>
  15. <capture-output/>
  16. </shell>
  17. <ok to="end"/>
  18. <error to="fail"/>
  19. </action>
  20. <kill name="fail">
  21. <message>Incorrect output</message>
  22. </kill>
  23. <end name="end"/>
  24. </workflow-app>

我的shell脚本-script.sh

  1. export SPARK_HOME=/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/spark
  2. export YARN_CONF_DIR=/etc/hadoop/conf
  3. export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
  4. export HADOOP_CMD=/usr/bin/hadoop
  5. /SparkR-pkg/lib/SparkR/sparkR-submit --master yarn-client examples/pi.R yarn-client 4

错误日志文件

  1. WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-5.4.2- 1.cdh5.4.2.p0.2/etc/hive-webhcat/conf.dist/webhcat-default.xml:
  2. CDH_KMS_HOME=/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/lib/hadoop-kms:
  3. LANG=en_US.UTF-8:
  4. HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-5.4.2- 1.cdh5.4.2.p0.2/lib/hadoop-mapreduce:

=================================================================
现在调用shell命令行>>

  1. Stdoutput Running /opt/cloudera/parcels/CDH-5.4.2-
  2. 1.cdh5.4.2.p0.2/lib/spark/bin/spark-submit --class edu.berkeley.cs.amplab.sparkr.SparkRRunner --files hdfs://ip-172-31-41-199.us-west-2.compute.internal:8020/user/karun/examples/pi.R --master yarn-client
  3. /SparkR-pkg/lib/SparkR/sparkr-assembly-0.1.jar hdfs://ip-172-31-41-199.us-west- 2.compute.internal:8020/user/karun/examples/pi.R yarn-client 4
  4. Stdoutput Fatal error: cannot open file 'pi.R': No such file or directory
  5. Exit code of the Shell command 2
  6. <<< Invocation of Shell command completed <<<
  7. <<< Invocation of Main class completed <<<
  8. Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
  9. Oozie Launcher failed, finishing Hadoop job gracefully
  10. Oozie Launcher, uploading action data to HDFS sequence file: hdfs://ip-172-31-41-199.us-west-2.compute.internal:8020/user/karun/oozie-oozi/0000035-150722003725443-oozie-oozi-W/shell-node--shell/action-data.seq
  11. Oozie Launcher ends

我不知道如何解决这个问题。任何帮助都将不胜感激。

bqjvbblv

bqjvbblv1#

  1. sparkR-submit ... examples/pi.R ...

致命错误:无法打开文件“pi.r”:没有这样的文件或目录
消息非常明确:shell尝试从本地文件系统读取r脚本。但是本地的,实际上是什么???
oozie用Yarn来跑你的壳;所以Yarn在随机机器上分配一个容器。这是你必须把它放进脑子里的东西,这样它就变成了一种反射:一个oozie操作所需的所有资源(脚本、库、配置文件等等)都必须是
预先在hdfs中提供
在执行时下载,感谢 <file> oozie脚本中的说明
在当前工作目录中作为本地文件访问
就你而言:

  1. <exec>script.sh</exec>
  2. <file>/user/karun/oozie-oozi/script.sh</file>
  3. <file>/user/karun/some/place/pi.R</file>

然后

  1. sparkR-submit ... pi.R ...
展开查看全部

相关问题