使用sqoop执行oozie工作流时出错

zpqajqem  于 2021-05-30  发布在  Hadoop
关注(0)|答案(1)|浏览(492)

我已经编写了一个sqoop导入脚本来将数据从teradata导入到hive。当我从命令行运行它时，它工作得很好，但是当我将它放入shell脚本并尝试通过oozie工作流执行时，我得到了以下错误

  1. `[0000069-150114201015959-oozie-oozi-W@sqoop-shell]**Launcher ERROR,  reason: Main class [org.apache.oozie.action.hadoop.ShellMain],**exit code [1]
  2. 2015-04-02 08:50:55,440  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
  3. 2015-04-02 08:50:55,459  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@sqoop-shell] ERROR is considered as FAILED for SLA
  4. 2015-04-02 08:50:55,505  INFO ActionStartXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] Start action [0000069-150114201015959-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
  5. 2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] [***0000069-150114201015959-oozie-oozi-W@fail***]Action status=DONE
  6. 2015-04-02 08:50:55,505  WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] [***0000069-150114201015959-oozie-oozi-W@fail***]Action updated in DB!
  7. 2015-04-02 08:50:55,522  INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[0000069-150114201015959-oozie-oozi-W@fail] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W
  8. 2015-04-02 08:50:55,556  WARN CoordActionUpdateXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[-]**E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100**`

下面是我的workflow.xml

  1. `<workflow-app xmlns='uri:oozie:workflow:0.3' name='sqoop-shell-wf'>
  2. <start to='sqoop-shell' />
  3. <action name='sqoop-shell'>
  4.     <shell xmlns="uri:oozie:shell-action:0.1">
  5.        <job-tracker>${resourceManager}</job-tracker>
  6.     <name-node>${nameNode}</name-node>
  7.     <configuration>
  8.         <property>
  9.           <name>mapred.job.queue.name</name>
  10.           <value>${queueName}</value>
  11.         </property>
  12.     </configuration>
  13.     <exec>Sqoopscript.sh</exec>
  14.     <file>Sqoopscript.sh#script.sh</file>
  15. </shell>
  16. <ok to="end" />
  17. <error to="fail" />
  18. </action>
  19. <kill name="fail">
  20. <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  21. </kill>

Map减少日志

  1. error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission denied)
  2. error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient     .jar (Permission denied)
  3. error: error reading /usr/lib/hadoop/lib/aster-  networking.jar; /usr/lib/hadoop/lib/aster-networking.jar (Permission  denied)  
  4. error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar (Permission denied)
  5. error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission denied)
  6. error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission denied)
  7. Note: /tmp/sqoop yarn/compile/f3aca99f37fa19e505b47bab1499bf5d/EXT_EVNT_ERR_ARC.java uses or  overrides a deprecated API.
  8. Note: Recompile with -Xlint:deprecation for details.
  9. Intercepting System.exit(1)
  10. Failing Oozie Launcher, Main class   [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
  11. che.sqoop.hive.HiveImport  -**Caused by: java.io.FileNotFoundException: File does not exist: hdfs:/user/yarn**`
hadoopHivesqoopoozie

1条答案

按热度按时间
lqfhib0f

lqfhib0f1#

看起来您正在使用shell操作，而您可以/应该使用sqoop操作。运行sqoop作业时需要包含几个jar。其中大部分都包含在oozie共享库中。下面是一个示例工作流：

  1. <workflow-app name="sqoop-to-hive" xmlns="uri:oozie:workflow:0.4">
  2.     <start to="sqoop2hive"/>
  3.     <action name="sqoop2hive">
  4.         <sqoop xmlns="uri:oozie:sqoop-action:0.2">
  5.             <job-tracker>${jobTracker}</job-tracker>
  6.             <name-node>${nameNode}</name-node>
  7.             <command>import --connect jdbc:mysql://mysql.example.com/sqoop --username sqoop --password sqoop --table test --hive-import --hive-table test</command>
  8.             <archive>/tmp/mysql-connector-java-5.1.31-bin.jar#mysql-connector-java-5.1.31-bin.jar</archive>
  9.             <file>/tmp/hive-site.xml#hive-site.xml</file>
  10.         </sqoop>
  11.         <ok to="end"/>
  12.         <error to="kill"/>
  13.     </action>
  14.     <kill name="kill">
  15.         <message>Action failed</message>
  16.     </kill>
  17.     <end name="end"/>
  18. </workflow-app>

我建议遵循关于如何使用oozie和sqoop的指南。

展开查看全部
赞(0）分享  回复(0）举报  2021-05-30
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com