oozie错误-将数据从mysql导入hdfs时

ryevplcw  于 2021-06-03  发布在  Sqoop
关注(0)|答案(2)|浏览(349)

我正在尝试通过sqoop将数据从mysql导入hdfs。我能够通过shell运行sqoop，它运行得很好。但当我通过oozie调用时。它是由以下错误和失败引起的。我还包括jar。
现在我在我的家里尝试这个——CDH5.3虚拟盒中的单节点，但错误仍然是一样的
我知道这已经讨论过了，但仍然没有帮助。。
主要错误是：

ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

作业属性文件如下所示，存在于本地环境中

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/harry/

workflow.xml文件如下所示，它存在于hdfs的${namenode}/user/harry下/

<workflow-app name="Trend_Analysi_Scoop_Script" xmlns="uri:oozie:workflow:0.4">
<start to="sqoop-c68d"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-c68d">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <arg>import</arg>
        <arg> --connect</arg>
        <arg>jdbc:mysql://localhost/oozie_training</arg>
        <arg>--username</arg>
        <arg>root</arg>
        <arg>--password</arg>
        <arg>cloudera</arg>
        <arg>--table</arg>
        <arg>user</arg>
        <arg>--target-dir</arg>
        <arg>/training/sqoop</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

我已经把所有必要的jar都放在下面的路径下了

[cloudera@quickstart ~]$ hadoop fs -ls /user/oozie/share/lib/lib_20141218070949/sqoop
Found 75 items
-rw-r--r--   1 root  oozie     960374 2016-05-26 22:00 /user/oozie/share/lib/lib_20141218070949/sqoop/mysql-connector-java-5.1.34-bin.jar
-rw-r--r--   1 oozie oozie      16924 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/oozie-sharelib-sqoop-4.0.0-cdh5.3.0.jar
-rw-r--r--   1 oozie oozie    1010790 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/sqoop-1.4.5-cdh5.3.0.jar

在添加jar之后，我还重新启动了oozie服务器。
我得到的错误日志如下。

2016-05-24 10:37:17,994  INFO ActionStartXCommand:520 - SERVER[***]             USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] Start action [0000001-160524103133053-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action status=DONE
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action updated in DB!
2016-05-24 10:37:18,164  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@:start:
2016-05-24 10:37:18,165  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:18,188  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Start action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:26,080  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:26,106  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] checking action, hadoop job ID [job_1462295072181_0163] status [RUNNING]
2016-05-24 10:37:26,108  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action status=RUNNING
2016-05-24 10:37:26,114  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action updated in DB!
2016-05-24 10:37:26,184  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@sqoop-c68d
2016-05-24 10:37:37,181  INFO CallbackServlet:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] callback for action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d]
2016-05-24 10:37:37,314  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:37,388  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] action completed, external ID [job_1462295072181_0163]
2016-05-24 10:37:37,395  WARN SqoopActionExecutor:523 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2016-05-24 10:37:37,484  INFO ActionEndXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] ERROR is considered as FAILED for SLA
2016-05-24 10:37:37,679  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] Start action [0000001-160524103133053-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action status=DONE
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action updated in DB!
2016-05-24 10:37:37,930  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@Kill
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi- W@sqoop-c68d

我认为oozie没有接受共享/库，如果是这样的话，如果有人能在同一方面帮助我，那就太好了
任何帮助都将不胜感激。

sqoopoozie

2条答案

按热度按时间
kx7yvsdv

kx7yvsdv1#

您正在使用 --query 中的选项 command 元素，这在oozie中不受支持。你需要使用 arg 单独标记并提供所有参数。这个 command 标记被空格分割，因此它不能与 --query 选项。以下是示例：

<arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://host:3306/oozie</arg>
        <arg>--username</arg>
        <arg>user</arg>
        <arg>--password</arg>
        <arg>pwd</arg>
        <arg>--driver</arg>
        <arg>com.mysql.jdbc.Driver</arg>
        <arg>--query</arg>
        <arg>"select BUNDLE_ACTIONS.* from BUNDLE_ACTIONS where $CONDITIONS"</arg>
        <arg>--hcatalog-table</arg>
        <arg>bundle_actions</arg>
        <arg>--hcatalog-database</arg>
        <arg>oozie</arg>
        <arg>--skip-dist-cache</arg>
        <arg>-m</arg>
        <arg>1</arg>
赞(0）分享  回复(0）举报  2021-06-04
yks3o0rb

yks3o0rb2#

看起来sqoop操作中的“command”标记有问题。因为它已经是一个sqoop操作，所以不需要“/usr/hdp/2.4.0.0-169/sqoop/bin/sqoop”。请参考下面链接中的示例进行更正。然后再次尝试重新运行作业。
https://oozie.apache.org/docs/4.0.0/dg_sqoopactionextension.html#sqoop_action

赞(0）分享  回复(0）举报  2021-06-04
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com