我想在阿兹卡班做Hive工作
我可以上载配置单元作业,但配置单元作业未正确执行。
ERROR [hive-demo] [Azkaban] Failed to build job executor for job hive-demoJob type 'hive' is unrecognized. Could not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated by ' ' stored as textfile;, working.dir: /home/hduser/Technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * from words limit 10;, hive.query.04: load data local inpath "res/input" into table words;, user.to.proxy: azkaban, hive.query.06: select freq, count(1) as f2 from words group by freq sort by f2 desc limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44, azkaban.flow.start.timezone: America/Los_Angeles, azkaban.flow.start.hour: 22, azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014, azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54, azkaban.flow.start.timestamp: 2014-01-31T22:54:17.111-08:00, azkaban.flow.start.month: 01, azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Failed to build job type, skipping this job 2014/02/01 12:24:17.295 +0530 ERROR [hive-demo] [Azkaban] Job run failed! 2014/02/01 12:24:17.295 +0530 INFO [hive-demo] [Azkaban] Finishing job hive-demo at 1391237657295 2014/02/01 12:24:17.382 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished 2014/02/01 12:24:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 43 from recently finished flows list. 2014/02/01 12:24:17.408 +0530 INFO [hive-demo] [Azkaban] Job Finished hive-demo with status FAILED 2014/02/01 12:24:17.463 +0530 INFO [hive-demo] [Azkaban] Finishing up flow. Awaiting Termination 2014/02/01 12:24:17.463 +0530 INFO [hive- demo] [Azkaban] Setting flow status to Failed. 2014/02/01 12:24:17.464 +0530 INFO [hive- demo] [Azkaban] Flow is set to FAILED 2014/02/01 12:24:17.464 +0530 INFO [hive-demo] [Azkaban] Setting end time for flow 44 to 1391237657464 2014/02/01 12:24:17.552 +0530 INFO [FlowRunnerManager] [Azkaban] Flow 44 is finished. Adding it to recently finished flows list. 2014/02/01 12:26:17.383 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning recently finished 2014/02/01 12:26:17.384 +0530 INFO [FlowRunnerManager] [Azkaban] Cleaning execution 44 from recently finished flows list.
========================================================================
这是我的plugin.properties中的内容
job.class=azkaban.jobtype.hiveutils.azkaban.HiveViaAzkaban
hive.home=HIVE_HOME
hive.aux.jars.path=/usr/lib/hive/lib
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib
# hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive- ${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.global.jvm.args=${hive.jvm.args}
这是我的private.properties jobtype.classpath=$hadoop\u home/conf,$hadoop\u home/lib/,$hive\u home/lib/,$hive\u home/conf,/usr/lib/hive/lib jobtype.class=azkaban.jobtype.hadoopjavajob
hive.aux.jars.path=/usr/lib/hive/lib
env.HADOOP_HOME=$HADOOP_HOME
env.HIVE_HOME=$HIVE_HOME
env.HIVE_AUX_JARS_PATH=/usr/lib/hive/lib
hive.home=$HIVE_HOME
jobtype.global.classpath=${hive.classpath.items}
hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/
我只在executor文件夹中添加了插件文件夹。
为什么不检测作业类型。请帮忙。
1条答案
按热度按时间c86crjj01#
很难说是怎么回事,因为你既不共享作业本身也不共享日志,但请尝试下面的方法,让我们知道这是否有效。创建一个文件
test.job
包括以下内容:并按照“创建流”部分的描述提交。有关azkaban(2.1)支持开箱即用的不同类型作业的更多详细信息,请参阅作业类型/插件文档。