我正在尝试用oozie在hadoop中运行作业,hadoop和oozie版本是:
[root@10-0-87-4 /]# hadoop version
Hadoop 2.6.0-cdh5.8.0
Subversion http://github.com/cloudera/hadoop -r 042da8b868a212c843bcbf3594519dd26e816e79
Compiled by jenkins on 2016-07-12T23:02Z
Compiled with protoc 2.5.0
From source with checksum 2b6c319ecc19f118d6e1c823175717b5
This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.8.0.jar
[root@10-0-87-4 /]# oozie version
Oozie client build version: 3.3.2.1.3.2.0-111
oozie不是cdh repo自带的,我是通过以下方式安装的:
$ wget -nv http://public-repo-1.hortonworks.com/HDP/centos6/1.x/updates/1.3.2.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
$ yum install -y oozie extjs-2.2-1 --disablerepo cloudera-cdh5
这是我需要它的工作方式,请不要建议安装从cdh5本地oozie,因为我需要这个特定的oozie。
当我想运行某个作业时,问题就开始了,我遇到以下错误:
/var/log/oozie/oozie.log:2016-10-10 12:22:03,245 WARN ActionStartXCommand:542 - SERVER[] USER[u_1813] GROUP[-] TOKEN[] APP[execute-wf] JOB[0000000-161010120015197-oozie-oozi-W] ACTION[0000000-161010120015197-oozie-oozi-W@run-luigi] Error starting action [run-luigi]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: No such file or directory]
/var/log/oozie/oozie.log:2016-10-10 12:22:04,275 WARN CoordActionUpdateXCommand:542 - SERVER[] USER[u_1813] GROUP[-] TOKEN[] APP[execute-wf] JOB[0000000-161010120015197-oozie-oozi-W] ACTION[0000000-161010120015197-oozie-oozi-W@run-luigi] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
我检查了工作流程,所有提到的文件都在hdfs中,
<file>python.sh#python.sh</file>
<file>oozie-luigi-pig-workflow.py#oozie-luigi-pig-workflow.py</file>
<file>/user/oozie/share/private_key#private_key</file>
<file>hdfs://hdfs.<some-link>:8020/user/u_1813/oozie/workflows/pig-execute/lib/file-s3distcp.jar#file-s3distcp.jar</file>
<file>hdfs://hdfs.<some-link>:8020/user/u_1813/oozie/workflows/pig-execute/lib/file_udfs.rb#file_udfs.rb</file>
namenode\datanode\tasktracker\jobtracker\pig-进程运行没有任何问题,唯一的问题是oozie。
额外日志来自 /var/log/oozie/oozie.log
```
2016-10-17 18:05:45,148 WARN UserGroupInformation:1696 - SERVER[] PriviledgedActionException as:u_1813 (auth:PROXY) via oozie (auth:SIMPLE) cause:ENOENT: No such file or directory
2016-10-17 18:05:45,148 WARN UserGroupInformation:1696 - SERVER[] PriviledgedActionException as:u_1813 (auth:PROXY) via oozie (auth:SIMPLE) cause:ENOENT: No such file or directory
2016-10-17 18:05:45,148 WARN ActionStartXCommand:542 - SERVER[] USER[u_1813] GROUP[-] TOKEN[] APP[execute-wf] JOB[0000000-161017180107504-oozie-oozi-W] ACTION[0000000-161017180107504-oozie-oozi-W@prepare-script] Error starting action [prepare-script]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: No such file or directory]
org.apache.oozie.action.ActionExecutorException: JA009: No such file or directory
at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:762)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:913)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
at org.apache.oozie.command.XCommand.call(XCommand.java:277)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:708)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:500)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:481)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:497)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:611)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:747)
... 8 more
顺便说一句,oozie正在使用postgresql server,它工作正常我可以访问oozie ui,可以看到作业和扩展。。当我尝试运行任何类型的作业时,问题就开始了。
你知道我怎么修吗?
谢谢,埃迪姆
暂无答案!
目前还没有任何答案,快来回答吧!