我想用aws oozie在s3中移动文件。我想跑
aws s3 mv s3://temp/*.zip s3://temp/processed_files/. --recursive我怎么能在奥齐做到这一点？
编辑1
2015-11-12 10:18:55,758 WARN ShellActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] Launcher exception: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:93)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:55)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:452)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 17 more
2015-11-12 10:18:55,838 INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] ERROR is considered as FAILED for SLA
2015-11-12 10:18:55,880 INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@killemail] Start action [0000118-151029144311676-oozie-oozi-W@killemail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
hue oozie屏幕截图和错误日志的屏幕截图。
3条答案
按热度按时间7kjnsjlb1#
在s3中编写一个java类来移动文件，并在oozie中将其实现为java操作。
fjaof16o2#
如果您按照helmut zechmann所说的做了，但它仍然不起作用，请检查问题是否与shell脚本有关，shell脚本的行结尾是windows而不是unix。
尝试使用notepad++或其他工具将其转换为正确的格式。
ltqd579y3#
问题是oozie找不到你的脚本。将脚本放入hdfs并使用shell操作的file元素为脚本提供alies：
这将创建一个从hdfs路径到
#，在本例中为move.sh。现在可以在exec脚本的元素。看到了吗https://oozie.apache.org/docs/3.3.0/dg_shellactionextension.html 供参考。