当我在本地运行pig脚本(pig-file script.pig-param input=val-param output=val)时,一切正常。但是当我使用oozie(协调器/工作流)安排pig脚本时,脚本就失败了。我不明白为什么。。。
有人能帮我吗?
Pig手稿
alarms = LOAD '$INPUT' USING PigStorage('|', '-noschema') AS (
row_num:long,
timestamp:chararray,
protocol_name:chararray,
source_ip:chararray,
destination_ip:chararray,
source_port:int,
destination_port:int
);
alarms_projection = FOREACH alarms {
GENERATE
SUBSTRING(timestamp, 0, 10) as alarm_date:chararray,
SUBSTRING(timestamp, 11, 19) as alarm_time:chararray,
protocol_name,
source_ip,
destination_ip,
source_port,
destination_port;
}
STORE alarms_projection INTO '$OUTPUT' USING PigStorage('|');
错误
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], exception invoking main(), Scheme not present in uri /etl/av/complete/alarms
org.apache.oozie.action.hadoop.LauncherException: Scheme not present in uri /etl/av/complete/alarms
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.oozie.action.hadoop.LauncherException: Scheme not present in uri /etl/av/complete/alarms
at org.apache.oozie.action.hadoop.LauncherURIHandlerFactory.getURIHandler(LauncherURIHandlerFactory.java:41)
at org.apache.oozie.action.hadoop.PrepareActionsDriver.doOperations(PrepareActionsDriver.java:65)
at org.apache.oozie.action.hadoop.LauncherMapper.executePrepare(LauncherMapper.java:444)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:173)
... 8 more
Oozie Launcher failed, finishing Hadoop job gracefully
1条答案
按热度按时间k5hmc34c1#
这是workflow.xml中的配置错误。我使用了一个prepare语句来清空输出目录。但不是设定路径hdfs://node:port/path/to/the/file i used/path/to/the/files。
正确的使用方法