使用hcatstorer时pig出错

xeufq47z  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(431)

我已经用pig拉丁语编写了一个代码,希望将输出存储在hive表中。我已经用这个模式创建了配置单元表,

create  table pedestrian_count(sensor_id INT, sensor_name STRING, total_hourly_counts BIGINT, highest_count INT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ':'
STORED AS TEXTFILE
location '/user/cloudera/output';

我的剧本是,

register 'piggybank-0.15.0.jar';
DEFINE CSVLoader org.apache.pig.piggybank.storage.CSVLoader();
part1 = LOAD '/home/cloudera/Pedestrian_Counts.csv' using CSVLoader(',') as (date_time, sensor_id: int, sensor_name: chararray, hourly_counts: int);
part2 = GROUP part1 BY (sensor_id, sensor_name);
part3 = FOREACH part2 GENERATE FLATTEN(group) AS (sensor_id, sensor_name), SUM(part1.hourly_counts), MAX(part1.hourly_counts);
STORE part3 into 'pedestrian_count' USING org.apache.hcatalog.pig.HCatStorer();

当我试图执行这个脚本时,我得到了一个错误,

Pig Stack Trace
---------------
ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
     at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1701)
     at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1421)
     at org.apache.pig.PigServer.parseAndBuild(PigServer.java:354)
     at org.apache.pig.PigServer.executeBatch(PigServer.java:379)
     at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
     at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
     at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
     at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
     at org.apache.pig.Main.run(Main.java:613)
     at org.apache.pig.Main.main(Main.java:158)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: Failed to parse: Pig script failed to parse:  <file pedestrian.pig, line 6, column 42> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
     at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
     at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688)
... 16 more
Caused by: <file pedestrian.pig, line 6, column 42> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
     at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1275)
     at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1260)
     at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5153)
     at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7751)
     at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
     at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
     at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
     at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
     at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 17 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
    at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:653)
    at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1272)

谁能告诉我哪里做错了,请纠正我。提前谢谢。

f45qwnt8

f45qwnt81#

我想你必须使用下面的命令才能进入咕噜shell。

pig -useHCatalog

请参阅此链接,它表明要处理配置单元,我们必须使用-use hcatalog,它从配置单元库中注册所有必需的jar。
第二个建议:尝试使用以下命令:

STORE part3 into 'pedestrian_count' USING org.apache.hive.hcatalog.pig.HCatStorer();

更换零件3如下:

part3 = FOREACH part2 GENERATE FLATTEN(group) AS (sensor_id, sensor_name), SUM(part1.hourly_counts) AS total_hourly_counts, MAX(part1.hourly_counts) AS highest_count;

相关问题