我在azure和 .csv hdfs（azure存储）中的文件。
我想使用apachepig处理这些文件并将输出存储在一个配置单元表中。为此，我编写了以下脚本：
A = LOAD '/test/input/t12007.csv' USING PigStorage(',') AS (year:chararray,ArrTime:chararray,DeptTime:chararray);
describe A;
dump A;
store A into 'testdb.tbl3' using org.apache.hive.hcatalog.pig.HCatStorer();此脚本成功加载文件，描述结构，并使用dump显示数据，但在执行store命令时抛出以下错误：
2017-05-02 06:18:41,476 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.hive.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 33> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hive.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Caused by: <file script.pig, line 4, column 33> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hive.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hive.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
2017-05-02 06:18:41,484 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hive.hcatalog.pig.HCatStorer using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
1条答案
按热度按时间t98cgbkg1#
pig -useHCatalog来自清管器hcatalog文档用hcatalog运行清管器
清管器不会自动拾取
HCatalog罐。要引入必要的jar，可以在pig命令中使用标志，也可以设置环境变量PIG_CLASSPATH以及PIG_OPTS如下所述。带上合适的jar一起工作HCatalog，只需在脚本中包含以下标志：备用方式：
指定文件的位置
HCatalog罐和添加一个REGISTER语句，其中jar的路径位于脚本顶部，如下所示。根据集群中的安装情况，您的路径可能不同。您可以使用以下命令找到这个jar位置：
locate *hcatalog-core*hcatstorer公司HCatStorer与pig脚本一起使用，以将数据写入HCatalog-managedtable。使用
HCatStorer通过pig store语句访问。