使用hcatalog问题编写hive动态分区

cld4siwp  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(368)

在使用动态分区将数据插入配置单元表时,我遇到了一个问题。
我有一个包含一个普通列和一个分区列的现有表,我正在尝试向这些列中插入数据。我的代码:

// Preparing writer
WriteEntity.Builder builder = new WriteEntity.Builder();
WriteEntity entity = builder.withDatabase(DATABASE_NAME).withTable(TABLE_NAME).withPartition(null).build();
HCatWriter masterHCatWriter = DataTransferFactory.getHCatWriter(entity, CUSTOM_CONFIG);
WriterContext writerContext = masterHCatWriter.prepareWrite();
HCatWriter hCatWriter = DataTransferFactory.getHCatWriter(writerContext);

// Preparing record to be written
List<HCatRecord> hCatRecordsBatch = new ArrayList<HCatRecord>();
HCatRecord hCatRecord = new DefaultHCatRecord(2);
hCatRecord.set(0, "aaa");
hCatRecord.set(1, "bbb");
hCatRecordsBatch.add(hCatRecord);

// Writing record
hCatWriter.write(hCatRecordsBatch.iterator());

但我有个例外:

org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Failed while writing. Cause : org.apache.hive.hcatalog.common.HCatException : 2010 : Invalid partition values specified : Unable to configure dynamic partitioning for storage handler, mismatch between number of partition values obtained[0] and number of partition values required[1]
at org.apache.hive.hcatalog.data.transfer.impl.HCatOutputFormatWriter.write(HCatOutputFormatWriter.java:112)
at ...private classes...
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hive.hcatalog.common.HCatException : 2010 : Invalid partition values specified : Unable to configure dynamic partitioning for storage handler, mismatch between number of partition values obtained[0] and number of partition values required[1]
at org.apache.hive.hcatalog.mapreduce.HCatBaseOutputFormat.configureOutputStorageHandler(HCatBaseOutputFormat.java:156)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.configureDynamicStorageHandler(FileRecordWriterContainer.java:264)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:183)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
at org.apache.hive.hcatalog.data.transfer.impl.HCatOutputFormatWriter.write(HCatOutputFormatWriter.java:98)
... 8 more

我浏览了hive库的代码,它看起来像 prepareWrite() 在主节点上调用的架构错误。它只加载普通列(缺少分区列)的模式,然后无法检索分区列的插入记录中的值(实际上在异常中这样说) ...number of partition values obtained[0] ...). 有同样的问题,所以这个问题,但在我的情况下,我不能附加列到模式,因为它是打包在 prepareWrite() 方法。
我正在使用clouderaversion5.3.2的库(它意味着hiveversion0.13.1)
我将感谢任何帮助。谢谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题