orc文件不能在hive中使用

watbbzwu  于 2021-05-27  发布在  Hadoop
关注(0)|答案(0)|浏览(607)

情况如下:
我通过flink将orc格式的数据写入hdfs:
我实现了一个矢量器接口来处理我的数据并将其转换为vectorizedrowbatch
我创造了一个作家:

OrcBulkWriterFactory<MyData> orcBulkWriterFactory = new OrcBulkWriterFactory<>(new MyVectorizerImpl(orcSchemaString));

我配置StreamingFileLink:

StreamingFileSink.forBulkFormat(hdfsPath, orcBulkWriterFactory)
                 .withBucketAssigner(new BaseBucketAssigner<>())
                 .build();

我将作业部署到flink集群中,并在hdfspath目录中查看orc文件。
我通过以下命令创建配置单元表:

CREATE TABLE flink_orc_test(STRING a, BIGINT b) STORED AS ORC 'hdfsPath';

我尝试执行查询

SELECT * FROM flink_orc_test LIMIT 10;

我有个错误

Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0,
operationId=THandleIdentifier(secret='a\x08\xc3U\xbb\xa7I\xce\x96\xa6\xdb\x82\xa4\xa9\xd1x', guid='\xcc:\xca\xcb\x08\xa5KI\x8a}7\x95\xc5\xcd\xd2\xf0')),
orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 6',
sqlState=None, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 6:25:24',
'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:496',
'org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:297',
'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:868',
'sun.reflect.GeneratedMethodAccessor25:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43',
'java.lang.reflect.Method:invoke:Method.java:498',
'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78',
'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63',
'java.security.AccessController:doPrivileged:AccessController.java:-2',
'javax.security.auth.Subject:doAs:Subject.java:422',
'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1731',
'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59',
'com.sun.proxy.$Proxy37:fetchResults::-1',
'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:507',
'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:708',
'org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1717',
'org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1702',
'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286',
'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149',
'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624',
'java.lang.Thread:run:Thread.java:748',
'*java.io.IOException:java.lang.ArrayIndexOutOfBoundsException: 6:29:4',
'org.apache.hadoop.hive.ql.exec.FetchOperator:getNextRow:FetchOperator.java:521', 'org.apache.hadoop.hive.ql.exec.FetchOperator:pushRow:FetchOperator.java:428',
'org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:146',
'org.apache.hadoop.hive.ql.Driver:getResults:Driver.java:2277', 'org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:491', '*java.lang.ArrayIndexOutOfBoundsException:6:37:8', 'org.apache.orc.OrcFile$WriterVersion:from:OrcFile.java:145',
'org.apache.orc.impl.OrcTail:getWriterVersion:OrcTail.java:74',
'org.apache.orc.impl.ReaderImpl:<init>:ReaderImpl.java:385',
'org.apache.hadoop.hive.ql.io.orc.ReaderImpl:<init>:ReaderImpl.java:62',
'org.apache.hadoop.hive.ql.io.orc.OrcFile:createReader:OrcFile.java:89', 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat:getRecordReader:OrcInputFormat.java:1690', 'org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit:getRecordReader:FetchOperator.java:695',
'org.apache.hadoop.hive.ql.exec.FetchOperator:getRecordReader:FetchOperator.java:333',
'org.apache.hadoop.hive.ql.exec.FetchOperator:getNextRow:FetchOperator.java:459'], statusCode=3), results=None, hasMoreRows=None)

我的依赖项:

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <version>3.0.0-cdh6.1.1</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-orc_2.12</artifactId>
  <version>1.11.2</version>
</dependency>

我认为flink兽人使用兽人核心依赖的一般问题,但是为了正确使用Hive,我需要一个Hive兽人。实际上,我不能用Hive兽人取代兽人核心,因为这和Flink兽人的职业不兼容。
我该怎么解决?对于我来说,最好使用streamingfilesink来编写orc文件,而不是使用配置单元的flink表api。
配置单元版本:2.1.1-cdh6.1.1
flink版本:1.11.2

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题