以序列文件格式将文件配置单元导出到hdfs

sshcrbum 于 2021-06-02 发布在 Hadoop

关注(0)|答案(3)|浏览(405)

我正在尝试执行一个配置单元查询，并以序列文件格式将其输出导出为hdfs。

beeline> show create table test_table;

+--------------------------------------------------------------------------------------+
|                                    createtab_stmt                                    |
+--------------------------------------------------------------------------------------+
| CREATE TABLE `test_table`(                                                           |
|   `XXXXXXXXXXXXXX` bigint,                                                           |
|   `XXXXXXXXXXXxx` int,                                                               |
|   `XXXXXXXXX` int,                                                                   |
|   `XXXXXX` int)                                                                      |
| PARTITIONED BY (                                                                     |
|   `XXXXXXXX` string,                                                                 |
|   `XXXX` string,                                                                     |
|   `XXXXXXXX` string)                                                                 |
| ROW FORMAT DELIMITED                                                                 |
|   FIELDS TERMINATED BY '\u00001'                                                     |
| STORED AS INPUTFORMAT                                                                |
|   'org.apache.hadoop.mapred.SequenceFileInputFormat'                                 |
| OUTPUTFORMAT                                                                         |
|   'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat'                        |
| LOCATION                                                                             |
|   'hdfs://localhost:8020/user/hive/warehouse/local_hive_report.db/test_table'        |
| TBLPROPERTIES (                                                                      |
|   'transient_lastDdlTime'='1437569941')                                              |
+--------------------------------------------------------------------------------------+

这是我试图导出数据的查询，

beeline> INSERT OVERWRITE DIRECTORY '/user/nages/load/date' 
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY ',' 
    LINES TERMINATED BY '\n' 
    STORED AS SEQUENCEFILE 
    SELECT * FROM test_table WHERE column=value;

这是错误，

Error: Error while compiling statement: FAILED: ParseException line 1:61 
cannot recognize input near 'ROW' 'FORMAT' 'DELIMITED' in statement (state=42000,code=40000)

我是不是漏了什么？
软件版本：cloudera hadoop cdh5.3.3，apache版本0.13.1。
编辑：更新我的临时解决方案如下。

hadoop Hive sequencefile cloudera cloudera-cdh

来源：https://stackoverflow.com/questions/31645847/export-file-file-hive-to-hdfs-in-sequence-file-format

3条答案

按热度按时间

xxhby3vn1#

作为临时修复，我创建了一个序列文件格式的配置单元表，并将选定的记录插入其中。

CREATE TABLE temp_table
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS SEQUENCEFILE
AS 
SELECT * FROM test_table WHERE column=value;

这将在hdfs中的以下位置创建序列文件。 /<HIVE_DATABASE_ROOT>/temp_table.db/

赞(0）回复(0）举报 2021-06-02

yhuiod9q2#

这是因为配置单元查询在默认情况下将使用 ^ 作为分隔符。您可以通过导出到本地文件系统来尝试相同的方法。这应该是受支持的。

beeline> INSERT OVERWRITE LOCAL DIRECTORY '/user/~local directoryname' 
    ROW FORMAT DELIMITED 
    FIELDS TERMINATED BY ',' 
    LINES TERMINATED BY '\n' 
    STORED AS SEQUENCEFILE 
    SELECT * FROM test_table WHERE column=value;

赞(0）回复(0）举报 2021-06-02

ccrfmcuu3#

这个脚本适合我：

CREATE EXTERNAL TABLE dept_seq (department_id int, department_name string) ROW FORMAT DELIMITED FIELDS TERMINATED by '\001' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.SequenceFileInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat' LOCATION 'hdfs:///user/cloudera/departments_seq';

赞(0）回复(0）举报 2021-06-02

我来回答

以序列文件格式将文件配置单元导出到hdfs

3条答案

相关问题

热门标签

最新问答