在hive转换中传递表头

vecaoik1  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(396)

我正在配置单元中创建一个查询来执行r脚本。我正在使用transform函数传递表。然而,当我收到r中的表时,它没有标题。我知道我可以创建一个变量,并要求用户手动插入标题,但我不想这样做。我想自动做一些事情,我在考虑两个选择:
1) 在使用transform函数时,找出传递包含标题的表的方法
2) 将头保存在一个变量中,并在transform中传递它(我已经用不同的方法尝试过了,但不是传递查询结果,而是传递查询字符串-如下所示)
以下是我所拥有的:

--Name of the origin table
set source_table = categ_table_small;
--Number of clusters
set k = "5";
--Distance to be used in the model
set distance = "euclidean";
--Folder where the results of the model will be saved
set dir_tar = "/output_r";
--Name of the model used in the naming of the files
set model_name ="testeclara_small";
--Samples: integer, number of samples to be drawn from the dataset.
set n_samples = "10";
--sampsize: integer, number of observations in each sample. This formula is suggested by the package. sampsize<-min(nrow(x), 40 + 2 * k)
set sampsize = "50";

--Creating a matrix which will store the sample number and the group of each sample according to the algorithm
CREATE TABLE IF NOT EXISTS  medoids_result AS SELECT * FROM categ_table_small;

--In the normal situation you don't have the output label, it means you just have 'x' and do not have 'y', so you need to add one extra column to receive
--the group of each observation
--ALTER TABLE medoids_result ADD COLUMNS (medoid INT);

set result_matrix = medoids_result;
set headerMatrix = show columns in categ_table_small;

--Trainning query
SET mapreduce.job.name = K medoids Clara- ${hiveconf:source_table};
SET mapreduce.job.reduces=1; 

INSERT OVERWRITE TABLE ${hiveconf:result_matrix} 
SELECT TRANSFORM ($begin(cols="${hiveconf:source_table}" delimiter= "," excludes="y")$column$end)
USING '/usr/bin/Rscript_10gb /programs_r/du8_dev_1.R ${hiveconf:k}${hiveconf:distance}${hiveconf:dir_tar}${hiveconf:model_name}${hiveconf:n_samples}${hiveconf:sampsize}${hiveconf:headerMatrix}'
AS 
(
$begin(table='${hiveconf:result_matrix}') $column$end
)
FROM
(SELECT *
FROM ${hiveconf:source_table}
DISTRIBUTE BY '1'
)t1;
wmvff8tz

wmvff8tz1#

您可以添加此行

hive -e 'set hive.cli.print.header=true;select * from tablename;'

其中tablename是指您的表名
如果您希望默认地为每个表工作,那么您需要使用以下命令更新$home/.hiverc文件 hive> set hive.cli.print.header=true; 在第一行。

相关问题