我有这样的说法:
--Insert a new column based on filename
Data = LOAD '/user/cloudera/Source_Data' using PigStorage('\t','-tagFile');
Data_Schema = FOREACH Data GENERATE
(chararray)$1 AS Date,
(chararray)$2 AS ID,
(chararray)$3 AS Interval,
(chararray)$4 AS Code,
(chararray)$5 AS S_In,
(chararray)$6 AS S_Out,
(chararray)$7 AS C_In,
(chararray)$8 AS C_Out,
(chararray)$9 AS Traffic;
--Split into different directories
SPLIT Data_Schema INTO Src1 IF (Date == '2016-06-25.txt'),
Src2 IF (Date == '2014-07-31.txt'),
Src3 IF (Date == '2016-01-01.txt');
STORE Src1 INTO '/user/cloudera/Source_DatA/2016-06-25' using PigStorage('\t');
STORE Src2 INTO '/user/cloudera/Source_Data/2014-07-31.txt' using PigStorage('\t');
STORE Src2 INTO '/user/cloudera/Source_Data/2016-01-01' using PigStorage('\t');
我的原始数据有一个例子:
10000 1388530800000 39 8.600870350350515 13.86183926855984 1.7218329193014124 3.424444103320796 25.972920214509095
但是当我执行它成功运行时,hdfs中的文件没有数据。。。
注意,我添加了一个基于文件名的新列。这就是为什么我在foreach statment还有一个专栏。。。
1条答案
按热度按时间mftmpeh81#
如果输入文件名为
2016-06-25.txt
,2014-07-31.txt
以及2016-01-01.txt
然后新添加的列将被引用$0
它将包含文件名。你必须这样做:或者只需在加载文件时指定模式并保持其余内容不变: