使用自定义换行符加载csv

j9per5c4 于 2021-06-02 发布在 Hadoop

关注(0)|答案(1)|浏览(389)

我有一个csv，如下所示，行以“+”而不是新行结束。如何通过在字符串“+”存在的地方换行来加载csv？

VTS,51,0071,9739965515,NM,GP,INF01,V,19,072219,291014,0000.0000,N,00000.0000,E,07AE

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205+++VTS,51,0071,9739965515,NM,GP,INF01,V,18,072311,291014,0000.0000,N,00000.0000,E,C24E+++VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358

VTS,51,0071,9739965515,NM,GP,INF01,V,18,072319,291014,0000.0000,N,00000.0000,E,012F
VTS,51,0071,9739965515,NM,GP,INF01,V,19,072326,291014,0000.0000,N,00000.0000,E,B2E6+++VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072333,291014,0000.0000,N,00000.0000,E,9896
VTS,51,0071,9739965515,NM,GP,INF01,V,18,072340,291014,0000.0000,N,00000.0000,E,9B23

首先，我需要在出现新行或“+++”符号的地方中断行并加载数据。然后，再次使用第二列中的值01进行过滤。
预期产量：

VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205
VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358
VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0

hadoop bigdata apache-pig hadoop-plugins

来源：https://stackoverflow.com/questions/26729359/loading-csv-using-custom-line-break

1条答案

按热度按时间

w1jd8yoj1#

Pig手稿：

A = LOAD 'input.csv' AS (line:chararray);
B = FOREACH A {
                 splitRow = TOKENIZE(line,'+++');
                 GENERATE FLATTEN(splitRow) AS newList;
              }
C = FOREACH B GENERATE FLATTEN(STRSPLIT(newList,',',16));
D = FILTER C BY $1==01;
DUMP D;

输出：

(VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205)
(VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358)
(VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0)

赞(0）回复(0）举报 2021-06-03

我来回答

使用自定义换行符加载csv

1条答案

相关问题

热门标签

最新问答