AWK过滤CSV文件

6ljaweal  于 2023-10-13  发布在  其他
关注(0)|答案(3)|浏览(106)

我想过滤掉第三列中有“同义词”的行。命令如下

awk '$3 !~ /^synonymous/' fileCSV.csv > fileCSV2.csv

但是fileCSV2.csv在第3列中仍然包含单词“synonymous”。我想知道可能是什么问题?
文件CSV.csv中的两行:

"exonic","LINC00115","synonymous SNV","uc010nxx.2:c.C299T:p.P100L",,"0.99",,0.56,rs3115849,,,,,,,,,,,,,chr1,762273,762273,G,A,"chr1","762273",".","G","A","30483.62","PASS","AC=24;AF=1.00;AN=24;DP=2972;FS=0.000;MLEAC=8;MLEAF=1.00;MQ0=0;VQSLOD=19.50;culprit=FS;set=Intersection","GT:AD:DP:GQ:PL","1/1:0,2:2:6:66,6,0","1/1:0,297:297:99:10476,951,0","1/1:0,304:304:99:10098,950,0","1/1:0,295:295:99:9869,929,0","1/1:0,292:292:99:8655,895,0","1/1:0,304:304:99:10006,965,0","1/1:0,179:179:99:5862,568,0","1/1:0,273:273:99:9328,851,0","1/1:0,279:279:99:7946,850,0","1/1:0,283:283:99:9214,866,0","1/1:0,8:8:21:229,21,0","1/1:0,456:456:99:16385,1285,0"    
"exonic","SAMD11","synonymous SNV","uc001abw.1:c.T1027C:p.W343R","559;Name=lod=249",,,1.00,rs6672356,1,0.916445,N,0.0,T,0.0,B,0.998605,N,4.19E-4,N,3.17,chr1,877831,877831,T,C,"chr1","877831",".","T","C","3594.56","PASS","AC=24;AF=1.00;AN=24;DP=387;FS=0.000;MLEAC=8;MLEAF=1.00;MQ=60.00;MQ0=0;VQSLOD=15.00;culprit=DP;set=Intersection","GT:AD:DP:GQ:PL","1/1:0,3:3:9:97,9,0","1/1:0,3:3:12:113,12,0","1/1:0,64:64:99:1805,189,0","1/1:0,57:57:99:1605,168,0","1/1:0,30:30:90:768,90,0","1/1:0,69:69:99:2026,216,0","1/1:0,15:15:45:428,45,0","1/1:0,23:23:81:809,81,0","1/1:0,22:22:69:562,69,0","1/1:0,40:40:99:1142,117,0","1/1:0,3:3:9:94,9,0","1/1:0,58:58:99:14,7,0"
o2g1uqev

o2g1uqev1#

如果您的文件CSV.csv中的列由,分隔,则需要

awk -F, '$3 !~ /^synonymous/' fileCSV.csv > fileCSV2.csv

如果-F不适用于您的awk版本,请尝试

awk 'BEGIN{FS=","} $3 !~ /^synonymous/' fileCSV.csv > fileCSV2.csv

EDIT:您还需要考虑",因此使用/^"synonymous/

uelo1irk

uelo1irk2#

要使用awk处理csv文件,我更喜欢以下方法来自动考虑引号,即使用sed进行预处理。
对于你的具体问题,我会用

sed -e 's/^"//;s/"$//' fileCSV.csv | awk -F '"?,"?' '$3 !~ /^synonymous/'

如果您还希望正确处理字符串字段包含引号的文件(在csv文件中将由双引号表示),则需要按以下方式更改sed表达式,

sed -e 's/^"//;s/"$//;s/""/"/g' fileCSV.csv | awk -F '"?,"?' '$3 !~ /^synonymous/'

这种方法的优点是,它允许您使用awk正确地打印或处理某些字段。例如,如果你想打印过滤行的第一个和第五个字段,用:分隔,你现在可以使用

sed -e 's/^"//;s/"$//;s/""/"/g' fileCSV.csv | awk -F '"?,"?' '$3 !~ /^synonymous/ { print $1,":",$5}'

(If方法之间的区别对你来说并不清楚,你可以尝试最后一个awk命令而不进行sed预处理)

mwg9r5ms

mwg9r5ms3#

csvgrep来自csvkit

对于awk,最可靠的方法是使用FPAT,如以下所述:使用awk高效解析CSV的最健壮的方法是什么?不幸的是,即使是FPAT也不能处理引号中的文字换行符。
相反,如果你想更理智,有几个CSV CLI工具在那里。一个非常容易通过pip安装的(虽然不一定是最快的,因为它是基于Python的)是来自csvkit的csvgrep

pip install csvkit

然后我们可以得到不匹配的行同义词:

csvgrep -H -i -c3 -r '^synonymous$' mytest.csv

命令的解释:

  • -H:第一行不是标题行
  • -i:反转匹配
  • -c3:对第三列进行操作
  • -r:匹配以下正则表达式

具体例子:

printf '00,11,synonymous,"22,\n""33"\n44,55,notsynonymous,66\n' > nohead.csv
printf 'col1,col2,col3,col4\n00,11,synonymous,"22,\n""33"\n44,55,notsynonymous,66\n' > head.csv

然后又道:

csvgrep -H -i -c3 -r '^synonymous$' nohead.csv | tail -n+2

产出:

44,55,notsynonymous,66

我们导入tail,因为-H添加了一个令人讨厌的虚拟头:

a,b,c,d
44,55,notsynonymous,66

这很烦人,但我不知道该怎么做。
如果没有-i,我们选择synonymous

csvgrep -H -c3 -r '^synonymous$' nohead.csv | tail -n+2

产出:

00,11,synonymous,"22,
""33"

当我们有一个标题时,我们可以使用列名:

csvgrep -c col3 -r '^synonymous$' head.csv

产出:

col1,col2,col3,col4
00,11,synonymous,"22,
""33"

在csvkit 1.0.7,Ubuntu 23.04上测试。

相关问题