我正在使用以下标准将一些配置单元输出转储到csv中:
beeline -f my_script.hql --output_format=csv2 > data.csv
但是,此文件似乎不是正确的csv:
unix系统无法读取它
$ file data.csv
data.csv data
它不能被人读 pandas
在 python
:
>>> import pandas as pd
>>> pd.read_csv("data.csv")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/estergiadis/opt/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "/Users/estergiadis/opt/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 463, in _read
data = parser.read(nrows)
File "/Users/estergiadis/opt/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1154, in read
ret = self._engine.read(nrows)
File "/Users/estergiadis/opt/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 2059, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 4 fields in line 3, saw 7
问题似乎与引用字符有关(引用字符不是 "
如标准csv)。我怎样才能解决这个问题?
1条答案
按热度按时间6psbrbz91#
配置单元使用\0符号作为引号符号,您可以在“”上替换它。
例如:
cat data.csv | sed 's/"/""/g' | tr '\0' '"' > fixed_data.csv