csv文件中的新行字符

fbcarpbf  于 2021-07-12  发布在  Spark
关注(0)|答案(1)|浏览(223)

我正在尝试用apachespark(版本3)读取csv文件。我面临两个问题
csv文件中的一个字段包含新行字符,因此该行被拆分为两行
其中一列包含逗号(也包含新行字符),因此记录被拆分为多个列
下面是我得到的代码和输出

import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

import scala.Tuple2;

public class MainApp {

    public static void main(String[] args) {
        // TODO Auto-generated method stub

        SparkSession session = SparkSession.builder().appName("Sample App").master("local[*]").getOrCreate();

        session.sparkContext().setLogLevel("ERROR");

        Dataset<Row> df = session.read().format("csv").
                option("quote","*" )
                .option("multiLine", "true")
                 .option("ignoreLeadingWhiteSpace", "true")
                 .option("ignoreTrailingWhiteSpace", "true")
                .load("/home/deepak/sample_dataset/*.csv");
        df.printSchema();
        df.show(false);

        session.stop();

    }

}

root
 |-- _c0: string (nullable = true)
 |-- _c1: string (nullable = true)
 |-- _c2: string (nullable = true)
 |-- _c3: string (nullable = true)

+--------------------------------------------+---------------+------+----------------------------+
|_c0                                         |_c1            |_c2   |_c3                         |
+--------------------------------------------+---------------+------+----------------------------+
|15169                                       |Sample data I  |RST1  |"*Insurance data conversion.|
|Sample (Sample/Sample)Sample Sample't Sample|Sample Box 9999|Sample|Sample 888888-7014*"        |
+--------------------------------------------+---------------+------+----------------------------+

下面是输入。如果我们用ms excel打开,它是一行。


* 15169*,Sample data I,RST1,"*Insurance data conversion.

Sample (Sample/Sample)Sample Sample't Sample, Sample Box 9999, Sample, Sample 888888-7014*"

我不知道如何把spark当作一张唱片来读。

cngwdvgl

cngwdvgl1#

您不需要指定 quote 因为你的引号是默认字符 " .

Dataset<Row> df = session.read().format("csv")
                 .option("multiLine", "true")
                 .option("ignoreLeadingWhiteSpace", "true")
                 .option("ignoreTrailingWhiteSpace", "true")
                 .load("/home/deepak/sample_dataset/*.csv");

df.show()
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+
|_c0    |_c1          |_c2 |_c3                                                                                                                   |
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+
|*15169*|Sample data I|RST1|*Insurance data conversion.
Sample (Sample/Sample)Sample Sample't Sample, Sample Box 9999, Sample, Sample 888888-7014*|
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+

相关问题