我正在尝试使用sparksession将csv数据加载到配置单元表中。我想在加载到配置单元表时跳过头数据，设置tblproperty（“skip.header.line.count”=“1”）也不起作用。
我正在使用以下代码。
import java.io.File
import org.apache.spark.sql.{SparkSession,Row,SaveMode}
case class Record(key: Int, value: String)
val warehouseLocation=new File("spark-warehouse").getAbsolutePath
val spark=SparkSession.builder().appName("Apache Spark Book Crossing Analysis").config("spark.sql.warehouse.dir",warehouseLocation).enableHiveSupport().getOrCreate()
import spark.implicits._
import spark.sql
//sql("set hive.vectorized.execution.enabled=false")
sql("drop table if exists BookTemp")
sql ("create table BookTemp(ISBN int,BookTitle String,BookAuthor String ,YearOfPublication int,Publisher String,ImageURLS String,ImageURLM String,ImageURLL String)row format delimited fields terminated by ';' ")
sql("alter table BookTemp set TBLPROPERTIES("skip.header.line.count"="1")")
sql("load data local inpath 'BX-Books.csv' into table BookTemp")
sql("select * from BookTemp limit 5").show控制台出错：
res55: org.apache.spark.sql.DataFrame = []
<console>:1: error: ')' expected but '.' found.
sql("alter table BookTemp set TBLPROPERTIES("skip.header.line.count"="1")")
2019-02-20 22:48:09 WARN LazyStruct:151 - Extra bytes detected at the end of the row! Ignoring similar problems.
+----+--------------------+--------------------+-----------------+--------------------+--------------------+--------------------+--------------------+
|ISBN| BookTitle| BookAuthor|YearOfPublication| Publisher| ImageURLS| ImageURLM| ImageURLL|
+----+--------------------+--------------------+-----------------+--------------------+--------------------+--------------------+--------------------+
|null| "Book-Title"| "Book-Author"| null| "Publisher"| "Image-URL-S"| "Image-URL-M"| "Image-URL-L"|
|null|"Classical Mythol...|"Mark P. O. Morford"| null|"Oxford Universit...|"http://images.am...|"http://images.am...|"http://images.am...|
|null| "Clara Callan"|"Richard Bruce Wr...| null|"HarperFlamingo C...|"http://images.am...|"http://images.am...|"http://images.am...|
|null|"Decision in Norm...| "Carlo D'Este"| null| "HarperPerennial"|"http://images.am...|"http://images.am...|"http://images.am...|
|null|"Flu: The Story o...| "Gina Bari Kolata"| null|"Farrar Straus Gi...|"http://images.am...|"http://images.am...|"http://images.am...|
+----+--------------------+--------------------+-----------------+--------------------+--------------------+--------------------+--------------------+
only showing top 5 rows如结果所示，我想跳过第一行数据
2条答案
按热度按时间xlpyo6sf1#
如果您使用的是sql，则解决方法是向sql添加筛选器：
这个jira是相关的：https://issues.apache.org/jira/browse/spark-11374
同时阅读以下内容：https://github.com/apache/spark/pull/14638 -您可以使用csv读取器选项：
或在加载前使用shell移除收割台：
qaxu7uf22#
另一种方法是，我尝试使用sparksql将带有标题的csv转换为parquet
val df=spark.sql（“select*from schema.table”）
df.coalesce（1）.write.options（map（“header”->“true”，“compression”->“snappy”））.mode（savemode.overwrite）.parquet（）