找不到\u corrupt\u record column pyspark

avwztpqn  于 2021-07-14  发布在  Spark
关注(0)|答案(0)|浏览(294)

我正在尝试选择\u损坏的\u记录,但找不到它。在我的情况下,所有列都应该损坏。列不显示。我试过过滤但没有成功。

  1. schema = StructType([
  2. StructField("TestID", DecimalType(25,10), True),
  3. StructField("Key", DecimalType(25,10), True),
  4. StructField("Company", DecimalType(25,10), True),
  5. StructField("Client", DecimalType(25,10), True),
  6. StructField("Project", DecimalType(25,10), True),
  7. StructField("ingestdatetime", DecimalType(25,10), True),
  8. StructField("_corrupt_record", StringType(), True)
  9. ])
  10. df = spark.read.csv(
  11. '/mnt/jakichan/isgood/ingestdatetime=20210202231912',
  12. schema=location_schema,
  13. header=True,
  14. sep=",",
  15. mode="PERMISSIVE",
  16. columnNameOfCorruptRecord="_corrupt_record",
  17. ).cache()
  18. df.select("_corrupt_record")

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题