错误文件不存在,即使在spark submit中以args形式传递文件名时文件存在

apeeds0o  于 2021-05-29  发布在  Spark
关注(0)|答案(1)|浏览(354)

我在spark2 submit中以args的形式传递配置文件名。但我得到的文件不存在错误,即使文件存在。如果我硬编码的文件名,它是工作良好。

  1. spark2-submit --files /data/app/Data_validation/target/input.conf --class "QualityCheck" DC_framework-jar-with-dependencies.jar "input.conf"

代码:

  1. import org.apache.spark.SparkFiles
  2. object QualityCheck {
  3. def main(args: Array[String]) : Unit={
  4. val configFile = SparkFiles.get("input.conf")
  5. val conf = new SparkConf().setAppName("Check Global")
  6. val sc = SparkContext.getOrCreate(conf)
  7. val spark = SparkSession.builder.getOrCreate()
  8. println(configFile)
  9. if (file.exists) {
  10. val config = ConfigFactory.parseFile(file)
  11. } else {
  12. println("Configuration file does not exist")

堆栈跟踪:

  1. /data/app/Data_validation/target/input.conf
  2. Configuration file does not exist
  3. Config(SimpleConfigObject({}))

请帮帮我!

wvmv3b1j

wvmv3b1j1#

请检查下面的代码。

  1. spark2-submit --class "QualityCheck" DC_framework-jar-with-dependencies.jar "/data/app/Data_validation/target/input.conf"
  2. object QualityCheck {
  3. def main(args: Array[String]) : Unit={
  4. val file = new File(args(0))
  5. if (file.exists) {
  6. val config = ConfigFactory.parseFile(file)
  7. } else {
  8. println("Configuration file does not exist")
  9. }
  10. val conf = new SparkConf().setAppName("Check Global")
  11. val sc = SparkContext.getOrCreate(conf)
  12. val spark = SparkSession.builder.getOrCreate()
  13. }
  14. }
展开查看全部

相关问题