val sparkSession = ...
trait Source
case class Parquet(path: String) extends Source
case class Jdbc(url: String, port: Int) extends Source
def readSource(s: Source)(spark: SparkSession): DataFrame = s match {
case Parquet(path) => spark.read.parquet(path)
case Jdbc(url, port) => ...
}
val aSourceFromConfig: Source = ...
val df: DataFrame = readSource(aSourceFromConfig)(spark)
1条答案
按热度按时间ltqd579y1#
您不需要实现数据源,您可以在驱动程序中创建此逻辑来选择要读取的格式。例如: