spark scala更改dataframe中列的数据类型

b4lqfgs4 于 2021-05-29 发布在 Spark

关注(0)|答案(1)|浏览(641)

条件是：以data-c开头的列名是stringtype列，data-d是datetype列，data-n是doubletype列。我有一个dataframe，其中所有列的数据类型都是一个字符串，因此我尝试以如下方式更新它们的数据类型：

import org.apache.spark.sql.functions._
import sparkSession.sqlContext.implicits._

val diff_set = Seq("col7", "col8", "col15", "Data-C-col1", "Data-C-col3", "Data-N-col2", "Data-N-col4", "Data-D-col16", "Data-D-col18", "Data-D-col20").toSet
var df = (1 to 10).toDF
df = df.select(df.columns.map(c => col(c).as(c)) ++ diff_set.map(c => lit(null).cast("string").as(c)): _*)
df.printSchema()

// This foreach loop yields slow performance
    df.columns.foreach(x => {
      if (x.startsWith("Data-C")) {
        df = df.withColumn(x, col(x).cast(StringType))
      } else if (x.startsWith("Data-D")) {
        df = df.withColumn(x, col(x).cast(DateType))
      } else if (x.startsWith("Data-N")) {
        df = df.withColumn(x, col(x).cast(DoubleType))
      }
    }
    )
df.printSchema()

在scala spark中是否可以更优雅、更高效地实现这一点（性能方面）？

DataFrame apache-spark apache-spark-sql

来源：https://stackoverflow.com/questions/62609586/spark-scala-change-datatype-of-columns-in-dataframe

1条答案

按热度按时间

1wnzp6jl1#

检查以下代码。

scala> df.printSchema
root
 |-- value: integer (nullable = false)
 |-- Data-C-col1: string (nullable = true)
 |-- Data-D-col18: string (nullable = true)
 |-- Data-N-col4: string (nullable = true)
 |-- Data-N-col2: string (nullable = true)
 |-- col15: string (nullable = true)
 |-- Data-D-col16: string (nullable = true)
 |-- Data-D-col20: string (nullable = true)
 |-- col8: string (nullable = true)
 |-- col7: string (nullable = true)
 |-- Data-C-col3: string (nullable = true)

val colum_datatype_mapping = 
Map(
  "Data-C" -> "string",
  "Data-D" -> "date",
  "Data-N" -> "double"
)

val columns = df
.columns
.map { c =>
          val key = c.split("-").init.mkString("-")
          if(colum_datatype_mapping.contains(key)) 
             col(c).cast(colum_datatype_mapping(key)) 
          else 
             col(c)
}

scala> df.select(columns:_*).printSchema
root
 |-- value: integer (nullable = false)
 |-- Data-C-col1: string (nullable = true)
 |-- Data-D-col18: date (nullable = true)
 |-- Data-N-col4: double (nullable = true)
 |-- Data-N-col2: double (nullable = true)
 |-- col15: string (nullable = true)
 |-- Data-D-col16: date (nullable = true)
 |-- Data-D-col20: date (nullable = true)
 |-- col8: string (nullable = true)
 |-- col7: string (nullable = true)
 |-- Data-C-col3: string (nullable = true)

赞(0）回复(0）举报 2021-05-29

我来回答

spark scala更改dataframe中列的数据类型

1条答案

相关问题

热门标签

最新问答