将scalaDataframe列激发为嵌套的json

kx1ctssn 于 2021-05-27 发布在 Spark

关注(0)|答案(1)|浏览(329)

我正在尝试将dataframe转换为嵌套的json。
基本上，最终输出是在现场进行的。” id “与其他字段的嵌套json格式保持一致。
“使用字段的json格式” rank “作为密钥和” desc “和” percent “作为价值观。谢谢你的帮助！
来源data:-

val df = Seq(
  ("1", "ABC", "1","91.68"),
  ("1", "BCD", "2","89.03"),
  ("1", "DEF", "3","78.32"),
  ("1", "XYZ", "4","70.64")
).toDF("id", "desc", "rank", "percent")

+---+----+----+-------+
|id |desc|rank|percent|
+---+----+----+-------+
|1  |ABC |1   |91.68  |
|1  |BCD |2   |89.03  |
|1  |DEF |3   |78.32  |
|1  |XYZ |4   |70.64  |
+---+----+----+-------+

预期产量：

+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|id |json                                                                                                                                                    |
+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|1  |{"1":{"desc":"ABC","percent":"91.68"},"2":{"desc":"BCD","percent":"89.03"},"3":{"desc":"DEF","percent":"78.32"},"4":{"desc":"XYZ","percent":"70.64"}}   |
+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+

scala JSON apache-spark

来源：https://stackoverflow.com/questions/63858239/spark-scala-dataframe-columns-to-nested-json

1条答案

按热度按时间

chhkpiq41#

您可以使用一个简单的自定义项来根据需要修复json：

val toJsonUDF = udf((key: String, jsonString: String) =>  "{ " + key + ":" + jsonString + "}")

然后用 withColumn 方法使用以前的自定义项和 to_json 方法，该方法从给定列创建jsonstring：

df.withColumn("json", toJsonUDF($"rank", to_json(struct("desc","percent"))))
  .select("id", "json")
  .groupBy("id")
  .agg(collect_list("json").as("json"))

输出

+---+------------------------------------------------------------------------------------------------------------------------------------------------------------+
      |id |collect_list(json)                                                                                                                                          |
      +---+------------------------------------------------------------------------------------------------------------------------------------------------------------+
      |1  |[{ 1:{"desc":"ABC","percent":"91.68"}}, { 2:{"desc":"BCD","percent":"89.03"}}, { 3:{"desc":"DEF","percent":"78.32"}}, { 4:{"desc":"XYZ","percent":"70.64"}}]|
      +---+------------------------------------------------------------------------------------------------------------------------------------------------------------+

赞(0）回复(0）举报 2021-05-27

我来回答

将scalaDataframe列激发为嵌套的json

1条答案

相关问题

热门标签

最新问答