我是pyspark的新手，我需要以这样一种方式分解我的值数组，即每个值都被分配到一个新列。我尝试使用explode，但无法获得所需的输出。下面是我的输出
+---------------+----------+------------------+----------+---------+------------+--------------------+|account_balance|account_id|credit_Card_Number|first_name|last_name|phone_number| transactions|+---------------+----------+------------------+----------+---------+------------+--------------------+| 100000| 12345| 12345| abc| xyz| 1234567890|[1000, 01/06/2020...|| 100000| 12345| 12345| abc| xyz| 1234567890|[1100, 02/06/2020...|| 100000| 12345| 12345| abc| xyz| 1234567890|[6146, 02/06/2020...|| 100000| 12345| 12345| abc| xyz| 1234567890|[253, 03/06/2020,...|| 100000| 12345| 12345| abc| xyz| 1234567890|[4521, 04/06/2020...|| 100000| 12345| 12345| abc| xyz| 1234567890|[955, 05/06/2020,...|+---------------+----------+------------------+----------+---------+------------+--------------------+
下面是程序的模式
root|-- account_balance: long (nullable = true)|-- account_id: long (nullable = true)|-- credit_Card_Number: long (nullable = true)|-- first_name: string (nullable = true)|-- last_name: string (nullable = true)|-- phone_number: long (nullable = true)|-- transactions: array (nullable = true)| |-- element: struct (containsNull = true)| | |-- amount: long (nullable = true)| | |-- date: string (nullable = true)| | |-- shop: string (nullable = true)| | |-- transaction_code: string (nullable = true)
我想要一个输出，其中我有额外的金额，日期，商店，交易代码与各自的值列
amount date shop transaction_code1000 01/06/2020 amazon buy1100 02/06/2020 amazon sell6146 02/06/2020 ebay buy253 03/06/2020 ebay buy4521 04/06/2020 amazon buy955 05/06/2020 amazon buy
1条答案
按热度按时间7kqas0il1#
使用
explode然后分头吃struct文件，最后删除新分解的和transactions数组列。Example:```from pyspark.sql.functions import *
got only some columns from json
df.printSchema()
root
|-- account_balance: long (nullable = true)
|-- transactions: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- amount: long (nullable = true)
| | |-- date: string (nullable = true)
df.selectExpr("","explode(transactions)").select("","col.").drop(['col','transactions']).show()
+---------------+------+--------+
|account_balance|amount| date|
+---------------+------+--------+
| 10| 1000|20200202|
+---------------+------+--------+