在pyspark中使用Group By和聚合函数？

xzlaal3s 于 2022-11-21 发布在 Spark

关注(0)|答案(1)|浏览(193)

我正在寻找如何在Pyspark中一起使用Group by Aggregate函数的解决方案？我的Dataframe看起来像这样：

df = sc.parallelize([
    ('23-09-2020', 'CRICKET'),
    ('25-11-2020', 'CRICKET'),
    ('13-09-2021', 'FOOTBALL'),
    ('20-11-2021', 'BASKETBALL'),
    ('12-12-2021', 'FOOTBALL')]).toDF(['DATE', 'SPORTS_INTERESTED'])

我想在SPORTS_INTERSTED列上应用group by，并从DATE列中选择日期的MIN。下面是我使用的查询

from pyspark.sql.functions import  min
df=df.groupby('SPORTS_INTERESTED').agg(count('SPORTS_INTERESTED').alias('FIRST_COUNT'),(F.min('DATE').alias('MIN_OF_DATE_COLUMN'))).filter((col('FIRST_COUNT')> 1))

但是，当我应用上述查询时，我不知道为什么在输出值DESIRED OUTPUT中给出MAX日期而不是MIN日期

## +-----------------+-------------------+
## |SPORTS_INTERESTED| MIN_OF_DATE_COLUMN|    
## +------+----------+-------------------+
## |  CRICKET        |23-09-2020         |
## +------+----------+-------------------+
## | FOOTBALL        |13-09-2021         |
   +-----------------+-------------------+

我得到的输出：

## +-----------------+----------------------+
    ## |SPORTS_INTERESTED| MIN_OF_DATE_COLUMN|    
    ## +------+----------+-------------------+
    ## |  CRICKET        |25-11-2020         |
    ## +------+----------+-------------------+
    ## | FOOTBALL        |12-12-2021         |
       +-----------------+-------------------+

两列都是字符串数据类型

pyspark

来源：https://stackoverflow.com/questions/74500675/use-group-by-and-aggregate-function-in-pyspark

1条答案

按热度按时间

v9tzhpje1#

首先，将字符串转换为日期格式，然后应用min：

import pyspark.sql.functions as F

df = spark.createDataFrame(data=[
    ('23-09-2020', 'CRICKET'),
    ('25-11-2020', 'CRICKET'),
    ('13-09-2021', 'FOOTBALL'),
    ('20-11-2021', 'BASKETBALL'),
    ('12-12-2021', 'FOOTBALL')    
], schema=['DATE', 'SPORTS_INTERESTED'])

df = df.withColumn("DATE", F.to_date("DATE", format="dd-MM-yyyy"))
df = df.groupBy("SPORTS_INTERESTED").agg(F.min("DATE").alias("MIN_OF_DATE"))

[Out]:
+-----------------+-----------+
|SPORTS_INTERESTED|MIN_OF_DATE|
+-----------------+-----------+
|BASKETBALL       |2021-11-20 |
|FOOTBALL         |2021-09-13 |
|CRICKET          |2020-09-23 |
+-----------------+-----------+

赞(0）回复(0）举报 2022-11-21

我来回答

在pyspark中使用Group By和聚合函数？

1条答案

相关问题

热门标签

最新问答