spark：有没有方法对不区分大小写的列进行排序(sort或orderby)

jbose2ul 于 2021-05-27 发布在 Spark

关注(0)|答案(2)|浏览(592)

我需要对行进行不区分大小写的排序。
我有这样的数据：

+---+---------------+--------------------+--------------------+------+--------------+
| id|      full_name|           job_title|               email|gender|    ip_address|
+---+---------------+--------------------+--------------------+------+--------------+
| 73|     Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54|
| 74|      Lois Hart|        Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236|
| 75|    Thomas Hall|    Senior Developer|   thall22@wired.com|  Male|76.255.197.231|
| 76|  Ernest Romero|             Teacher|eromero23@amazon....|  Male|  99.21.57.239|
| 77|  Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230|
| 78|Jacqueline Cruz|account Represent...|     jcruz25@cdc.gov|Female| 167.49.98.213|
| 79|    Sara Martin|        Geologist IV|    smartin26@a8.net|Female| 10.145.49.204|
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...|  Male| 138.251.4.102|
| 81|      Fred Dean|Nuclear Power Eng...|fdean28@kickstart...|  Male| 173.10.122.12|
| 82|   Ralph Greene|       Senior Editor|rgreene29@omnitur...|  Male| 57.230.33.105|
+---+---------------+--------------------+--------------------+------+--------------+

当我根据 job_title 使用 df.orderBy('job_title') . 这就是我得到的。

+---+---------------+--------------------+--------------------+------+--------------+
| id|      full_name|           job_title|               email|gender|    ip_address|
+---+---------------+--------------------+--------------------+------+--------------+
| 77|  Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230|
| 73|     Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54|
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...|  Male| 138.251.4.102|
| 74|      Lois Hart|        Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236|
| 79|    Sara Martin|        Geologist IV|    smartin26@a8.net|Female| 10.145.49.204|
| 81|      Fred Dean|Nuclear Power Eng...|fdean28@kickstart...|  Male| 173.10.122.12|
| 75|    Thomas Hall|    Senior Developer|   thall22@wired.com|  Male|76.255.197.231|
| 82|   Ralph Greene|       Senior Editor|rgreene29@omnitur...|  Male| 57.230.33.105|
| 76|  Ernest Romero|             Teacher|eromero23@amazon....|  Male|  99.21.57.239|
| 78|Jacqueline Cruz|account Represent...|     jcruz25@cdc.gov|Female| 167.49.98.213|
+---+---------------+--------------------+--------------------+------+--------------+

但我需要的是

+---+---------------+--------------------+--------------------+------+--------------+
| id|      full_name|           job_title|               email|gender|    ip_address|
+---+---------------+--------------------+--------------------+------+--------------+
| 78|Jacqueline Cruz|account Represent...|     jcruz25@cdc.gov|Female| 167.49.98.213|
| 77|  Irene Bradley| Assistant Professor|ibradley24@squido...|Female| 16.51.179.230|
| 73|     Tina Mccoy|Desktop Support T...|tmccoy20@techcrun...|Female| 23.196.170.54|
| 80| Johnny Bradley| Executive Secretary|jbradley27@cocolo...|  Male| 138.251.4.102|
| 74|      Lois Hart|        Food Chemist|lhart21@mapquest.com|Female| 145.52.30.236|
| 79|    Sara Martin|        Geologist IV|    smartin26@a8.net|Female| 10.145.49.204|
| 81|      Fred Dean|Nuclear Power Eng...|fdean28@kickstart...|  Male| 173.10.122.12|
| 75|    Thomas Hall|    Senior Developer|   thall22@wired.com|  Male|76.255.197.231|
| 82|   Ralph Greene|       Senior Editor|rgreene29@omnitur...|  Male| 57.230.33.105|
| 76|  Ernest Romero|             Teacher|eromero23@amazon....|  Male|  99.21.57.239|
+---+---------------+--------------------+--------------------+------+--------------+

apache-spark apache-spark-sql

来源：https://stackoverflow.com/questions/40228762/spark-is-there-any-method-to-sort-sort-or-orderby-on-column-with-case-insens

2条答案

按热度按时间

pbpqsu0x1#

简单的解决方法是创建一个 job_title_lower_case 列，然后按该列排序。最后，只需删除新列。

赞(0）回复(0）举报 2021-05-27

mf98qq942#

可以将计算表达式作为参数传递给 orderBy . 所以你可以导入 lower 功能：

from pyspark.sql.functions import col, lower

并用它来 Package 列名

df.orderBy(lower(col("job_title")))

赞(0）回复(0）举报 2021-05-27

我来回答

spark：有没有方法对不区分大小写的列进行排序(sort或orderby)

2条答案

相关问题

热门标签

最新问答