如何在pyspark上将所有函数组合成一列？

目前，我正在尝试将所有功能合并到一个名为“性别”的专栏中。我已经用Pandas成功地做到了这一点，但现在我想用Pypark做到这一点，它是有点不同的Pandas相比。我无法调用函数 .apply 在Pypark。
这是我用Pandas做的版本：

df['Gender'] = df['Gender'].str.lower()

male = ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"]
female = ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"]
other = ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]

new_df['Gender'] = new_df['Gender'].apply(lambda x:"Male" if x in male else x)
new_df['Gender'] = new_df['Gender'].apply(lambda x:"Female" if x in female else x)
new_df['Gender'] = new_df['Gender'].apply(lambda x:"Other" if x in other else x)

这是我尝试使用pyspark复制的版本，但是我很难将所有转换的值放回“gender”列：

from pyspark.sql.functions import lower, col, udf
import pyspark.sql.functions as f 

na_df = na_df.withColumn('Gender', lower(col('Gender')))

Male = ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"]
Female = ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"]
Other = ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]

na_df2 = na_df.withColumn('Gender',f.when(f.col('Gender').isin(Male),f.lit('Male')).\
when(f.col('Gender').isin(Other),f.lit('Other')).\
when(f.col('Gender').isin(Female),f.lit('Female')).\
otherwise(f.col('Gender'))).show()

na_df2.select('Gender').distinct().show()

这是我尝试的解决方案的另一个版本，但它给了我一个错误：无法将列转换为bool：

from pyspark.sql.functions import lower, col, udf

na_df = na_df.withColumn('Gender', lower(col('Gender')))

genders = {
    'Male': ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"],
    'Female': ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"],
    'Other': ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]
}

na_df.withColumn('Gender', (lambda x: [g for g in genders if x in genders[g]][0])(col('Gender'))).show()

结果，我得到的是，列“性别”还没有更新，所以请就我可以做什么来解决这个问题的建议。提前谢谢！

您可以通过在函数

import pyspark.sql.functions as f
+---+----------+
| id|    gender|
+---+----------+
|  1|      male|
|  1|         m|
|  1|  male-ish|
|  1|     maile|
|  1|       mal|
|  1|male (cis)|
|  1|      make|
|  1|     male |
|  1|       man|
|  1|      msle|
|  1|      mail|
|  1|      malr|
|  1|   cis man|
|  1|  cis male|
|  1|cis female|
|  1|         f|
|  1|    female|
|  1|     woman|
|  1|    femake|
|  1|   female |
+---+----------+

df = df.withColumn('gender',f.when(f.col('gender').isin(male),f.lit('Male')).\
when(f.col('gender').isin(other),f.lit('Other')).\
when(f.col('gender').isin(female),f.lit('Female')).\
otherwise(f.col('gender')))

df.select('Gender').distinct().show()
+---+------+
| id|gender|
+---+------+
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|  Male|
|  1|Female|
|  1|Female|
|  1|Female|
|  1|Female|
|  1|Female|
|  1|Female|
+---+------+

如何在pyspark上将所有函数组合成一列？

1条答案

相关问题

热门标签

最新问答