目前，我正在尝试将所有功能合并到一个名为“性别”的专栏中。我已经用Pandas成功地做到了这一点，但现在我想用Pypark做到这一点，它是有点不同的Pandas相比。我无法调用函数 .apply 在Pypark。
这是我用Pandas做的版本：

df['Gender'] = df['Gender'].str.lower()
male = ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"]
female = ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"]
other = ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]
new_df['Gender'] = new_df['Gender'].apply(lambda x:"Male" if x in male else x)
new_df['Gender'] = new_df['Gender'].apply(lambda x:"Female" if x in female else x)
new_df['Gender'] = new_df['Gender'].apply(lambda x:"Other" if x in other else x)

这是我尝试使用pyspark复制的版本，但是我很难将所有转换的值放回“gender”列：

from pyspark.sql.functions import lower, col, udf
import pyspark.sql.functions as f 
na_df = na_df.withColumn('Gender', lower(col('Gender')))
Male = ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"]
Female = ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"]
Other = ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]
na_df2 = na_df.withColumn('Gender',f.when(f.col('Gender').isin(Male),f.lit('Male')).\
when(f.col('Gender').isin(Other),f.lit('Other')).\
when(f.col('Gender').isin(Female),f.lit('Female')).\
otherwise(f.col('Gender'))).show()
na_df2.select('Gender').distinct().show()

这是我尝试的解决方案的另一个版本，但它给了我一个错误：无法将列转换为bool：

from pyspark.sql.functions import lower, col, udf
na_df = na_df.withColumn('Gender', lower(col('Gender')))
genders = {
    'Male': ["male", "m", "male-ish", "maile", "mal", "male (cis)", "make", "male ", "man", "msle", "mail", "malr","cis man", "cis male"],
    'Female': ["cis female", "f", "female", "woman",  "femake", "female ","cis-female/femme", "female (cis)", "femail", "trans-female",  "trans woman", "female (trans)"],
    'Other': ["non-binary", "nah", "all", "enby", "fluid", "genderqueer", "androgyne", "agender", "male leaning androgynous", "guy (-ish) ^_^", "neuter", "queer", "ostensibly male, unsure what that really means", "queer/she/they", "something kinda male?", "a little about you", "p"]
}
na_df.withColumn('Gender', (lambda x: [g for g in genders if x in genders[g]][0])(col('Gender'))).show()

结果，我得到的是，列“性别”还没有更新，所以请就我可以做什么来解决这个问题的建议。提前谢谢！