when+otherwise 我要做的是：

import pyspark.sql.functions as F

pyspark_df.select([F.when(F.col(i).cast("Integer") <0 , None).otherwise(F.col(i)).alias(i)
                  for i in df.columns]).show()

+------+------+-----+
|Number|Letter|Value|
+------+------+-----+
|     1|     A|   30|
|     2|  null|   30|
|  null|     B|   30|
|  null|     A| null|
+------+------+-----+

另一种不那么冗长的方法是使用 replace .

pyspark_df.replace(-1,None).replace('-1',None).show()

使用 reduce 申请 when+otherwise 在dataframe的所有列上。

df.show()

# +------+------+-----+

# |Number|Letter|Value|

# +------+------+-----+

# |     1|     A|   30|

# |     2|    -1|   30|

# |    -1|     B|   30|

# +------+------+-----+

from functools import reduce

(reduce(lambda new_df, col_name: new_df.withColumn(col_name, when(col(col_name)== '-1',lit(None)).otherwise(col(col_name))),df.columns,df)).show()

# +------+------+-----+

# |Number|Letter|Value|

# +------+------+-----+

# |     1|     A|   30|

# |     2|  null|   30|

# |  null|     B|   30|

# +------+------+-----+

可以扫描所有列并替换 -1 没有的：

import pyspark.sql.functions as F

for x in pyspark_df.columns:
    pyspark_df = pyspark_df.withColumn(x, F.when(F.col(x)==-1, F.lit(None)).otherwise(F.col(x)))

pyspark_df.show()

输出：

+------+------+-----+
|Number|Letter|Value|
+------+------+-----+
|     1|     A|   30|
|     2|  null|   30|
|  null|     B|   30|
|  null|     A| null|
+------+------+-----+