当您将这两个列移动到相同的 Dataframe 时，可以使用类似以下pandas_udf的代码。pandas_udf进行了矢量化以提高性能。它与普通的Spark udf不同。

输入：

from pyspark.sql import functions as F
import pandas as pd
from fuzzywuzzy import fuzz

df = spark.createDataFrame([('danial khilji', 'danial'), ('a','as',)], ['col1', 'col2'])
df.show()

# +-------------+------+

# |         col1|  col2|

# +-------------+------+

# |danial khilji|danial|

# |            a|    as|

# +-------------+------+

脚本：

@F.pandas_udf('long')
def fuzz_ratio(c1: pd.Series, c2: pd.Series) -> pd.Series:
    return pd.concat([c1, c2], axis=1).apply(lambda x: fuzz.token_sort_ratio(x[0], x[1]), axis=1)

df.withColumn('fuzz_ratio', fuzz_ratio('col1', 'col2')).show()

# +-------------+------+----------+

# |         col1|  col2|fuzz_ratio|

# +-------------+------+----------+

# |danial khilji|danial|        63|

# |            a|    as|        67|

# +-------------+------+----------+