我找不到任何解决问题的办法。
如果我试图定义spark udf函数(pyspark),例如:
@udf(returnType=IntegerType())
def _return_an_integer(y):
return y + random.randint(1, 99)
df = df.withColumn('randomInteger', _return_an_integer(col('1')))
我收到错误消息:
TypeError: udf() missing 1 required positional argument: 'f'
有人能解决这个问题吗?
提前谢谢!
编辑:不带注解:
Py4JError: An error occurred while calling None.org.apache.spark.sql.execution.python.UserDefinedPythonFunction. Trace:
py4j.Py4JException: Constructor org.apache.spark.sql.execution.python.UserDefinedPythonFunction([class java.lang.String, class org.apache.spark.api.python.PythonFunction, class org.apache.spark.sql.types.IntegerType$]) does not exist
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:179)
at py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:196)
at py4j.Gateway.invoke(Gateway.java:237)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
1条答案
按热度按时间j8yoct9x1#
在尝试了很多东西之后,问题是我的pyspark版本与spark版本不匹配。