使用“when otherwise”时出现超过64 kb的错误

ztigrdn8 于 2021-05-27 发布在 Spark

关注(0)|答案(1)|浏览(421)

当我在scala中运行这个spark代码时：

df.withColumn(x, when(col(x).isin(values:_*),col(x)).otherwise(lit(null).cast(StringType)))

我面临这个错误：

java.lang.RuntimeException: Compiling "GeneratedClass": Code of method
 "apply(Lorg/apache/spark/sql/catalyst/InternalRow;)Lorg/apache/spark/sql
 /catalyst /expressions/UnsafeRow;" of class
 "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
 grows beyond 64 KB
        at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:361)
        at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:234)

df:spark数据集
x:stringtype列，每行类似“us，washington，seattle”
值：数组[字符串]

scala apache-spark apache-spark-sql compiler-errors

来源：https://stackoverflow.com/questions/63093710/grows-beyond-64-kb-error-when-using-when-otherwise