flink无法序列化scala类/无法序列化的任务

clj7thdc  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(612)

我有一个scala类,有两个字段是VAL,但flink说它没有setter。因此,任务是不可序列化的。
我用var尝试了setter,但是它说是重复setter。VAL是公开的,那么为什么它要求设置者呢。flink版本1.1.0

class Impression(val map: Map[String, String],val keySet:Set[String])

我的代码:

val preAggregate = stream
.filter(impression => {
    true
})
 .map(impression => {
  val xmap = impression.map
  val values = valFunction(xmap)
  new ImpressionRecord(impression, values._1, values._2, values._3)
})

例外情况:


**class Impression does not contain a setter for field map**

19:54:49.995 [main] INFO  o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid POJO type
19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - accessedFields: Map(class  -> Set())
Exception in thread "main"**org.apache.flink.api.common.InvalidProgramException: Task not serializable
at**org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172)
at )
Caused by: java.io.NotSerializableException: org.apache.flink.streaming.api.scala.DataStream
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301)
at org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170)
... 18 more
ddarikpa

ddarikpa1#

尽量不要直接使用类的字段,而是将其 Package 在一个方法中,这样就差不多可以了。

相关问题