我正在尝试使用docker-compose设置Apache Zeppelin和Apache Flink。
我可以让Zeppelin在本地模式下与Flink很好地工作,当Zeppelin生成本地Flink集群时,但当Zeppelin连接到现有集群时,不能在远程模式下工作。
我决定在Zeppelin上使用Flink 1.12.0**,因为这是我能在本地模式下工作的唯一版本。
我使用下面的docker-compose.yml,它主要受Flink文档的启发:https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/
version: "2.2"
services:
jobmanager:
image: flink:1.12.0
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.12.0
depends_on:
- jobmanager
command: taskmanager
scale: 1
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
taskmanager.numberOfTaskSlots: 2
zeppelin:
image: apache/zeppelin:0.10.1
hostname: zeppelin
depends_on:
- taskmanager
container_name: zeppelin
ports:
- "9080:8080"
- "9081:8081"
volumes:
- /home/administrator/flink/flink-1.12.0:/opt/flink
environment:
FLINK_HOME: /opt/flink
我在端口9081上使用Zeppelin UI来配置远程Flink集群。参数为:
flink.execution.mode: remote
flink.execution.remote.host: jobmanager
flink.execution.remote.port: 8081
我在执行“FlinksBasic”笔记本Batch WordCount单元格时得到此错误。
data: org.apache.flink.api.scala.DataSet[String] = org.apache.flink.api.scala.DataSet@6adbed6a
org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: c55932eec9cc32b8df64630be00b2532)
at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:119 undefined)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616 undefined)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591 undefined)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488 undefined)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975 undefined)
at org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$22(RestClusterClient.java:602 undefined)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774 undefined)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750 undefined)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488 undefined)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975 undefined)
at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:379 undefined)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774 undefined)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750 undefined)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488 undefined)
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575 undefined)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943 undefined)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456 undefined)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149 undefined)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 undefined)
at java.lang.Thread.run(Thread.java:748 undefined)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147 undefined)
at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:117 undefined)
... 19 more
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116 undefined)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78 undefined)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:224 undefined)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:217 undefined)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:208 undefined)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:610 undefined)
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89 undefined)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:419 undefined)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 undefined)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 undefined)
at java.lang.reflect.Method.invoke(Method.java:498 undefined)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:286 undefined)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:201 undefined)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74 undefined)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154 undefined)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26 undefined)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21 undefined)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123 undefined)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122 undefined)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21 undefined)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171 undefined)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172 undefined)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172 undefined)
at akka.actor.Actor.aroundReceive(Actor.scala:517 undefined)
at akka.actor.Actor.aroundReceive$(Actor.scala:515 undefined)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225 undefined)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592 undefined)
at akka.actor.ActorCell.invoke(ActorCell.scala:561 undefined)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258 undefined)
at akka.dispatch.Mailbox.run(Mailbox.scala:225 undefined)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235 undefined)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260 undefined)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339 undefined)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979 undefined)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107 undefined)
Caused by: java.lang.RuntimeException: The initialization of the DataSource's outputs caused an error: Could not read the user code wrapper: org.apache.flink.api.scala.typeutils.CaseClassTypeInfo; local class incompatible: stream classdesc serialVersionUID = -3156538608383968041, local class serialVersionUID = -8170136754691974512
at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:110 undefined)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722 undefined)
at org.apache.flinktime.taskmanager.Task.run.run(Task.java:547 undefined)
at java.lang.Thread.run(Thread.java:748 undefined)
Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could not read the user code wrapper: org.apache.flink.api.scala.typeutils.CaseClassTypeInfo; local class incompatible: stream classdesc serialVersionUID = -3156538608383968041, local class serialVersionUID = -8170136754691974512
at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:290 undefined)
at org.apache.flink.runtime.operators.BatchTask.instantiateUserCode(BatchTask.java:1464 undefined)
at org.apache.flink.runtime.operators.chaining.SynchronousChainedCombineDriver.setup(SynchronousChainedCombineDriver.java:90 undefined)
at org.apache.flink.runtime.operators.chaining.ChainedDriver.setup(ChainedDriver.java:91 undefined)
at org.apache.flink.runtime.operators.BatchTask.initOutputs(BatchTask.java:1333 undefined)
at org.apache.flink.runtime.operators.DataSourceTask.initOutputs(DataSourceTask.java:317 undefined)
at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:107 undefined)
... 3 more
Caused by: java.io.InvalidClassException: org.apache.flink.api.scala.typeutils.CaseClassTypeInfo; local class incompatible: stream classdesc serialVersionUID = -3156538608383968041, local class serialVersionUID = -8170136754691974512
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699 undefined)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003 undefined)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850 undefined)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003 undefined)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850 undefined)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160 undefined)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667 undefined)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405 undefined)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329 undefined)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187 undefined)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667 undefined)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405 undefined)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329 undefined)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187 undefined)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667 undefined)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503 undefined)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461 undefined)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:576 undefined)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:562 undefined)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:550 undefined)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:511 undefined)
at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288 undefined)
... 9 more
这是我尝试的
- 在Flink的日志文件中查看其他端口以尝试代替8081。没有一个成功。
- 尝试了Fink的其他版本,但得到了不同的错误,甚至无法让Zeppelin在本地模式下工作。
有什么建议吗?
非常感谢,米歇尔
1条答案
按热度按时间ua4mk5z41#
我设法让Apache Zeppelin使用Apache Flink的远程示例,方法是在要下拉的图像中显式地提到Scala的版本(2.11)。
此外,我升级到了Apache Flink的最新版本(1.14.6)。我知道1.14.6是Scala 2.11附带的Flink的最后一个版本。
下面是我使用的docker-compose.yml:
Apache Zeppelin可以通过HTTP//<server_ip>:9081访问,并配置了以下设置: