zeppelin段落使用pyspark在1小时后超时

i7uaboj4  于 2021-07-12  发布在  Spark
关注(0)|答案(0)|浏览(207)

我正在运行ubuntu16.04的docker容器中使用zeppelin0.8.2
长时间运行的pyspark段落都会在1小时后停止,并显示如下消息: Took 1 hrs 0 min 0 sec. Last updated by anonymous at February 09 2021, 10:24:25 PM. 我已经确认spark仍然在后台运行我的代码,所以似乎齐柏林飞艇与spark的连接在1小时后就被切断了。
我试着编辑 zeppelin.interpreter.lifecyclemanager.timeout.threshold 变量输入 zeppelin-site.xml 但这对问题没有影响,即使更改在web ui中清晰可见(屏幕截图)。其他值在zeppelin读取和执行的xml文件中设置。我还验证了 zeppelin-env.sh 与xml文件中设置的值不冲突。
有时,段落输出包含以下错误消息:

org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:449)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我也在齐柏林飞艇的官方问题追踪网站上发布了这个https://issues.apache.org/jira/browse/zeppelin-5279

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题