livy开始抛出java.lang.illegalstateexception:未能创建子事件循环

6ovsh4lw  于 2021-07-13  发布在  Spark
关注(0)|答案(0)|浏览(419)

我正在使用emr 6.2.0和livy 0.7.0
我有一个工人，他一直在启动spark会话，并对其完成情况进行投票。这运行了一段时间，但突然利维开始扔

  1. java.lang.IllegalStateException: failed to create a child event loop
  2.         at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
  3.         at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
  4.         at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
  5.         at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
  6.         at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
  7.         at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
  8.         at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
  9.         at org.apache.livy.rsc.RSCClient.<init>(RSCClient.java:81)
  10.         at org.apache.livy.rsc.RSCClientFactory.createClient(RSCClientFactory.java:76)
  11.         at org.apache.livy.LivyClientBuilder.build(LivyClientBuilder.java:130)
  12.         at org.apache.livy.server.interactive.InteractiveSession$.$anonfun$create$1(InteractiveSession.scala:111)
  13.         at scala.Option.orElse(Option.scala:447)
  14.         at org.apache.livy.server.interactive.InteractiveSession$.create(InteractiveSession.scala:81)
  15.         at org.apache.livy.server.interactive.InteractiveSessionServlet.createSession(InteractiveSessionServlet.scala:55)
  16.         at org.apache.livy.server.interactive.InteractiveSessionServlet.createSession(InteractiveSessionServlet.scala:40)
  17.         at org.apache.livy.server.SessionServlet.$anonfun$new$16(SessionServlet.scala:139)
  18.         at org.scalatra.ScalatraBase.liftAction(ScalatraBase.scala:280)
  19.         at org.scalatra.ScalatraBase.$anonfun$invoke$1(ScalatraBase.scala:274)
  20.         at org.scalatra.ApiFormats.withRouteMultiParams(ApiFormats.scala:183)
  21.         at org.scalatra.ApiFormats.withRouteMultiParams$(ApiFormats.scala:169)
  22.         at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39)
  23.         at org.scalatra.ScalatraBase.invoke(ScalatraBase.scala:273)
  24.         at org.scalatra.ScalatraBase.invoke$(ScalatraBase.scala:272)
  25.         at org.apache.livy.server.JsonServlet.invoke(JsonServlet.scala:39)
  26.         at org.scalatra.ScalatraBase.$anonfun$runRoutes$3(ScalatraBase.scala:248)
  27.         at scala.Option.flatMap(Option.scala:271)
  28.         at org.scalatra.ScalatraBase.$anonfun$runRoutes$1(ScalatraBase.scala:246)
  29.         at scala.collection.immutable.Stream.flatMap(Stream.scala:493)
  30.         at org.scalatra.ScalatraBase.runRoutes(ScalatraBase.scala:245)
  31.         at org.scalatra.ScalatraBase.runRoutes$(ScalatraBase.scala:243)
  32.         at org.apache.livy.server.JsonServlet.runRoutes(JsonServlet.scala:39)
  33.         at org.scalatra.ScalatraBase.runActions$1(ScalatraBase.scala:169)
  34.         at org.scalatra.ScalatraBase.$anonfun$executeRoutes$6(ScalatraBase.scala:181)
  35.         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  36.         at org.scalatra.ScalatraBase.cradleHalt(ScalatraBase.scala:199)
  37.         at org.scalatra.ScalatraBase.executeRoutes(ScalatraBase.scala:181)
  38.         at org.scalatra.ScalatraBase.executeRoutes$(ScalatraBase.scala:150)
  39.         at org.apache.livy.server.JsonServlet.executeRoutes(JsonServlet.scala:39)
  40.         at org.scalatra.ScalatraBase.$anonfun$handle$1(ScalatraBase.scala:123)
  41.         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  42.         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
  43.         at org.scalatra.DynamicScope.withResponse(DynamicScope.scala:79)
  44.         at org.scalatra.DynamicScope.withResponse$(DynamicScope.scala:77)
  45.         at org.apache.livy.server.JsonServlet.withResponse(JsonServlet.scala:39)
  46.         at org.scalatra.DynamicScope.$anonfun$withRequestResponse$1(DynamicScope.scala:59)
  47.         at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
  48.         at org.scalatra.DynamicScope.withRequest(DynamicScope.scala:70)
  49.         at org.scalatra.DynamicScope.withRequest$(DynamicScope.scala:68)
  50.         at org.apache.livy.server.JsonServlet.withRequest(JsonServlet.scala:39)
  51.         at org.scalatra.DynamicScope.withRequestResponse(DynamicScope.scala:58)
  52.         at org.scalatra.DynamicScope.withRequestResponse$(DynamicScope.scala:56)
  53.         at org.apache.livy.server.JsonServlet.withRequestResponse(JsonServlet.scala:39)
  54.         at org.scalatra.ScalatraBase.handle(ScalatraBase.scala:123)
  55.         at org.scalatra.ScalatraBase.handle$(ScalatraBase.scala:119)
  56.         at org.apache.livy.server.JsonServlet.org$scalatra$servlet$ServletBase$$super$handle(JsonServlet.scala:39)
  57.         at org.scalatra.servlet.ServletBase.handle(ServletBase.scala:53)
  58.         at org.scalatra.servlet.ServletBase.handle$(ServletBase.scala:46)
  59.         at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:45)
  60.         at org.scalatra.MethodOverride.handle(MethodOverride.scala:35)
  61. ...skipping...
  62. 21/02/05 14:15:57 INFO LineBufferedStream: java.lang.IllegalArgumentException: Unexpected client ID '40f2d9b7-6460-45b1-9521-6a92ff733766' in SASL handshake.
  63. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.Utils.checkArgument(Utils.java:42)
  64. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.RpcServer$SaslServerHandler.update(RpcServer.java:288)
  65. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:61)
  66. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:36)
  67. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
  68. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  69. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  70. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
  71. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
  72. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
  73. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
  74. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  75. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  76. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
  77. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  78. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  79. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  80. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  81. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
  82. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
  83. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
  84. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
  85. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
  86. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
  87. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  88. 21/02/05 14:15:57 INFO LineBufferedStream:      at java.lang.Thread.run(Thread.java:748)
  89. 21/02/05 14:15:57 INFO LineBufferedStream: 21/02/05 14:15:57 WARN DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
  90. 21/02/05 14:15:57 INFO LineBufferedStream: java.lang.IllegalArgumentException: Unexpected client ID '40f2d9b7-6460-45b1-9521-6a92ff733766' in SASL handshake.
  91. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.Utils.checkArgument(Utils.java:42)
  92. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.RpcServer$SaslServerHandler.update(RpcServer.java:288)
  93. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:61)
  94. 21/02/05 14:15:57 INFO LineBufferedStream:      at org.apache.livy.rsc.rpc.SaslHandler.channelRead0(SaslHandler.java:36)
  95. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
  96. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  97. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  98. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
  99. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
  100. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
  101. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
  102. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  103. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  104. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
  105. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  106. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  107. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  108. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  109. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
  110. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
  111. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
  112. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
  113. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
  114. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
  115. 21/02/05 14:15:57 INFO LineBufferedStream:      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  116. 21/02/05 14:15:57 INFO LineBufferedStream:      at java.lang.Thread.run(Thread.java:748)
  117. 21/02/05 14:15:58 WARN ContextLauncher: Child process exited with code 1.
  118. 21/02/05 14:15:58 ERROR rejectedExecution: Failed to submit a listener notification task. Event loop shut down?
  119. java.util.concurrent.RejectedExecutionException: event executor terminated
  120.         at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
  121.         at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
  122.         at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
  123.         at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
  124.         at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
  125.         at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
  126.         at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
  127.         at org.apache.livy.rsc.ContextLauncher$ChildProcess.fail(ContextLauncher.java:417)
  128.         at org.apache.livy.rsc.ContextLauncher$ChildProcess.access$1100(ContextLauncher.java:369)
  129.         at org.apache.livy.rsc.ContextLauncher$ChildProcess$1.run(ContextLauncher.java:400)
  130.         at org.apache.livy.rsc.ContextLauncher$ChildProcess$2.run(ContextLauncher.java:451)
  131.         at java.lang.Thread.run(Thread.java:748)

同时运行的会话并不多。最多可同时运行5个会话。作业完成后，工作进程将删除所有旧会话，以确保一次打开的会话不会太多。
糟糕的是，在worker停止创建新会话后，它不会自动恢复。只在livy服务器重新启动后工作。

hadoopnettyapache-sparkamazon-emrlivy

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com