spark中的cancelledkeyexception和closedchannelexception

rqenqsqc  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(188)

我不停地 CancelledKeyExpection 以及 ClosedChannelException 我的星火计划。我试图通过增加 spark.network.timeout 至600和 spark.akka.frameSize 到1280。而且,我已经为每个执行器和驱动程序分配了足够的内存。另外,我使用的是spark的最新版本,即spark 1.5。我甚至尝试了1.4,但得到了相同的错误。那么,有什么问题吗?快把我逼疯了!错误如下所示。

java.nio.channels.CancelledKeyException
        at org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:461)
        at org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:193)
15/09/16 16:27:38 INFO nio.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
15/09/16 16:27:38 INFO nio.ConnectionManager: Removing SendingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
15/09/16 16:27:38 ERROR nio.SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
        at org.apache.spark.network.nio.SendingConnection.read(Connection.scala:421)
        at org.apache.spark.network.nio.ConnectionManager$$anon$9.run(ConnectionManager.scala:259)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
15/09/16 16:27:38 ERROR nio.ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
        at org.apache.spark.network.nio.SendingConnection.read(Connection.scala:421)
        at org.apache.spark.network.nio.ConnectionManager$$anon$9.run(ConnectionManager.scala:259)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题