Sentinel webflux io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1

j0pj023g  于 2022-10-19  发布在  .NET
关注(0)|答案(1)|浏览(370)

Type: bug report

Describe what happened (or what feature you want)

We use sentinel in Spring Cloud Gateway, and found that when backend response incorrently,  SentinelReactorSubscriber will produce `IllegalReferenceCountException: refCnt: 0, decrement: 1`.

StackTrace

Operator called default onErrorDropped
reactor.core.Exceptions$BubblingException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at reactor.core.Exceptions.bubble(Exceptions.java:173)
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:635)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:159)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:139)
	at com.alibaba.csp.sentinel.adapter.reactor.SentinelReactorSubscriber.hookOnComplete(SentinelReactorSubscriber.java:136)
	at com.alibaba.csp.sentinel.adapter.reactor.InheritableBaseSubscriber.onComplete(InheritableBaseSubscriber.java:191)
	at com.alibaba.csp.sentinel.adapter.reactor.SentinelReactorSubscriber.onComplete(SentinelReactorSubscriber.java:37)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:207)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
	at org.springframework.http.server.reactive.ChannelSendOperator$WriteCompletionBarrier.onComplete(ChannelSendOperator.java:423)
	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:191)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:207)
	at reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:345)
	at reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419)
	at reactor.netty.channel.MonoSendMany$SendManyInner.trySuccess(MonoSendMany.java:562)
	at reactor.netty.channel.MonoSendMany$SendManyInner.trySuccess(MonoSendMany.java:114)
	at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
	at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
	at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
	at io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:364)
	at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260)
	at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:471)
	at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:429)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:953)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:917)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)
	at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
	at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
	at reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:758)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
	at io.netty.buffer.AbstractPooledDerivedByteBuf.deallocate(AbstractPooledDerivedByteBuf.java:86)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.handleRelease(AbstractReferenceCountedByteBuf.java:110)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
	at org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:320)
	at org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter.lambda$filter$0(RemoveCachedBodyFilter.java:43)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:156)
	... 46 more

Describe what you expected to happen

Can sentinel handle this kind of exception, or should I handle it through Hooks.onErrorDropped by myself.

How to reproduce it (as minimally and precisely as possible)

  1. backend service response incorrent.

Tell us your environment

spring-cloud-starter-gateway : 2.2.8.RELEASE
spring-cloud-alibaba-sentinel-gateway : 2.2.3.RELEASE
sentinel-reactor-adapter : 1.8.0

fiei3ece

fiei3ece1#

Hi, it seems like this error was emitted by Reactor Netty and caught by Sentinel Reactor Adapter (and "bubbled up"). Maybe we could provide an option to drop the exception by default.

相关问题