LittleEndianByteBufAllocator uses UnpooledByteBufAllocator is not optimal and potentially causing OOM error #435
Description
MySQLConnectionHandler uses the default LittleEndianByteBufAllocator for netty
this.bootstrap.option(ChannelOption.ALLOCATOR, LittleEndianByteBufAllocator.INSTANCE)
LittleEndianByteBufAllocator uses UnpooledByteBufAllocator as default
class LittleEndianByteBufAllocator(private val allocator: UnpooledByteBufAllocator = UnpooledByteBufAllocator(false)) :
ByteBufAllocator by allocator {
companion object {
val INSTANCE = LittleEndianByteBufAllocator()
}
I guess the jasync-sql team want to use heap memory instead of direct memory here.
However, on the netty side, netty's SSLHandler will choose to use direct memory or not by himself
io.netty.handler.ssl.SSLHandler
private ByteBuf allocate(ChannelHandlerContext ctx, int capacity) {
ByteBufAllocator alloc = ctx.alloc();
if (engineType.wantsDirectBuffer) {
return alloc.directBuffer(capacity);
} else {
return alloc.buffer(capacity);
}
}
And unfortunately, in some cases, like when using tcnative libaray, netty will choose to use directBuffer explicitly. This is not efficient to allocate and de-allocate directly using unpooled allocator. And also, using Unpooled direct memory sometimes causing OutOfMemory error because large amount of direct memory has already got pooled by netty.
<mysql-connection-35> Transport failure (mysql.MySQLConnection:)
java.lang.OutOfMemoryError: Cannot reserve 4110 bytes of direct buffer memory (allocated: 134217321, limit: 134217728)
at java.base/java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]
at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]
at io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:567) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledDirectByteBuf.allocateDirectBuffer(UnpooledDirectByteBuf.java:121) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf.allocateDirectBuffer(UnpooledByteBufAllocator.java:207) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:66) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:42) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf.<init>(UnpooledByteBufAllocator.java:202) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:93) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.2.3.Final.jar!/:4.2.3.Final]
at com.github.jasync.sql.db.mysql.codec.LittleEndianByteBufAllocator.directBuffer(LittleEndianByteBufAllocator.kt:43) ~[jasync-mysql-2.2.4.jar!/:?]
at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2442) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1502) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1398) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1449) ~[netty-handler-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-base-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:356) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:167) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:381) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:575) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:550) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:491) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:468) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:207) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:178) ~[netty-transport-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.2.3.Final.jar!/:4.2.3.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
So, I would like to suggest to use PooledByteBufAllocator for LittleEndianByteBufAllocator. Or at least make it configurable.
Thanks!
My environment:
Linux
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>2.0.72.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>4.2.3.Final</version>
</dependency>
<dependency>
<groupId>com.github.jasync-sql</groupId>
<artifactId>jasync-r2dbc-mysql</artifactId>
<version>2.2.4</version>
</dependency>