Version
4.x
What happened?
Jena Fuseki: 1 MB output buffers are never pooled by the default Jetty ByteBufferPool, causing DirectMemory churn under load
Summary
Jena Fuseki (jena-fuseki-main) builds its Jetty ServerConnector without configuring a ByteBufferPool, so Jetty falls back to a default ArrayByteBufferPool whose maximum pooled buffer capacity is 65536 bytes (64 KB). At the same time, Fuseki configures the connector's HttpConfiguration with a 1 MB outputBufferSize. Because 1 MB is larger than the pool's 64 KB maximum capacity, every output buffer that Jetty acquires for an HttpConnection is allocated fresh and then dropped on release rather than being retained for reuse. The result is continuous allocation and garbage collection of 1 MB direct buffers with no pooling benefit. Under sustained high request throughput this drives elevated DirectMemory usage and GC pressure. This report describes how the pieces connect, the root cause, and options the maintainers may want to weigh for a fix on the Jena side.
Environment
- Jena Fuseki: jena-fuseki-main 4.10.0
- Jetty: 10.0.17 (the version Jena 4.10.0 declares via the
jetty-bom import in its root pom.xml, see pom.xml line 71)
- Java: Java 17 (Amazon Corretto 17.0.19)
- OS: Amazon Linux 2
Background: how the pieces fit together
Three separate configuration points combine to produce this behavior.
First, Fuseki provisions the Jetty connector in JettyHttps.java around line 101:
ServerConnector plainConnector = new ServerConnector(server, new HttpConnectionFactory(http_config));
No ByteBufferPool is passed to the connector. When a Jetty ServerConnector (via AbstractConnector) is constructed without an explicit ByteBufferPool, it creates a default ArrayByteBufferPool (see AbstractConnector.java line 198). That default ArrayByteBufferPool has a maximum pooled buffer capacity of 65536 bytes (64 KB), which is DEFAULT_MAX_CAPACITY_BY_FACTOR * DEFAULT_FACTOR = 16 * 4096 (see AbstractByteBufferPool.java line 62).
Second, the HttpConfiguration (http_config above) is built in JettyLib.httpConfiguration() around line 92, where it sets:
setRequestHeaderSize(512 * 1024) for 512 KBsetOutputBufferSize(1024 * 1024) for 1 MB (1,048,576 bytes)
Third, each Jetty HttpConnection acquires buffers sized to the configured outputBufferSize (1 MB) from the connector's pool (see HttpConnection.java line 779).
So the connector hands HttpConnection a pool that tops out at 64 KB, while HttpConnection repeatedly asks that pool for 1 MB buffers.
Root Cause
The mismatch between the configured 1 MB outputBufferSize and the default pool's 64 KB maximum capacity means the buffers can never be pooled. In ArrayByteBufferPool.acquire(), a request whose size maps to a bucket index outside the pool's range falls through to allocating a fresh, unpooled buffer, and the requested capacity is not clamped to the maximum, so an oversized request yields an oversized fresh buffer. However, ArrayByteBufferPool.release() will not retain any buffer larger than the maximum capacity. Oversized buffers are discarded rather than being returned to the pool.
The net effect is that every acquire of a 1 MB output buffer allocates fresh memory and every release throws it away. Under sustained or high TPS it produces heavy DirectMemory allocation churn and GC pressure. In my environment this contributed to memory growth and an OS-level OutOfMemoryError. Our situation was aggravated by a separate, environment-specific local misconfiguration in which DirectMemory could grow uncapped without any garbage collection cleanup. That local misconfiguration was an aggravating factor that led me to find this issue.
Reproduction
- Start Fuseki, which builds the connector as described above with a 1 MB output buffer size and the default
ByteBufferPool.
- Drive sustained HTTP traffic so that many
HttpConnection instances acquire and release 1 MB output buffers.
- Observe that the
ByteBufferPool never retains these buffers, since its 64 KB maximum capacity is smaller than 1 MB, so DirectMemory allocation churns continuously with no reuse.
A minimal Jetty-level reproduction is to acquire buffers larger than the pool's maximum capacity, release them, and repeat. Pooled memory never grows because oversized buffers are never retained on release.
Suggested Fix / Recommendation
According to Jetty, the buffer pools not reusing buffers but allowing provisioning of large buffers is an intentional design, so the fix belongs on consumers of the Jetty library (See upstream Jetty ticket). On the Jena side, this could be an explicit pool configuration or by keeping the pool capacity in sync with the configured output buffer size.
The workaround I currently have is for Fuseki to explicitly configure the connector's ByteBufferPool with a maximum capacity at least as large as the configured output buffer size, so that the 1 MB buffers can actually be pooled and reused. The following configuration sets the pool maximum capacity to 2 MB, exceeding the 1 MB output buffer size:
ArrayByteBufferPool byteBufferPool = new ArrayByteBufferPool(0, 2048, 2 * 1024 * 1024, -1, -1, -1);
server.addBean(byteBufferPool);
With this in place, the output buffers are pooled and reused rather than churned. I haven't checked if other versions of Apache Jena have this issue, but it may be worth verifying on 5.x as well.
References
Relevant output and stacktrace
Are you interested in making a pull request?
Maybe
Version
4.x
What happened?
Jena Fuseki: 1 MB output buffers are never pooled by the default Jetty ByteBufferPool, causing DirectMemory churn under load
Summary
Jena Fuseki (jena-fuseki-main) builds its Jetty
ServerConnectorwithout configuring aByteBufferPool, so Jetty falls back to a defaultArrayByteBufferPoolwhose maximum pooled buffer capacity is 65536 bytes (64 KB). At the same time, Fuseki configures the connector'sHttpConfigurationwith a 1 MBoutputBufferSize. Because 1 MB is larger than the pool's 64 KB maximum capacity, every output buffer that Jetty acquires for anHttpConnectionis allocated fresh and then dropped on release rather than being retained for reuse. The result is continuous allocation and garbage collection of 1 MB direct buffers with no pooling benefit. Under sustained high request throughput this drives elevated DirectMemory usage and GC pressure. This report describes how the pieces connect, the root cause, and options the maintainers may want to weigh for a fix on the Jena side.Environment
jetty-bomimport in its rootpom.xml, seepom.xmlline 71)Background: how the pieces fit together
Three separate configuration points combine to produce this behavior.
First, Fuseki provisions the Jetty connector in
JettyHttps.javaaround line 101:No
ByteBufferPoolis passed to the connector. When a JettyServerConnector(viaAbstractConnector) is constructed without an explicitByteBufferPool, it creates a defaultArrayByteBufferPool(seeAbstractConnector.javaline 198). That defaultArrayByteBufferPoolhas a maximum pooled buffer capacity of 65536 bytes (64 KB), which isDEFAULT_MAX_CAPACITY_BY_FACTOR * DEFAULT_FACTOR= 16 * 4096 (seeAbstractByteBufferPool.javaline 62).Second, the
HttpConfiguration(http_configabove) is built inJettyLib.httpConfiguration()around line 92, where it sets:setRequestHeaderSize(512 * 1024)for 512 KBsetOutputBufferSize(1024 * 1024)for 1 MB (1,048,576 bytes)Third, each Jetty
HttpConnectionacquires buffers sized to the configuredoutputBufferSize(1 MB) from the connector's pool (seeHttpConnection.javaline 779).So the connector hands
HttpConnectiona pool that tops out at 64 KB, whileHttpConnectionrepeatedly asks that pool for 1 MB buffers.Root Cause
The mismatch between the configured 1 MB
outputBufferSizeand the default pool's 64 KB maximum capacity means the buffers can never be pooled. InArrayByteBufferPool.acquire(), a request whose size maps to a bucket index outside the pool's range falls through to allocating a fresh, unpooled buffer, and the requested capacity is not clamped to the maximum, so an oversized request yields an oversized fresh buffer. However,ArrayByteBufferPool.release()will not retain any buffer larger than the maximum capacity. Oversized buffers are discarded rather than being returned to the pool.The net effect is that every acquire of a 1 MB output buffer allocates fresh memory and every release throws it away. Under sustained or high TPS it produces heavy DirectMemory allocation churn and GC pressure. In my environment this contributed to memory growth and an OS-level OutOfMemoryError. Our situation was aggravated by a separate, environment-specific local misconfiguration in which DirectMemory could grow uncapped without any garbage collection cleanup. That local misconfiguration was an aggravating factor that led me to find this issue.
Reproduction
ByteBufferPool.HttpConnectioninstances acquire and release 1 MB output buffers.ByteBufferPoolnever retains these buffers, since its 64 KB maximum capacity is smaller than 1 MB, so DirectMemory allocation churns continuously with no reuse.A minimal Jetty-level reproduction is to acquire buffers larger than the pool's maximum capacity, release them, and repeat. Pooled memory never grows because oversized buffers are never retained on release.
Suggested Fix / Recommendation
According to Jetty, the buffer pools not reusing buffers but allowing provisioning of large buffers is an intentional design, so the fix belongs on consumers of the Jetty library (See upstream Jetty ticket). On the Jena side, this could be an explicit pool configuration or by keeping the pool capacity in sync with the configured output buffer size.
The workaround I currently have is for Fuseki to explicitly configure the connector's
ByteBufferPoolwith a maximum capacity at least as large as the configured output buffer size, so that the 1 MB buffers can actually be pooled and reused. The following configuration sets the pool maximum capacity to 2 MB, exceeding the 1 MB output buffer size:With this in place, the output buffers are pooled and reused rather than churned. I haven't checked if other versions of Apache Jena have this issue, but it may be worth verifying on 5.x as well.
References
pom.xmlline 71 (Jetty version, 10.0.17): https://github.com/apache/jena/blob/jena-4.10.0/pom.xml#L71JettyHttps.javaline 101 (connector provisioning): https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/JettyHttps.java#L101JettyLib.javaline 92 (1 MB output buffer size): https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/sys/JettyLib.java#L92AbstractConnector.javaline 198 (default pool provisioning): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/AbstractConnector.java#L198HttpConnection.javaline 779 (acquires output buffer sized tooutputBufferSize): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/HttpConnection.java#L779AbstractByteBufferPool.javaline 62 (max capacity defaults to 65536 bytes): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/AbstractByteBufferPool.java#L62ArrayByteBufferPool.javaline 146 (acquire()): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L146ArrayByteBufferPool.javaline 169 (release()discarding oversized buffers): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L169Relevant output and stacktrace
Are you interested in making a pull request?
Maybe