highCVE-2026-54251Aug 20, 2026

CVE-2026-54251: netty-incubator-codec-ohttp Native Direct-Memory Leak on AEAD Decryption Failure

Rohit Hatagale
AI Security Researcher, SecureLayer7

An attacker can crash an OHTTP gateway by repeatedly sending requests with a deliberately broken encryption tag, causing the server to leak native memory until it runs out and crashes.

Packageio.netty.incubator:netty-incubator-codec-ohttp
Ecosystemmaven
Affected< 0.0.23.Final
Fixed in0.0.23.Final
CVE-2026-54251: netty-incubator-codec-ohttp Native Direct-Memory Leak on AEAD Decryption Failure

The problem

The OHttpServerCodec in netty-incubator-codec-ohttp allocates a pooled direct (off-heap) ByteBuf to hold decrypted plaintext before the AEAD tag is verified.

If the tag is invalid, BoringSSL throws a CryptoException. Because no try/finally guard wraps the allocation, the ByteBuf is never released. Each failed decryption permanently leaks one native buffer. An unauthenticated remote attacker can repeat this in a loop to exhaust the JVM's direct-memory pool, causing an OutOfMemoryError and taking down the gateway.

Proof of concept

A working proof-of-concept for CVE-2026-54251 in io.netty.incubator:netty-incubator-codec-ohttp, with the exact payload below.

http
POST /ohttp-gateway HTTP/1.1
Host: gateway.example.com
Content-Type: message/ohttp-req
Content-Length: 84

<binary: valid OHTTP key-id + KEM/KDF/AEAD suite prefix, then random bytes replacing the real HPKE ciphertext+AEAD tag>

# Repeat in a tight loop (e.g., 10k req/s) to exhaust direct memory:
# while true; do
#   curl -s -o /dev/null --data-binary @invalid_ohttp_req.bin \
#        -H 'Content-Type: message/ohttp-req' \
#        https://gateway.example.com/ohttp-gateway
# done

The root cause is a missing try/finally block in the OHttpServerCodec decode path. The codec calls alloc.directBuffer() to hold plaintext output, then passes it into the AEAD open() call. When EVP_AEAD_CTX_open fails (bad tag), a CryptoException propagates up before any release() call, leaving the buffer's reference count at 1 forever (CWE-401: Missing Release of Memory after Effective Lifetime, grouped under CWE-664).

The fix in 0.0.23.Final wraps the buffer allocation in a try/finally block so that plaintext.release() is always called on the exception path, preventing the leak regardless of decryption outcome.

No public standalone PoC script exists. The payload above is derived directly from the advisory description: any well-formed OHTTP outer envelope carrying an invalid or truncated AEAD tag will trigger the leak on every request.

The fix

Upgrade to netty-incubator-codec-ohttp 0.0.23.Final or later. In Maven: set <version>0.0.23.Final</version> for artifact io.netty.incubator:netty-incubator-codec-ohttp. No workaround exists short of rate-limiting or rejecting oversized/malformed OHTTP requests at a WAF or reverse proxy in front of the gateway.

Reporter not attributed.

References: [1][2][3]

Related research