Skip to the yard
Waxsealai

GLM 5.3 KV Cache Exposure: Root Cause, Fixes and SGLang Safeguards

Shipped
Shipper
Kevin Flansburg
Readers
934

A KV pool mismatch and SGLang handoff bug exposed data in 155 long-prompt responses. The fixes add safer rejection, transfer checks and broader cache safeguards.

GLM 5.3 KV Cache Exposure: Root Cause, Fixes and SGLang Safeguards

A mismatch between GLM 5.3’s advertised context window and the GPU memory available for its key-value (KV) cache led to 155 Workers AI responses containing data from other customers’ requests. The exposure affected prompts longer than 681,280 tokens between August 27 and September 9, 2026. Cloudflare found no evidence of deliberate exploitation, and reported that no other model on Workers AI was affected.

The incident exposed a failure at the boundary between two parts of SGLang’s disaggregated inference setup. Prefill rejected prompts that exceeded its cache pool, but the rejection did not stop those requests from being handed to decode. Decode then treated old cache pages as if prefill had filled them with the current prompt. Cloudflare corrected the model configuration, patched SGLang’s handling of rejected requests and added safeguards for related cache lifetime and transfer failures.

What customers experienced

GLM 5.3, served as @cf/zai-org/glm-5.3, launched on Workers AI on August 27 with a 1,048,576-token context window. A prompt exceeding the smaller prefill KV pool could follow a faulty path through the inference engine. In those cases, a response could include content left in cache pages by earlier requests to the same model.

Cloudflare counted 155 affected responses. Each was associated with a prompt longer than 681,280 tokens, the capacity of the prefill cache pool in the original deployment. After excluding its own tests and the researcher’s, Cloudflare said the remaining request volume and timing were consistent with ordinary long-context use. It found no evidence of someone repeatedly exploiting the flaw.

Gadi Evron of Knostic reported the issue on September 8 through Cloudflare’s HackerOne program. Cloudflare reproduced it the next day using two independent accounts, then made the model private at 14:00 UTC on September 9. The service returned HTTP 400 while offline. A corrected configuration and patched inference engine were deployed, and the model was restored at 19:40 UTC that day.

Why a rejected prompt reached decode

Workers AI runs GLM 5.3 in disaggregated prefill/decode mode. Prefill GPUs process the prompt and build its KV cache. The system transfers that cache to decode GPUs over RDMA using Mooncake; decode then uses it to generate the response. Splitting the work lets the two stages use separate GPU pools, but it also means request state and cache ownership must remain consistent across machines.

The cache is made up of fixed-size pages in GPU memory. Requests receive pages when admitted and release them when they finish. Pages are not cleared between requests: a later request overwrites them as needed. That approach supports inference performance, but it puts the burden on the engine to track which request owns each page and when the page can safely be reused.

In Cloudflare’s original deployment, the prefill pool held 681,280 tokens, while the model context limit was 1,048,576 tokens. SGLang sizes the pool at startup from GPU memory left after loading model weights; context length is configured separately. Nothing automatically guarantees the two values match.

Cloudflare knew about the mismatch before launch. Its tests showed successful HTTP responses and streaming bodies without error frames, so the team believed SGLang handled prompts beyond the pool capacity. The gap was that testing checked response status and performance, but not whether model output was coherent or correct. Cloudflare now checks the running engine’s pool size against the configured context length before deploying a change to a disaggregated setup.

The mismatch alone would have caused an availability problem if SGLang had stopped the request cleanly. The data exposure required a second failure: how the engine handled an over-capacity request after marking it rejected.

The faulty request path

In SGLang v0.5.21’s earlier behavior, the scheduler recorded an error for a prompt that exceeded the KV pool, then replaced the prompt with a one-token placeholder. That path was intended to let the request move through the usual batch machinery without allocating memory for the original prompt. On a single node, the placeholder could complete and the client would receive an error.

In disaggregated mode, the prefill and decode admission queues checked whether a request was finished, rather than whether it had been rejected. The rejection lived in a separate field, so the request still appeared active to both queues. It could therefore continue through the handoff:

  1. Decode reservation: The router sent decode the original prompt length. Decode reserved the corresponding eight KV pages in the reproduction described by Cloudflare.
  2. Prefill admission: The request passed the queue’s “not finished” check, even though it already carried a rejection.
  3. Placeholder computation: Prefill processed the one-token placeholder instead of the full prompt.
  4. Cache transfer: Prefill transferred one page and signaled that the transfer had completed.
  5. Decode admission and generation: Decode accepted the handoff and generated a response while attending across all eight reserved pages. Only the first page had been written for this request; the others retained data from previous requests.

The client could receive HTTP 200 rather than HTTP 400. Cloudflare observed that the reported prompt-token count matched the long prompt and that the answer could appear coherent, even though generation was faster than a full eight-page prefill should have been. The output could contain material from earlier users’ cache pages. Every request beyond the relevant threshold followed this faulty path.

A related SGLang issue, #30233, had been reported on July 6, 2026. Cloudflare said it had been raising concerns about this type of failure with the SGLang team since v0.5.10, and that the incident accelerated work already under way with the maintainers. The issue’s fixes appeared across SGLang v0.5.20, v0.5.21 and v0.5.22.

Immediate fixes and deployment changes

Cloudflare first changed the GLM 5.3 configuration to match the one used for GLM 5.2. The resulting prefill KV pool holds about 2.04 million tokens, roughly twice the model’s 1,048,576-token context limit. A valid prompt within the context window now fits in the pool, preventing this particular mismatch from triggering the rejection path.

That configuration change reduces the chance of reaching the bug, but pool sizes can change when model weights, quantization, tensor parallelism, CUDA graph sizes or memory settings change. Cloudflare also patched the engine so a request carrying a pending rejection is retired at the disaggregation admission queue with its original error, before bootstrap, allocation or cache transfer. Its proposed upstream change, SGLang pull request #38704, also added a check that the number of tokens transferred matches the number decode reserved.

The SGLang team had already merged #36988 to address a related failure path on September 6, though the fix had not yet been released when the exposure was reported. The maintainers later merged #38935, which rejects these requests earlier, following the approach in Cloudflare’s patch. Cloudflare tested the configuration and engine changes against the researcher’s reproduction script and its own canary-marker tests before restoring service.

The exposure required the specific pool mismatch in GLM 5.3. Cloudflare said other disaggregated Workers AI deployments had KV pools larger than their context lengths on both prefill and decode, so their requests could not reach this rejection path. It applied the engine fix and additional safeguards to every model it serves with SGLang, and made the checks part of its deployment process.

Cache safety extends beyond this bug

The incident is one example of a broader class of KV cache memory safety failures: a page can be freed, reused or read while another operation still depends on it. Within one decode node, generation, host-memory offload, transfer reception and abort handling can all interact with the same page. If one task releases it while another still reads or writes it, a new request may receive a page that remains in use.

Cloudflare cited earlier SGLang fixes for decode offload and grammar-error double releases, including pull requests #24226, #30899 and #30937. Those address page lifetime within a node. A separate class of failures involves writes still in flight between prefill and decode when a request is aborted. RDMA writes already sent cannot be recalled. If decode releases those pages immediately, another request may reuse them before the late writes arrive.

Cloudflare had proposed fixes and a reproduction script for this cross-node race in #31357, #38961 and #32564. In that class of failure, corrupted output or another user’s data can appear without an error. The GLM 5.3 incident had a different trigger: a rejected request went through the handoff every time, rather than reaching the bad state through a timing race. In both cases, decode could read pages the current request had not written.

After the incident, Cloudflare extended deferred page release to cover prefill’s source pages and all transfer failure types, not just aborts. Work proposed in #38961 was superseded by #40645, #41402 and #41404. The team also deployed bounded Mooncake transfer batches, so a single transfer call cannot outlast the release timeout. Its load balancer now keeps the prefill connection open until prefill completes and the decode connection until the first token, avoiding a mid-transfer abort caused by a client disconnect.

Cloudflare added deployment checks for pool size versus context length, deferred release on both prefill and decode, transfer batch bounds within the release timeout, and required patches. It reported that these changes were deployed across its disaggregated production systems.

Upstream fixes and operator guidance

Cloudflare met with SGLang maintainers at RadixArk on September 10, the day after the incident. The teams agreed to investigate reported races, add integrity checks and build reproduction tests for this family of failures. SGLang v0.5.21 was released on October 2, and v0.5.22 on October 19. Cloudflare recommended that operators of disaggregated SGLang deployments use v0.5.22 or later, or backport the relevant fixes.

The upstream changes cover request admission, transfer cancellation, page release and observability. The incident-specific fixes in #36988 and #38935 retire rejected requests before they reach decode; they shipped in v0.5.20 and v0.5.21 respectively. Other changes cancel both request halves on client disconnect or prefill failure, enable deferred KV release by default on supported backends, preserve confirmation signals, and hold pages after transfer failures. The release table in the report places these follow-ups across v0.5.20, v0.5.21 and v0.5.22.

Further work targets failure cases that can leave pages in use or incorrectly marked as released. Examples include #43719, which prevents a failed Mooncake send from reporting its writes as drained, and #37077, which tags abort messages per request so a reused request ID cannot consume an old confirmation. SGLang also added a checksum option for decode to verify transferred KV before generation (#39500), fixes to its KV canary checks, and metrics for deferred release duration and completion by confirmation or timeout.

Several items remained in progress at publication. Cleanup for the checksum feature in #39729 had been merged but was awaiting release; zeroing KV pages on release was proposed in #46435; and the teams were working to extend checksums from the handoff to each decode step. Broader support for the KV canary and checksums across models was also unfinished.

For operators, the practical checks are direct. Measure the KV pool on every prefill and decode node after changes to weights, quantization, parallelism or memory fraction, and compare it with the context length. If the pool is smaller, reduce the configured context to fit. Install a build with #36988 and #38935, use deferred release on both roles, and bound Mooncake transfer batches so a timed-out call cannot leave an unbounded tail of writes in flight.

Operators can also enable --disaggregation-enable-kv-checksum and --kv-canary where their model architecture supports them, then confirm from logs that requests are being checked. Cloudflare said support was not yet available for every architecture, including GLM 5.3, GLM 5.3 Flash and DeepSeek V4. It also recommended configuring the proxy to let requests proceed through the first token, reducing unnecessary aborts that require coordination between nodes.

Longer-term integrity checks

Configuration checks and careful page lifetimes reduce the ways cache pages can be misused. They cannot anticipate every failure mode. Cloudflare’s longer-term goal is to have the engine detect at read time whether a request is about to attend over a page it does not own, and fail that request before returning data from another session.

A prototype Cloudflare built over the summer attached a changing tag to each physical KV page when it was reallocated. Requests recorded the tags they expected, and the system checked the mapping before decode read the cache. In tests, the prototype added under 1% to throughput and tail latency. Cloudflare tested it on GLM 5.2 in mid-August, then disabled it because of reliability issues. It also depended on particular attention backends and model architectures, so it was not a general solution.

The per-transfer checksum in #39500 is a narrower step: it checks the cache as it moves from prefill to decode. Page-level tags could check every read, including reads well after transfer. Cloudflare said it is working with SGLang on an engine-level implementation that can cover different models rather than relying on provider-specific patches.

A second line of work is to prove that the transfer protocol preserves its safety properties. RDMA moves data directly between node memory, so the machines must agree on the destination, size and completion state of each transfer. A disagreement can leave them with different views of which request owns a page, as happened in this incident.

Cloudflare plans to model its production SGLang and Mooncake cache-transfer system using formal methods, then identify and correct any divergence from the desired safety properties. Such proofs have limits, but a verified core can make the unproven parts easier to inspect and monitor. The immediate engineering lesson is more concrete: measure pool capacity from the running engine, reject invalid requests before cross-node handoff, and verify cache ownership throughout the page’s lifetime.

Source: Kevin Flansburg · blog.cloudflare.com

Same vesselRelated cargo