Cloudflare’s recent GLM-5.3 incident began with a contradiction that should concern every team running production workloads through large language models. Customers sent valid requests to the model, the API returned HTTP 200 responses, and the resulting text was coherent. Yet some responses had been generated from data belonging to other requests.
Between August 27 and September 9, Cloudflare says that 155 requests to GLM-5.3 on Workers AI were affected by a vulnerability in the model’s key-value cache. The incident was narrow: it involved unusually long prompts, one model and a specific configuration of the SGLang inference engine. Cloudflare found no evidence of deliberate exploitation.
Its significance extends well beyond those 155 responses because it exposes a weakness in the way software teams verify AI systems. The request succeeded at the network level, the response conformed to the expected format, and the model produced plausible language. Nearly every conventional application health check would have passed. The computation itself was still wrong.
Strictly speaking, Cloudflare did not serve customers a different model. GLM-5.3 generated the responses. The deeper problem was that the model operated on inference state left by another request. That distinction matters technically, but it offers little comfort operationally. The customer received bytes from the advertised endpoint without receiving the computation implied by the API contract.
The incident belongs to a larger class of failure in which an AI service remains available while the integrity of its work has broken down.
The response belonged to the endpoint, but the state did not belong to the request
Large language models avoid recomputing every token in a conversation by storing intermediate attention data in a key-value, or KV, cache. For long prompts and repeated generation, this cache is fundamental to acceptable latency and cost. In a shared inference environment, however, the cache also contains representations derived from customer inputs, making its lifecycle part of the security boundary.
Cloudflare configured GLM-5.3 with a context window of 1,048,576 tokens, while the KV cache pool used during the prefill stage could accommodate only 681,280. When a prompt exceeded the smaller limit, SGLang rejected it and substituted a one-token placeholder. In Cloudflare’s disaggregated architecture, that rejection did not stop the separate decode stage. The decoder had already reserved cache pages for the original prompt and proceeded to read pages containing state left by previous requests.
The system therefore produced a linguistically valid response from invalid provenance.
Cloudflare’s technical account of the incident is unusually valuable because it describes what its prelaunch tests missed. The company had tested HTTP status codes, streaming behavior and inference-engine performance. It had not validated whether the returned text was coherent with the submitted prompt. The oversized requests therefore returned 200 responses when they should have failed with a 400 error, and the success-path monitoring treated them as healthy.
SGLang’s own disclosure confirms the cross-request cache exposure and documents the engine-level changes made in response. The relevant lesson is architectural: an error identified by one stage of a distributed inference system has to terminate the entire transaction. An internal rejection that becomes an externally successful response is more dangerous than a visible outage because it preserves customer trust in a result the system should never have produced.
Cloudflare increased the cache capacity, corrected the rejection path and added token-count checks, canaries and checksums between inference stages. It also prototyped page-ownership tags that associate cache pages with individual requests, reporting less than 1% overhead in its tests. Those controls are more interesting than the immediate configuration fix because they verify the identity of the state moving through the system rather than assuming that successful execution proves correct ownership.
Shared infrastructure has produced this failure pattern before
Cloudflare’s incident is specific to modern inference architecture, but the underlying failure mode is familiar: a pooled resource contains data from several users, a request loses synchronization with that resource, and structurally valid information reaches the wrong customer.
In March 2023, OpenAI took ChatGPT offline after users began seeing titles from other people’s conversation histories. The company traced the problem to the asynchronous Redis client used by its application. When a request was canceled after entering a connection’s incoming queue but before its response was removed from the outgoing queue, that connection could be returned to the shared pool in a corrupted state. The next request could then receive data left behind for somebody else.
OpenAI explained that many occurrences generated obvious server errors. The more troubling cases were those in which the misplaced data had the type expected by the next caller. The application accepted it because it looked valid.
The same bug may also have exposed payment-related information for 1.2% of the ChatGPT Plus subscribers active during a nine-hour window. OpenAI’s remediation included redundant checks that cached data belonged to the requesting user, rather than relying on the cache and connection pool to maintain that association implicitly. Its incident report could almost serve as a conceptual precursor to Cloudflare’s: both systems returned plausible data through a shared resource whose internal ownership had become misaligned.
The similarity matters because it prevents teams from treating the GLM-5.3 leak as an obscure SGLang defect. AI applications combine conventional distributed systems with newer inference-specific components. They inherit established failure modes from connection pools, caches and queues while adding KV-cache allocators, continuous batching, speculative decoding, model routers and accelerator memory. Each optimization introduces another place where the request’s identity must remain attached to its data.
A security review confined to prompts and model outputs will miss much of that surface.
Isolation has to continue all the way down to the GPU
In 2024, Trail of Bits disclosed LeftoverLocals, a vulnerability affecting GPUs from several manufacturers. On vulnerable devices, one process could read local GPU memory left by another process, including across process or container boundaries.
The researchers demonstrated the problem against an interactive LLM session. A co-resident process repeatedly read uninitialized local memory, identified computations from the victim’s model and reconstructed much of the model’s generated response. On an AMD Radeon RX 7900 XT, the researchers found that approximately 5.5 MB could leak per GPU invocation, amounting to roughly 181 MB during one query to the seven-billion-parameter model used in their demonstration.
Nvidia and Arm said the devices they assessed were not affected, while Apple, AMD, Qualcomm and Imagination provided or discussed mitigations for affected products. The vendor differences are important: this was not evidence that every GPU deployment leaks LLM data. It was evidence that process and container boundaries do not automatically guarantee memory isolation at every layer of an accelerated-computing stack.
For a company consuming a managed model API, these details are largely hidden behind the provider’s service boundary. For teams hosting models themselves, they are part of the platform’s threat model. Multi-tenant inference cannot be evaluated solely by inspecting Kubernetes namespaces or application permissions. The scheduler, runtime, inference engine, device driver and accelerator all participate in the isolation guarantee.
Cloudflare’s stale KV pages and LeftoverLocals’ uninitialized GPU memory are technically different vulnerabilities. Together, they show why inference security has to follow the data through the entire computational path.
“The model” is increasingly a routing decision
Even when no information crosses tenant boundaries, a successful response may still have been produced through a different path than the application owner assumes.
Modern AI stacks increasingly use gateways that distribute requests across providers, regions and model variants. This can improve resilience, capacity and price performance. It can also make the identity of the serving system conditional.
OpenRouter, for example, documents fallback behavior that can move a request after provider downtime, rate limiting, context-length errors or moderation decisions. Its guidance explicitly tells developers to inspect the response’s model field to determine which model actually handled the request, particularly when a fallback has occurred. A 200 response establishes that the router found a successful execution path; it does not establish that the preferred model or provider completed the work.
That behavior can be entirely intentional and useful. The operational mistake is treating routing configuration as an implementation detail when it can change cost, latency, data residency, safety behavior and output quality. If an application requires a particular model because it has passed a regulated evaluation, supports a specific context window or runs in an approved region, an unobserved fallback becomes a policy violation even when the response is perfectly good.
Model names themselves may also represent moving targets. OpenAI’s API documentation warns that prompting behavior can change between model snapshots and recommends pinned versions plus application evaluations when consistency matters. An alias makes upgrades easier; a pinned snapshot makes model identity auditable. Neither is universally preferable, but teams should choose deliberately rather than discovering after a production change that the name stayed constant while the behavior moved.
The relevant production question is therefore more precise than “Did the API respond?” It is: which model version, running through which provider and infrastructure path, processed which input state under which policy?
Most organizations cannot answer that from their application logs today.
AI systems need inference integrity, not merely availability
Traditional observability separates availability, latency and correctness because a service can be reachable without doing the right work. LLM systems need an additional concept: inference integrity.
Inference integrity means that the output can be associated with the intended request, customer, model, version and execution policy. It requires evidence across the transaction rather than confidence in its final HTTP status.
For teams operating AI systems, that changes what should be recorded and tested:
- The effective model, version and provider should be captured from the response or trusted gateway metadata, rather than inferred from the requested endpoint.
- Every inference stage should carry a request and tenant identity, with the system failing closed when those identifiers disagree.
- Fallbacks should be observable events governed by policy. A substitution that is acceptable for summarization may be unacceptable for code generation, healthcare or financial decisions.
- Cache pages, intermediate tensors and GPU memory should have explicit allocation, ownership and release controls, particularly in multi-tenant environments.
- Canary prompts should test semantic correspondence between input and output. A response that is fluent but unrelated to the prompt should trigger an integrity alarm even when latency and status remain normal.
- Evaluations should run against pinned model snapshots where reproducibility matters, and again whenever an alias, router or provider changes the effective execution path.
- Logs should preserve enough provenance to reconstruct a decision without storing sensitive prompts more broadly than necessary.
None of these controls can prove that a probabilistic model produced the ideal answer. They address a more elementary requirement: proving that the system performed the inference transaction it claims to have performed.
Cloudflare’s incident is useful precisely because the model was not hallucinating in the familiar sense. GLM-5.3 generated a reasonable continuation from the state it received. The infrastructure supplied the wrong state and then certified the transaction as successful.
A single request may pass through an API gateway, policy engine, prompt cache, model router, prefill worker, decode worker, GPU scheduler and post-processing layer before reaching the application. Reliability improvements at each layer can make the overall system more efficient while making its execution path harder to see.
In that environment, a 200 OK means that something completed; production-grade AI requires evidence of what completed, on whose data, with which model and under whose rules.
