Choose who can share an inference prefix cache

Start with the sharing decision

Partition inference prefix reuse by the smallest group whose members are allowed to learn about one another's prompt presence. Derive that group in trusted application code, supply its cache scope on every accepted request, and establish that the scope survives offload, transfer and restore. If a relevant path cannot enforce the decision, disable cross-request reuse there or separate the serving runtime. Authenticating two callers does not, by itself, authorize them to share inference state.

This is a narrower question than whether a model endpoint is private. A caller can have a valid credential, use the intended API and receive only its own generated answer while still observing timing influenced by someone else's earlier request. vLLM documents a cache-salting mechanism for separating such reuse and warns that an omitted salt preserves shared behavior. Its security guidance calls for unpredictable secret values rather than a username or account identifier. ^[2]

Start the design record with a sentence about permitted knowledge, not a configuration field. For example, members of a shared support team may be permitted to know that a public product manual was processed, but not that a particular employee's disciplinary record was submitted. The two prompts might arrive under the same corporate tenant. That does not make their permissible sharing populations identical. Conversely, isolating every request would discard reuse that the application could intentionally and safely permit.

In a hypothetical internal assistant, a legal team and a sales team share one model-serving pool. Both can use the public policy prefix. Only legal staff can process a confidential negotiation document. A tenant-wide salt is an insufficient boundary if every employee belongs to that tenant and prompt presence itself is sensitive. A team scope could be appropriate for the negotiation workflow, while a user scope may be necessary for personal material. These are design choices, not claims about a deployed system.

Ask who controls group membership, whether a user can choose a different workflow through request parameters, and what happens when a conversation changes classification. A cache group should not silently widen because an administrator renamed a tenant, migrated an account or enabled a second API. Record the approved data classes, the sharing principals, the authority that assigns scope, and the paths on which it is enforced. Leave unresolved paths visibly unresolved instead of describing the whole service as isolated.

Mixed-sensitivity prefixes need particular care. A long request can begin with public instructions and later include private material. A runtime's supported reuse granularity determines what can be shared, but the application still owns the permission decision. Do not assume that a public opening sentence makes all following state public, or that a team label covers every document inserted into the prompt. If the system cannot express the intended distinction reliably, choose a more restrictive scope for the whole request and assess the performance effect afterward.

Group overlap is another useful design test. A person may belong to both the legal and sales teams, while the other members of those teams must remain separate. Giving that person a special scope that merges both populations would make the person's membership a bridge between otherwise isolated caches. Instead, resolve the scope from the authorized workflow and data class for each request. This example is a policy reasoning aid, not a statement that any named runtime automatically merges groups. It demonstrates why an account-level setting may be too coarse for an application-level sharing rule.

The decision record should also state whether revealing use of a public document is itself acceptable. A document can be public while an organization's interest in it is sensitive. Permission to read its contents does not always include permission to infer that another team recently analyzed it. That distinction may justify narrower reuse even when every token in the prefix is publicly available.

TensorRT LLM also documents matching cache salts as a condition for shared KV blocks. Its accepted string format is not an authorization system. A string that is technically valid can still represent the wrong sharing population or be supplied by the wrong actor. ^[5] Treat the inference server's mechanism as an enforcement primitive that the application must use correctly, then select the smallest practical design whose boundaries can be explained and verified.

Source. Original decision model informed by the vLLM security guidance reviewed September 2, 2026. ^[2]

Method. Conceptual policy flow. It is not a security score, certification or performance measurement.

Accessible table and figure data

Figure 1 accessible table

Question	If yes	If no
May these principals learn about each other's prompt presence?	Consider a shared scope for this data class	Choose a narrower group or user scope
Can trusted code assign scope on every accepted route?	Inspect all reuse and transfer paths	Disable reuse or use assessed dedicated runtime
Does evidence cover every participating state plane?	Enable only the verified scoped path	Contain or disable the unresolved path

Figure 1 accessible table

Question	If yes	If no
May these principals learn about each other's prompt presence?	Consider a shared scope for this data class	Choose a narrower group or user scope
Can trusted code assign scope on every accepted route?	Inspect all reuse and transfer paths	Disable reuse or use assessed dedicated runtime
Does evidence cover every participating state plane?	Enable only the verified scoped path	Contain or disable the unresolved path

Download imageDownload table CSV

Locate every copy of reusable state

Prefix caching saves repeated prefill work. The reusable object is attention key/value state associated with an input prefix, not a stored final answer. vLLM's design identifies blocks using preceding-prefix identity, token content and relevant extra inputs; a salt participates in the first block's hash and thereby influences following prefix identities. Multimodal content and adapters also affect whether reuse is valid. These correctness inputs and the permitted sharing scope solve different problems. ^[3]

A model revision answers which computation produced the state. A tenant scope answers which callers may share it. Neither can substitute for the other. A complete inventory should record both, including the tokenizer or template assumptions that determine the effective input. This article does not prescribe a universal cache-key tuple because runtimes implement these details differently. The useful invariant is that distinct security scopes never become equivalent merely because the text or another content identifier matches.

Draw the physical state path separately from the request path. A request might enter a gateway, reach a prefill worker and complete on a decode worker. Reusable state might stay on a GPU, be offloaded to host memory, cross a connector or live in a separate cache process. vLLM's disaggregated-prefilling documentation explicitly describes an experimental architecture with connectors and separate prefill and decode roles. A common front door therefore does not establish a common enforcement boundary for every state transfer. ^[6]

For each plane, identify the writer, the reader, the lookup key, the lifetime and the deletion operation. Ask whether its network listener is reachable only from intended components, whether its storage identity can read other tenants' entries, and whether a restore can revive data that an application believed retired. The answers should name concrete owners. 'The serving platform handles it' is not enough when the serving and storage teams configure different parts of the path.

Keep a response cache out of this proof. A semantic answer cache can return previously generated content even when the underlying prefix cache is perfectly partitioned. Likewise, an embedding store, a media-processing cache and a conversation log can expose different representations. OWASP's RAG guidance treats authorization, provenance and lifecycle handling as explicit application concerns. ^[7] Use separate acceptance criteria for these surfaces rather than borrowing a successful KV-cache test as evidence that all retained information is protected.

Consider a hypothetical migration from a single process to a shared remote cache. The old runtime generated separate keys correctly, but an export worker writes entries using only a content fingerprint. If a later import reconstructs lookup identity without the original scope, the application has lost its intended separation even though the gateway still authenticates every request. This is an illustrative failure mode, not a report about a particular connector. Its purpose is to expose the exact serialization and restore questions an integration review must answer.

An inventory should also distinguish inaccessibility from erasure. Changing scope may prevent future requests from finding an old entry without removing its bytes from every storage layer. A retention requirement may demand both actions, potentially on different schedules. Record that difference before promising deletion to users. Otherwise an operational shortcut, such as waiting for normal eviction, can be mistaken for a verified purge.

Read the historical audit narrowly

The research motivating this boundary should not become a current provider league table. An ICML 2025 paper reports an audit conducted in September and early October 2024 from California. Across 17 providers, the authors detected caching through timing at eight and global sharing at seven. The main procedure used 5,000-token prompts and 250 samples per timing procedure. Provider model selections differed. ^[1]

The chart reorganizes those totals into mutually exclusive provider categories: seven with global sharing detected, one with caching detected without global sharing, and nine where timing did not detect caching. The last category does not mean no cache existed. The paper discusses actual hits without a detectable timing difference. Its global-sharing definition concerns cross-user and cross-organization reuse, with a provider-specific adjustment where organizations were unavailable. ^[1]

These results establish a historical reason to ask about shared state. They do not establish present-day behavior at any named provider, random-sample industry prevalence or successful reconstruction of arbitrary private prompts. Detecting that a guessed prefix appears to be cached and recovering an unknown document are different claims. A useful architecture review does not need to exaggerate one into the other.

Timing is particularly easy to misuse as a local acceptance test. An engineer can see a slow response from a second account and conclude that isolation works, even though queueing, batching or the tested prefix made the cache effect hard to distinguish. The opposite shortcut is also weak: a fast response can reflect unrelated warm infrastructure. NVIDIA's discussion of KV-cache security emphasizes the dependence of timing observations on application structure and surrounding conditions. ^[4] That is a reason to improve evidence, not a reason to assume noise creates a dependable boundary.

A procurement question should therefore ask for the scope of cache sharing, the mechanism enforcing it, the interfaces covered and the evidence date. Ask whether separate API keys under one organization share cache state, whether custom scope controls exist, and which data-handling commitments include derived inference state. A provider may legitimately decline to reveal internal implementation details. In that case, distinguish contractual assurance from independently observed technical behavior and decide whether that assurance fits the data involved.

The same discipline applies internally. Record what an experiment could falsify before collecting timings. If the goal is to establish that different security groups cannot reuse a block, an instrumented cache lookup is more directly connected to that claim than a single end-to-end latency number. If internal instrumentation is unavailable, document the narrower conclusion the external measurement supports. Never turn an inconclusive result into an affirmative isolation statement.

Source. Calculated from the provider totals in sections 4.1 and 4.2 of Auditing Prompt Caching in Language Model APIs. September and early October 2024 sample. ^[1]

Method. In this September to early October 2024 audit, timing tests detected caching at eight of 17 providers and global sharing at seven. Categories are calculated from reported provider totals. Nondetection does not establish that a provider had no cache, and these results do not describe current services. Derivations are 7, 8 minus 7, and 17 minus 8. This article did not reproduce the audit.

Accessible table and figure data

Figure 2 accessible table

Historical audit category	Providers
Global sharing detected	7
Caching detected without global sharing	1
Caching not detected by timing	9

Figure 2 accessible table

Historical audit category	Providers
Global sharing detected	7
Caching detected without global sharing	1
Caching not detected by timing	9

Download imageDownload table CSV

Derive scope at a trusted boundary

The recommended control point is a gateway or application service that has already verified the caller and resolved its authorized workflow. That component derives the cache scope and supplies the runtime parameter. Do not let a public request body select an arbitrary internal reuse group. Reject an attempted override or replace it under a clearly documented policy before the request enters any cache-bearing component. This recommendation follows from the need to control who can obtain a valid scope; it is not a vendor-provided complete gateway recipe.

A scope value has two jobs that should not be confused. It must consistently identify the intended sharing group, and, where a caller who learns the value could reuse another group's state, it must not be guessable or casually disclosed. The vLLM security guidance explicitly treats the salt as secret. ^[2] A globally unique tenant identifier may satisfy uniqueness while failing secrecy. Logging the secret next to ordinary request diagnostics can similarly defeat an otherwise sound assignment policy.

Use an internal reference for observability. The request trace can record that policy version A selected scope record B without copying B's secret material into customer-visible errors, support exports or routine dashboards. The reference needs its own access controls if it reveals sensitive relationships. This is an original operational recommendation: the trace should help an authorized operator explain a decision without becoming a convenient catalog of values that bypass it.

In the legal-and-sales example, the gateway first resolves the authenticated identity and selected workflow. It checks that the caller belongs to the workflow's permitted population, then looks up the current private scope value. Two legal users may deliberately receive the same internal value; a sales user cannot request that value by naming the legal team. A public-manual workflow can use a different policy. The decision is about this use of this data, not merely the organization's billing account.

Handle absence as a policy event. Missing identity, an unavailable membership service or a failed secret lookup should not quietly fall back to an empty salt on a shared serving pool. The safest response depends on the service: reject the request, route it to a verified dedicated runtime, or perform uncached work if that path is known to avoid cross-request reuse. Establish the alternate path in advance. An improvised retry that drops optional parameters is exactly the kind of behavior this review is meant to catch.

Rotation needs two separate decisions. First, when does the gateway stop assigning the old scope and start assigning the new one? Second, what happens to already retained state under the old value? A policy revision can answer the first immediately while storage cleanup takes longer. Keep a record of both states, including in-flight requests and retries. If a group loses a member, make sure that member cannot continue submitting requests with a previously learned value through a bypass route.

Do not rotate so aggressively that every request becomes an accidental private namespace, unless request-level isolation is the intended design. That would make the system look secure while obscuring that the promised team reuse is not operating. Positive reuse tests and negative isolation tests belong together. They verify different halves of the contract and make a change in either behavior visible.

Carry the decision through the cache path

An accepted parameter at one endpoint is only the beginning of the proof. Build an endpoint inventory from the deployed application and runtime, including batch, compatibility, pooling and internal job interfaces that your service actually exposes. Moving documentation is useful for identifying questions, but the installed release and enabled extensions determine the reachable paths. The current vLLM security documentation warns that its API-key mechanism does not secure every sensitive endpoint on the server. ^[2]

For each supported input route, follow scope assignment to the first cache lookup. Ask whether an adapter preserves the value, a queue serializes it, a retry reconstructs it and a worker validates it. Reject unknown scope versions before lookup rather than accepting them into a default namespace. If the endpoint intentionally does not support reuse, verify that it cannot read shared state indirectly through another feature. The review is incomplete while an alternate route is merely assumed to behave like the main chat endpoint.

Repeat the exercise at every transfer boundary. A remote cache may use one key for object storage, another for index lookup and a third for a transfer session. Require an explicit statement of where the sharing identity is represented and how it is bound to the reusable state. An arbitrary metadata field attached to a transfer is not enough if the receiving lookup ignores it. Do not infer that generic connector support implies release-specific salt propagation. The disaggregated-prefilling documentation establishes the architecture, not that guarantee. ^[6]

Content correctness deserves a separate test set. Changing model weights, an adapter, tokenization or media input must not cause an incompatible block to be reused. That remains necessary even when every request has the right tenant scope. Conversely, a cryptographically strong content hash does not authorize two tenants to share a cache. TensorRT LLM's documentation describes salt isolation through block-key hashing and requires a cryptographic hash for that mechanism. ^[5] Preserve the configured security property when optimizing the key path.

A concrete review artifact can be small. For each hop, record the incoming scope representation, the trusted component that creates it, the outgoing representation, and the lookup or access decision that consumes it. Mark whether the evidence comes from source inspection, configuration inspection or an executed test. This lets a reviewer distinguish 'the field exists in the schema' from 'a different-scope request was shown not to reuse a resident block.' Both are useful, but they are not interchangeable.

The before-and-after figure is a conceptual contrast. On the left, authentication feeds an unscoped pool. On the right, trusted assignment precedes separated reuse keys, and an external state path carries the same identity. The right side is not certified merely because it has more boxes. Its benefit depends on the edges being real enforcement relationships. If a storage or connector edge remains unverified, keep it outside the approved design or use a containment option whose behavior is understood.

An often-missed boundary is operational access. An administrator or worker identity with unrestricted read access to retained state may bypass application-level separation. Cache salting addresses lookup sharing; it is not storage encryption, process isolation or an administrator authorization policy. The data-path review should therefore name privileged readers and transfer protections as well as request principals. Avoid claiming that a single configuration parameter supplies all of those controls.

Finish this stage with a deliberately boring question: which exact deployed components are covered? Record runtime build, gateway policy version, connector version, enabled endpoints and storage configuration. The answer should survive a handoff to someone who did not attend the design meeting. Without that inventory, the next performance optimization can quietly move state into a path nobody assessed.

Source. Original engineering comparison informed by vLLM cache salting, prefix design and disaggregated-prefilling documentation reviewed September 2, 2026. ^{[2] [3] [6]}

Method. Conceptual before-and-after design, not an observed deployment or measured improvement. Every proposed edge requires release-specific verification.

Accessible table and figure data

Figure 3 accessible table

Boundary	Unscoped design	Proposed scoped design
Request	Authenticated caller	Authenticated caller plus authorized workflow
Scope	Caller omitted or selected value	Trusted assignment with no public override
Local reuse	One shared lookup population	Separate keys for approved sharing groups
External state	Propagation assumed	Scope retained and consumed at transfer and restore
Unknown path	Default fallback	Reject, disable reuse or use assessed containment

Figure 3 accessible table

Boundary	Unscoped design	Proposed scoped design
Request	Authenticated caller	Authenticated caller plus authorized workflow
Scope	Caller omitted or selected value	Trusted assignment with no public override
Local reuse	One shared lookup population	Separate keys for approved sharing groups
External state	Propagation assumed	Scope retained and consumed at transfer and restore
Unknown path	Default fallback	Reject, disable reuse or use assessed containment

Download imageDownload table CSV

Specify evidence for negative cases

The following acceptance plan is a proposed method, not a report of tests performed for this article. Run it only in an authorized environment with synthetic, non-sensitive fixtures. Choose a prompt long enough to exercise the deployed runtime's relevant cache behavior, but derive that choice from the actual block and model configuration. A historical paper's prompt length is not a universal test requirement for a different serving stack.

First establish a positive control. Submit the same synthetic prefix twice under one approved scope and inspect trusted instrumentation showing whether the intended reusable state was found. Record request correlation, worker identity, model revision and the cache plane involved. If no reuse occurs, the negative test is not yet informative: different-scope requests might fail to reuse because nothing was cached in the first place. Repair the test conditions before interpreting the isolation result.

Then warm the fixture under scope A and submit it under scope B. The acceptance condition is no cross-scope reuse on the path being tested, not merely a slower response. Where available, use restricted cache metrics or traces that identify the lookup result without retaining prompt contents or secrets. Repeat the sequence through each approved endpoint and transfer mode. Keep timing distributions as supporting observations when useful, while retaining the narrower conclusions warranted by the available instrumentation.

Exercise omission and forgery explicitly. A request without an assigned scope must take the documented rejection or containment path. A client-supplied value purporting to be another group's scope must not replace trusted assignment. An unknown scope version must not become the unsalted default. A retry, batch conversion or queued-job replay must preserve the same policy. These cases can be tested with fabricated tenant identities and private fixtures; there is no need to probe another real tenant.

Add lifecycle cases because the boundary persists beyond a single request. Warm state, restart the relevant component and examine restored behavior. Exercise an offload and reload if that feature is enabled. Rotate a scope while work is in flight. Remove a test principal from its group. Trigger the configured fallback path in a controlled environment. For each case, define the expected decision before running it and retain enough configuration evidence to reproduce the result.

Separate a failed test from a missing observation. If trusted instrumentation demonstrates a cross-scope hit, the boundary failed for that configuration. If instrumentation cannot identify whether state was reused, the evidence is incomplete. If the test never produced a same-scope hit, its setup failed to exercise the feature. These distinctions prevent a broad 'passed' label from hiding an untested connector or a disabled cache. The historical audit's nondetection limitation makes this discipline especially important. ^[1]

Performance checks follow the security decision. Measure representative request mixes, concurrency, prefix lengths and warm-state conditions under the chosen sharing policy. Compare results against the service's own latency and resource objectives. Do not present an invented percentage cost for per-user isolation, and do not enlarge the sharing group solely to rescue a benchmark. A resource-budget problem may instead call for scheduling, capacity or workload changes.

Deliver the evidence as a scoped release record: configurations exercised, expected outcomes, observed outcomes, unresolved paths and approving owner. Preserve failed cases and limitations as well as successful ones. A new connector or endpoint should reopen the relevant part of the record. The purpose is not to prove that the entire inference system can never leak information; it is to support one precise assertion about permitted prefix reuse.

Operate rotation and exceptions

Cache isolation is operational state, not a one-time checklist. Assign an owner for scope issuance, an owner for the serving path and an owner for retained copies. They may be the same team, but the responsibilities should remain explicit. A release that changes runtime, connector, model family, media handling or endpoint exposure can invalidate part of the earlier evidence. Treat these changes as review triggers rather than assuming that an unchanged product name means unchanged behavior.

Build alerts around policy failures you can explain. Examples include a missing assigned scope on a shared route, an unknown scope version at a worker, use of a prohibited fallback or a connector configuration outside the approved inventory. These are suggested control signals, not vendor-provided metrics or measured incident indicators. Avoid logging secret values to make the alerts convenient. An internal scope reference and policy version should normally be enough for an authorized investigation.

Exceptions need a containment decision and an expiry. A temporary dedicated pool can be a reasonable way to support sensitive work while a shared connector is evaluated. Disabling cross-request reuse can also be acceptable if capacity and availability remain adequate. Neither choice removes the need to protect logs, response caches or storage. State exactly which risk the exception addresses and which surrounding controls remain required.

Be careful with the phrase dedicated runtime. It should name the boundary that is actually separate: process, cache namespace, storage, worker pool or physical capacity. A dedicated endpoint that still reads a shared external cache may not satisfy the intended policy. Similarly, two deployments with different credentials can share a backing store. Ask for a concrete state-path description instead of relying on a service label.

Retire old namespaces deliberately. Record when new requests stopped using them, how in-flight work finished, which retained copies were removed and which backups or storage layers remain under a documented lifecycle. If the service cannot verify immediate erasure, say so. A change that prevents lookup and a deletion that removes bytes are separate outcomes, and both may matter to the data owner.

The release decision can be concise: this defined population may share this prefix state through these verified components under this policy version. Everything else is rejected, uncached or contained in a separately assessed runtime. That statement is more useful than 'prefix caching is secure' because it tells operators exactly what must remain true. When the system changes, it also tells them where to look first.