A budget model for bounded AI inference

Limit admitted work before the bill arrives

An inference request can consume resources after the caller has stopped waiting. A long input, generous output allowance, repeated retry, or abandoned stream can create work that a simple request counter does not describe. The useful security boundary is the admission decision: before execution starts, can the application identify the tenant, bound the work it is authorizing, and reserve the relevant allowance? If those questions are answered only after a billing notification, the service has observation without a synchronous control.

OWASP's 2025 Unbounded Consumption category recommends controls such as input limits, quotas, resource allocation, deadlines, and graceful degradation.^[1] These controls operate at different layers. None should be renamed a universal spend cap merely because it rejects some requests. A rate limit can reduce arrival pressure while allowing individually expensive requests. A context limit can bound one request while allowing unlimited repetition. An account quota can protect a provider's capacity while leaving one internal tenant able to consume another's allocation.

The design proposed here combines bounded requests with an admission ledger and runtime enforcement. The ledger reserves work before dispatch, tracks the request through completion, and reconciles evidence of actual usage. It also preserves unresolved usage when completion is uncertain. This is an application architecture, not a claim that a particular vendor exposes exactly these transactions or guarantees that cancellation stops all billable work. The implementation must match the provider's actual accounting and execution behavior.

Start by stating the objective. The service may need to prevent one tenant from monopolizing concurrency, constrain cumulative token use, limit spending exposure, or preserve capacity for an essential workflow. These are related but different outcomes. Give each a unit and enforcement point. A budget expressed in currency needs a documented pricing and conversion policy; a token quota needs model-specific counting rules. Without that clarity, an apparently consistent dashboard can compare quantities that do not mean the same thing.

Separate four different kinds of limit

Separate request, tenant, provider, and runtime limits. A request limit constrains input, output, deadline, or allowed operation. A tenant allowance governs cumulative use or concurrent work for an authenticated customer or internal group. Provider quotas apply to the dimensions defined by the service. Runtime settings constrain scheduling and resource use in a particular serving system. The same request can pass one boundary and fail another, so preserve the reason for rejection rather than collapsing every failure into a generic quota message.

API Gateway's HTTP API documentation describes throttling as a best-effort target rather than a guaranteed ceiling.^[2] Its token bucket concerns requests, not language-model tokens. That distinction matters when request sizes vary. AWS Budgets is different again: its documentation says budget data updates up to three times daily, typically at intervals of eight to twelve hours.^[3] Such notifications are useful for cost oversight, but their update cycle cannot serve as the synchronous decision to admit the next inference request.

For self-hosted workloads, Kubernetes ResourceQuota can constrain specified resource requests and limits within a namespace.^[7] It does not identify which application tenant consumed a generated token or implement that tenant's cumulative inference allowance. Similarly, a serving scheduler may limit active sequences without knowing the organization's spending policy. Use these controls for the boundaries they enforce, then add application accounting where the required identity and unit exist. Do not assume that an infrastructure limit automatically implements a business allocation.

Map every execution path that can consume the allowance. Include primary inference, fallback models, retries, background processing, and any service-side expansion the application requests. If an operation can call more than one model, the admission design needs to bound the combined work or reserve each step under an overall envelope. A hidden fallback can bypass an otherwise careful ledger. The goal is a complete accounting boundary for the approved workflow, with explicit exclusions where the application cannot observe or control provider internals.

Reserve capacity before admitting the request

The sequence below begins by resolving identity from authenticated application context. Do not accept a tenant identifier from model output or trust an unverified request field to select the budget account. Validate the operation and input, determine applicable limits, and obtain an input count where the chosen model and request format support it. Amazon Bedrock's CountTokens API provides a model-specific input token count for supported requests; it does not predict generated output or establish a complete cost estimate.^[6]

Reserve a bounded envelope before dispatch. Conceptually, admission requires that available allowance remains after subtracting existing reservations and the new request's maximum authorized work. If the ledger tracks more than one dimension, such as tokens and concurrency, all required reservations must succeed consistently. The exact transaction design depends on the storage system. Two concurrent requests must not both treat the same remaining balance as uncommitted. A read-then-write sequence without suitable concurrency control can defeat the budget at the moment it is needed most.

Bedrock documents its own token-quota reservation and adjustment behavior.^[5] That is useful evidence that quota accounting can involve an initial hold followed by reconciliation, but it is not a specification for the application's ledger. Provider quota units, billable usage, and model-specific output accounting can differ. Keep those quantities separate and verify the current endpoint and model rules. The application can adopt the reserve-and-reconcile pattern without assuming one universal multiplier or copying provider quota values into a financial balance.

Source. Cloud Security Desk conceptual synthesis, 2026-08-28, informed by OWASP and AWS. ^[1][6][5]

Method. Conceptual design, not measured data. Unit: process steps or control relationships; no numeric scale. Scope: Application budget reservations, provider work and usage reconciliation. Limits: Conceptual sequence, not a vendor API contract. Units and cancellation behavior must be verified for the chosen provider and runtime.

Accessible table and figure data

Figure 1 accessible table

Stage	Responsible component	Accounting event
Authenticate	Gateway and application	Resolve the tenant and permitted operation
Bound	Request validator	Count input and enforce output and deadline limits
Reserve	Admission ledger	Atomically hold budget and capacity
Execute	Provider or runtime	Enforce the accepted request limits
Reconcile	Usage collector	Settle known usage or preserve an unresolved reservation

Figure 1 accessible table

Stage	Responsible component	Accounting event
Authenticate	Gateway and application	Resolve the tenant and permitted operation
Bound	Request validator	Count input and enforce output and deadline limits
Reserve	Admission ledger	Atomically hold budget and capacity
Execute	Provider or runtime	Enforce the accepted request limits
Reconcile	Usage collector	Settle known usage or preserve an unresolved reservation

Download imageDownload table CSV

Enforce limits where work actually happens

The executor must receive the limits admission approved. A ledger that reserves a short response while the runtime permits a much longer one is not bounding the authorized work. Apply input bounds, output limits, deadlines, and concurrency controls at the components that can enforce them. Verify the accepted values and any provider defaults. If a requested control is unsupported, admission should use a supported bounded contract or decline the operation rather than record a limit that the executor ignores.

In vLLM 0.21.0, --max-model-len concerns the combined prompt and output context length. The documented --max-num-batched-tokens and --max-num-seqs settings bound work in a scheduler iteration.^[4] Those settings are valuable runtime controls, but they do not implement a cumulative budget for an application tenant. Treat the cited version as part of the configuration record. A later release can change behavior or options, and a deployment should verify its actual serving configuration rather than assume that an example from another version applies.

Concurrency limits deserve separate attention because equal token totals can create different pressure when scheduled together. A bounded queue and per-tenant active-work limit can reduce monopolization, but the queue itself consumes resources and delays responses. Set a maximum wait and reject expired work before dispatch. Recheck the reservation if the request's execution conditions change while queued. A request that was valid for one model or budget period should not quietly run under a different contract because it waited long enough.

Cancellation should reach the executor, not merely close the browser connection. Determine how the provider or runtime acknowledges cancellation and what usage evidence remains available afterward. A local deadline can protect the caller's wait time without proving that remote work stopped. Record those as different observations. If the service cannot bound remote continuation tightly enough for the intended exposure, reduce the admitted envelope or choose a workflow with stronger control. Application timeouts are useful, but they should not be described as guarantees the underlying execution system does not provide.

Follow a disconnected request through settlement

Completion is an accounting event supported by evidence. A successful response may provide usage information that can settle the reservation. A failed response may still correspond to work performed. A client disconnect may leave the final provider usage unavailable to the caller. Design the collector to receive terminal information independently where the architecture permits it, and preserve the request identifier needed to reconcile later records. Do not equate absence of a success response with zero resource consumption.

Consider this hypothetical sequence, proposed as a design exercise rather than a recorded service test. Request A and Request B belong to the same authenticated tenant. Each could fit the remaining allowance on its own, but their combined envelopes would not fit. The admission operation commits A's reservation before deciding B. B must then be rejected or remain outside dispatch until sufficient allowance is available. Checking both requests against the earlier balance and writing their reservations afterward would authorize more work than the stated policy permits.

A is dispatched, then its client disconnects before the collector receives terminal usage. The service records the observed disconnect and moves the accounting record to unresolved. It keeps the hold. Sending a cancellation request records another event, but does not by itself settle the amount or prove that remote execution stopped. The implementation needs the completion or cancellation evidence its provider actually exposes. A local connection state is not a substitute for that evidence, even when the user interface has already reported a timeout.

The caller now retries. A retry of the same admission request should resolve to A's existing reservation and state, without silently dispatching another execution. If policy permits a new inference attempt, it needs its own authorization under the remaining overall envelope. Internal deduplication does not establish provider execution idempotency. A worker that crashes after dispatch but before recording the provider identifier creates an especially important ambiguity: the request may already be running. The recovery path must preserve that possibility instead of assuming the missing identifier means nothing happened.

Suppose a trustworthy terminal usage record later arrives for A. The collector matches the execution, converts the reservation to settled use under the recorded accounting policy, and releases only the unused portion. Reprocessing the same terminal event must not charge A twice or credit the unused portion twice. Keep a settlement identity and a consistent state transition so the retry of an accounting operation cannot mint new allowance. This is a proposed application invariant, not a claim that every provider delivers one terminal event or offers the same reconciliation API.

Budget-period changes need a policy too. If A remains unresolved when a new period begins, do not discard the hold merely because a dashboard resets. Decide whether dispatched work is attributed to its admission period, completion period, or another documented rule, and preserve that association during reconciliation. The same request must not disappear from both periods. Model routing changes require similar care: an envelope calculated for one model should not authorize an alternative model whose resource or financial conversion rules were never reserved.

If terminal usage never arrives, settlement remains a policy decision with a tradeoff. Holding every unresolved request indefinitely can deny service after an evidence outage; releasing every hold can allow repeated ambiguous executions to escape accounting. Assign an owner and a conservative, bounded resolution procedure. Bedrock's documentation illustrates that provider quota units, output accounting and billed use differ, while CountTokens covers supported input counting.^[5][6] Preserve those distinctions when choosing what the application settles. The sequence above defines questions to test; it supplies no measured cost or cancellation guarantee.

Choose a controlled response to overload

Overload is not a single condition. Invalid input, tenant exhaustion, temporary service pressure, and unresolved prior usage require different responses. The decision matrix below separates them before the application admits more work. Reject an unbounded or unauthorized operation before execution. When a tenant's allowance is exhausted, return a clear renewal or retry policy. When shared capacity is temporarily constrained, use a bounded queue or an evaluated degraded mode. Unknown accounting should trigger its own conservative handling rather than masquerading as ordinary traffic.

A queue is useful only when its limits are part of the admission contract. Bound its depth, wait time, and retained payload size. Decide whether queued work reserves tenant allowance and capacity, and release or reconcile those reservations when the request expires before dispatch. A request that times out in the queue should not start later because a worker eventually reaches it. This requires an execution-time check of the request's state and deadline, not merely a timestamp displayed in a monitoring panel.

Degraded modes can reduce work, but they can also change the quality or data handling assumptions that justified the service. A shorter answer, smaller model, or asynchronous workflow should be an explicitly supported mode with its own acceptance evidence. Do not silently route sensitive input to another provider or remove an essential validation step to relieve pressure. OWASP's consumption guidance supports graceful degradation, while the details remain an application decision.^[1] The fallback must preserve the boundaries that matter to the workflow.

Make rejection distinguishable from failure inside the service. A tenant should not repeatedly retry a permanent input violation because it looks like a transient outage. Operators should be able to separate exhausted allowance from provider throttling or a ledger dependency failure. API Gateway's best-effort throttling may help manage arrival pressure, but the application still needs these decisions at its own boundary.^[2] Clear states improve both client behavior and incident evidence without promising that every overload event can be absorbed.

Source. Cloud Security Desk conceptual synthesis, 2026-08-28, informed by OWASP and AWS. ^[1][2][5]

Method. Conceptual design, not measured data. Unit: process steps or control relationships; no numeric scale. Scope: Separate responses to budget exhaustion and temporary service capacity limits. Limits: Conceptual operating policy. Queue lengths, deadlines and allowances are application decisions and no measured capacity is implied.

Accessible table and figure data

Figure 2 accessible table

Condition	Proposed action	Evidence required
Invalid or unbounded request	Reject before execution	Validation result
Tenant allowance exhausted	Reject or await explicit renewal	Consistent tenant accounting
Temporary service pressure	Use a bounded queue or tested degraded mode	Capacity state and deadline
Unknown prior usage	Keep a conservative reservation and investigate	Completion or cancellation evidence
All required limits available	Admit the bounded request	Reservation receipt

Figure 2 accessible table

Condition	Proposed action	Evidence required
Invalid or unbounded request	Reject before execution	Validation result
Tenant allowance exhausted	Reject or await explicit renewal	Consistent tenant accounting
Temporary service pressure	Use a bounded queue or tested degraded mode	Capacity state and deadline
Unknown prior usage	Keep a conservative reservation and investigate	Completion or cancellation evidence
All required limits available	Admit the bounded request	Reservation receipt

Download imageDownload table CSV

Acceptance tests for every terminal state

Test the budget with concurrent and failed requests, not only a successful call that reports usage. A useful suite attempts simultaneous reservations against the same remaining allowance and verifies that admitted work stays within the chosen accounting rule. It also exercises duplicate admission requests, expired queued work, provider timeouts, interrupted streams, cancellation, and delayed usage records. These tests should inspect the ledger state and whether execution occurred, because a correct error response can coexist with unintended background work.

Use deterministic provider stubs to test application accounting, then verify integration behavior against the actual supported service under an authorized bounded test. The stub can reliably produce missing terminal usage or a delayed cancellation acknowledgment. The integration test establishes whether the real client and provider expose the events the design expects. Neither test alone proves the entire boundary. Keep their evidence separate so a passing unit suite does not become an unsupported claim about remote cancellation or billing behavior.

Inspect failure handling for the ledger itself. If the reservation service is unavailable, decide which operations must stop and whether any explicitly bounded essential path may continue. A fail-open path can defeat the cumulative budget; a fail-closed path can affect availability. The choice should be made for the workflow, documented, and tested. If an exception exists, give it a separate finite allowance and evidence trail. Do not let a generic error handler silently turn an accounting outage into unlimited admission.

Include reordered and repeated accounting events. Deliver the terminal usage record twice, restart the collector between receipt and settlement, and let a late record arrive after the request was marked unresolved. The expected assertion is one settlement for one authorized execution, not merely a successful callback response. Also test a queued request whose deadline expires before a worker claims it. The worker must inspect the current state before dispatch, even if the earlier admission check succeeded. These are proposed fixtures, not tests completed for this article.

Before increasing an allowance or enabling a new route, keep the relevant failure evidence with the accounting policy revision. A tokenizer, model, retry or fallback change can alter the work admitted; a collector change can alter how that work is settled. NIST's Generative AI Profile supports evaluation in the intended deployment context.^[8] For this budget, the operational question is whether a missing or repeated event can create permission to do more work. Demonstrate the chosen state transitions in the real integration before claiming that the application enforces that limit.