Budget and spend limits

Verified

Control LLM spending by enforcing token budget limits per API key or user.

Advanced patterns for enforcing token budget limits per API key or user.

About

Budget limits (also known as spend limits or quota management) help you control LLM costs by restricting how many tokens each user or API key can consume within a time window. This prevents runaway spending and ensures fair resource allocation across teams and applications.

This guide focuses on advanced patterns that are not covered in the virtual keys guide, such as per-route budgets, local rate limiting, and cost calculations.

How budget limits work

Budget limits enforce token consumption quotas using token bucket rate limiting. Each user or API key gets a virtual “budget” measured in tokens rather than requests.

Key concepts:

Token bucket: A virtual bucket that holds a certain number of tokens (your budget)
Token consumption: Each LLM request consumes tokens based on the input + output token count
Refill interval: How often the bucket refills (e.g., daily, hourly)
Keying: How to identify users (by header, JWT claim, or remote address)

When a request arrives:

    flowchart TD
  A[Request arrives] --> B[Validate API key]
  B --> C[Count against token budget]
  C --> D{Budget available?}
  subgraph refill["Budget refills periodically"]
    D
  end
  D -->|Yes| E[Request proceeds]
  D -->|No| F[Reject with 429]

Agentgateway validates the API key (if required)
The request is counted against the user’s token budget
If the budget has tokens available, the request proceeds
If the budget is exhausted, the request is rejected with a 429 status code
The bucket refills at the configured interval

More considerations

Evaluation order: Rate limiting is evaluated before prompt guards (content safety checks). This means that requests rejected by guardrails (403 Forbidden) still consume quota from the user’s token budget. In contrast, authentication (JWT/OPA) is evaluated before rate limiting, so unauthenticated requests do not consume quota.

Multiple policies: When multiple AgentgatewayPolicy resources target the same Gateway or HTTPRoute with overlapping backend.ai fields, one policy silently overwrites the other based on creation order. Both policies will show ACCEPTED/ATTACHED status. To avoid conflicts, use separate policies for different configuration areas (such as one for authentication, one for rate limiting, one for prompt guards).

Before you begin

Complete the Virtual key management guide to:

Create API keys for users
Configure API key authentication
Set up token-based rate limiting
Configure the rate limit server

Per-route budget limits

Apply different budgets to different routes, such as higher limits for production and lower limits for development.

Create separate AgentgatewayPolicy resources for each HTTPRoute instead of targeting the Gateway.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: prod-token-budget
  namespace: agentgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: openai-prod
  traffic:
    rateLimit:
      global:
        domain: token-budgets
        backendRef:
          kind: Service
          name: rate-limit-server
          namespace: agentgateway-system
          port: 8081
        descriptors:
          - entries:
              - name: route
                expression: '"prod"'
              - name: user_id
                expression: 'request.headers["x-user-id"]'
            unit: Tokens

Configure the rate limit server with nested descriptors for route-specific budgets.

domain: token-budgets
descriptors:
  - key: route
    value: "prod"
    descriptors:
      - key: user_id
        rate_limit:
          unit: day
          requests_per_unit: 200000  # Higher limit for prod
  - key: route
    value: "dev"
    descriptors:
      - key: user_id
        rate_limit:
          unit: day
          requests_per_unit: 50000  # Lower limit for dev

Local token budget limits

Use local rate limiting instead of global for simpler setups that don’t require shared state across agentgateway instances.

Limitations of local rate limiting:

The tokens field is request count, not LLM token count. With tokens: 10000, you get ~10,000 requests, regardless of how many LLM tokens each request consumes. For actual LLM-token-based budgets (e.g., limit users to 100K LLM tokens/day), use global rate limiting with unit: Tokens descriptors instead.
Limits apply per agentgateway instance. If you have 3 instances and set a 100,000 token limit, each instance enforces 100,000 tokens, for a total of 300,000 tokens across all instances.
Local rate limiting only supports Seconds, Minutes, and Hours units. For daily budgets, use global rate limiting instead, which supports day-based limits.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: local-token-budget
  namespace: agentgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway-proxy
  traffic:
    rateLimit:
      local:
        - tokens: 10000
          unit: Hours

Monitor budget usage

Track how much of each user’s budget has been consumed using Prometheus metrics.

Port-forward the agentgateway proxy metrics endpoint.

kubectl port-forward deployment/agentgateway-proxy -n agentgateway-system 15020

Query the token usage metric filtered by user.

# Total tokens consumed by user over the last 24 hours
sum by (user_id) (
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h]) +
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])
)

# Percentage of daily budget used
(sum by (user_id) (
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h]) +
  increase(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h])
) / 100000) * 100

Set up alerts when users approach their budget limits.

groups:
- name: budget_alerts
  rules:
  - alert: BudgetNearlyExhausted
    expr: |
      (sum by (user_id) (
        rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input"}[24h]) * 86400 +
        rate(agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output"}[24h]) * 86400
      ) / 100000) > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "User {{ $labels.user_id }} has used over 80% of their daily token budget"

Convert budget to cost

To convert token budgets to dollar amounts, multiply by your provider’s pricing.

For example, with OpenAI GPT-4:

Input tokens: $30 per 1M tokens
Output tokens: $60 per 1M tokens

A 100,000 token budget (assuming 50/50 input/output mix):

cost = (50,000 / 1,000,000 × $30) + (50,000 / 1,000,000 × $60)
     = $1.50 + $3.00
     = $4.50 per day

For more information on cost calculation, see the cost tracking guide.

Enforce a dollar budget

The budgets in the previous sections are measured in tokens. If you configure a model cost catalog, agentgateway computes the realized USD cost of each request and exposes it to CEL as llm.cost. You can then rate limit on dollars directly, which enforces a true spend cap regardless of which model or input/output token mix each user hits.

Known limitation: Cost-based (dollar) rate limiting is not enforced correctly on the current build. The response-time cost amendment fails with an empty-descriptor error, so the budget is not applied. Token-based budgets (the sections above) are unaffected. Track the fix before relying on dollar enforcement in production. This section documents the intended configuration.

Requirements and behavior:

A model cost catalog must be configured so that llm.cost is populated. Without a catalog, llm.cost.total is 0 and no spend is counted.
Cost is evaluated after the response completes. The request that crosses the budget still completes; the user’s next request is rejected with a 429. Budgets are therefore approximate at the boundary.
The rate limit cost expression must return an unsigned integer. Because USD costs are fractional (for example, $0.0000057), scale them to micro-dollars (USD × 1,000,000) with uint(). A budget of 1000000 micro-dollars equals $1.00.

Create a AgentgatewayPolicy with a global rate limit descriptor that uses unit: Tokens and a cost expression that converts the realized cost to micro-dollars. This example keys the budget on the API key’s user_id metadata, so it builds on the API key authentication from the Virtual key management guide.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: dollar-spend-limit
  namespace: agentgateway-system
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: agentgateway-proxy
  traffic:
    rateLimit:
      global:
        domain: spend-budgets
        backendRef:
          kind: Service
          name: ratelimit
          namespace: agentgateway-system
          port: 8081
        descriptors:
          - entries:
              - name: user_id
                expression: apiKey.user_id
            unit: Tokens
            # Realized USD cost scaled to integer micro-dollars (USD x 1,000,000).
            cost: 'uint(llm.cost.total * 1000000)'

Configure the rate limit server with the user’s daily budget expressed in micro-dollars. Reuse the same rate limit server that you deployed for token budgets in the Virtual key management guide. Dollar enforcement uses the identical Envoy rate limit service and protocol. Add the spend-budgets domain to the server’s configuration so that it does not collide with your token-budget descriptors. This example caps each user at $1.00 per day (1000000 micro-dollars).
```
domain: spend-budgets
descriptors:
  - key: user_id
    rate_limit:
      unit: day
      requests_per_unit: 1000000  # $1.00/day in micro-dollars
```
Send requests as a user. After each response, agentgateway computes the request’s micro-dollar cost and sends it to the rate limit server as the request’s cost. When the user’s accumulated spend reaches the daily budget, further requests are rejected with a 429 until the budget refills.

To budget on a single cost component instead of the total, use that field in the expression. For example, cost: 'uint(llm.cost.output * 1000000)' budgets only on output-token spend.

Cleanup

You can remove the resources that you created in this guide.

kubectl delete AgentgatewayPolicy api-key-auth per-user-metrics -n agentgateway-system --ignore-not-found
kubectl delete secret llm-api-keys -n agentgateway-system
kubectl delete httproute openai -n agentgateway-system
kubectl delete AgentgatewayBackend openai -n agentgateway-system

To remove the rate limit server, follow the cleanup steps in the global rate limiting guide.

LLM cost tracking

Was this page helpful?