Incident response
Budget exceeded: response playbook for LLM product teams
Exceeded status is an incident trigger. Teams need a fixed response sequence, not ad-hoc Slack threads.
Full guide: LLM budget alert policy: thresholds and escalation
First-hour sequence
- Validate spike window and recent deploy history.
- Identify top endpoint and top tenant contributors.
- Pause non-critical feature paths by internal policy.
- Publish owner + ETA for remediation updates.
Ownership model (avoid ad-hoc Slack threads)
Exceeded events are cross-functional incidents: engineering contains spend, product decides degraded modes, and finance needs an audit trail.
Define roles ahead of time so “who owns this” does not consume the first 30 minutes.
- Incident owner: drives triage, updates, and rollback decisions.
- Feature owner: mitigates the dominant endpointTag driver (caps, routing, prompt rollback).
- Finance/ops: confirms reporting window (UTC) and tracks cost impact.
- Security owner: investigates abuse/leaked keys when unknown-user bursts appear.
Communication template (what to post in the first update)
- Window: start time, burn-rate, and projected impact if unchanged.
- Driver: top endpointTag and top tenant/user (if applicable).
- Containment: what changed (caps/throttles/routing) and user impact.
- Next ETA: when you will re-evaluate and whether rollback is planned.
Post-incident requirements
- Document root cause and threshold updates.
- Add regression checks to release checklist.
- Review whether plan tier and budgets still match volume.
Containment options that preserve user experience (safe order)
- Throttle batch jobs and internal tooling first (avoid breaking core UX).
- Cap output tokens on public and long-form endpoints (immediate spend control).
- Reduce context size (topK, chunk overlap) on RAG flows during the incident window.
- Disable optional tool calls and multi-step agent loops until stable.
- Route low-risk traffic to smaller models while you stabilize and verify.
- Communicate impact clearly: what changed, user impact, and revert criteria.
What to alert on
- budget warning/exceeded state with burn-rate above baseline
- endpointTag concentration shift (dominant feature share jumps)
- cost/request drift after a promptVersion change
- tokens/request inflation (input or output) on critical endpoints
- unknown-user or tenant concentration spikes that suggest abuse or misuse
Execution checklist
- Assign an incident owner and start an incident note (time window, driver, actions).
- Contain first: caps, throttles, routing, and degraded modes scoped by endpointTag.
- Stop multipliers: fix retry storms, timeouts, and abuse patterns.
- Verify: cost/request and concentration return to baseline before reverting mitigations.
- Harden: update thresholds, release gates, and ownership rules so the incident cannot repeat silently.
FAQ
Is "budget exceeded" a finance problem or an engineering problem?
Both. Finance cares about the invoice, but engineering owns the operational driver (endpointTag, promptVersion, retries, abuse). Treat exceeded events as incidents with shared ownership and an audit trail.
Should we stop the product when the budget is exceeded?
Usually no. Prefer degraded modes first: shorter outputs, smaller context, fewer tool calls, and throttling non-critical endpoints. Use hard blocks only for abuse-prone or low-criticality routes.
What is the fastest way to reduce spend without shipping a code change?
Scope containment by endpointTag: throttle or disable non-critical flows, reduce max output tokens, and route low-risk traffic to cheaper models. Then verify cost/request and concentration trends before reverting.
How do we prevent exceeded events from repeating next month?
Make the driver attributable and enforceable: promptVersion tagging, endpoint-level budgets and caps, burn-rate alerts with a named owner, and release gates that check tokens/request deltas before rollout.
Related guides
Evaluation resources
For security and procurement reviews, use our trust summary before final tool selection.