Monitoring
How to detect LLM cost spikes before month-end
Month-end surprises usually mean daily controls are missing. You need burn-rate views plus clear action owners.
Full guide: LLM budget alert policy: thresholds and escalation
What to monitor daily
- Spend versus budget trajectory
- Cost/request drift per endpoint and promptVersion
- Tenant concentration changes
- Retry or latency anomalies that inflate request volume
Alert thresholds that work
- Configure warning threshold before the hard cap.
- Route exceeded state to clear owner channels.
- Require response runbook for every exceeded alert.
- Review 24-hour delta and top three spend drivers.
Action sequence after a spike
- Identify the dominant endpoint and tenant.
- Check recent promptVersion deployments.
- Apply temporary model tiering for non-critical paths.
- Contain retries and abusive traffic patterns.
Burn-rate math (simple and effective)
Month-end surprises usually mean you did not run a daily projection. Burn-rate checks are a lightweight way to detect drift early.
Use a projection to decide when to escalate, then use attribution to explain the driver.
- Projected month-end spend = (Spend so far / days elapsed) * days in month
- Confirm with baseline: compare spend/day and cost/request to last 7/14/30 days
- Escalate when both projection and burn-rate indicate abnormal change
What to include in an alert (so teams can act fast)
- budget state (warning/exceeded) and projection vs budget
- top endpointTag drivers and their cost/request deltas
- top tenants/users and concentration %
- promptVersion changes in the same window
- retry ratio and status distribution
Common mistakes that delay detection
- Reviewing totals weekly instead of daily burn-rate checks.
- Not separating demo/test traffic from production (dataMode).
- Alerting without ownership (no one knows who should respond).
- Optimizing prompts before containment (spend keeps growing).
- Ignoring tail outliers where regressions hide.
A lightweight weekly routine (keeps you out of trouble)
- Review top endpoints and tenants by spend.
- Review promptVersion changes and token deltas after deploys.
- Review retries and error rates for hidden multipliers.
- Adjust thresholds based on traffic growth and pricing changes.
- Write one action item: cap, routing rule, quota, or runbook update.
What to alert on
- burn-rate acceleration vs baseline
- endpointTag concentration changes in short windows
- unexpected tenant concentration in Top Users
- budget warning, spend-alert, and exceeded state transitions
Execution checklist
- Confirm alert is real: dataMode, environment, and time window.
- Identify dominant endpointTag and tenant/user contributors.
- Contain: cap output, lower max tokens, or throttle non-critical paths.
- Assign one incident owner and one communication channel.
- Update policy thresholds or ownership to prevent repeat incidents.
FAQ
Is userId required?
No. userId is optional, but recommended for tenant-level attribution. If needed, send a hashed identifier.
Where should token usage values come from?
Prefer provider usage fields first. If unavailable, use tokenizer estimates and mark uncertainty in your workflow.
How should retries be handled?
Keep the same externalRequestId for the same logical request so idempotency remains stable across retries.
Can telemetry break production flow?
It should not. Use short timeouts, catch errors, and keep telemetry asynchronous so provider calls keep running.
Related guides
Evaluation resources
For security and procurement reviews, use our trust summary before final tool selection.