Hard vs soft caps for AI spend control

Definitions (what teams mean by caps)

A soft cap is a budget policy that triggers alerts and workflows. A hard cap is enforcement that blocks or degrades requests once a threshold is reached.

Most teams should treat caps as per-endpoint policies (endpointTag), not a single global toggle.

• Soft cap: warn early and contain with human decision loops.
• Hard cap: enforce boundaries for non-critical or abuse-prone endpoints.
• Degraded mode: keep the product usable while reducing cost (smaller context, shorter outputs, fewer tools).

When soft caps are enough

• You have reliable on-call ownership.
• Spikes are infrequent and quickly reversible.
• User experience impact from hard blocks is unacceptable.

When hard caps are required

• Regulated budget boundaries must be enforced.
• Abuse events can escalate quickly.
• High-variance tenants can consume shared margin.

Scope caps by feature criticality (endpointTag)

Hard caps on critical user paths can create outages. The safer pattern is to classify endpoints by business value and user impact, then apply different guardrails.

Put the strictest controls on endpoints that are expensive, public-facing, or easy to abuse.

• Tier 1: revenue-critical endpoints - soft cap + fast escalation + degraded mode.
• Tier 2: helpful but non-critical endpoints - soft cap + automatic degrade at warning.
• Tier 3: experimental or abuse-prone endpoints - hard cap with clear user messaging.

A hybrid policy that works for most teams

1. Use soft caps (warning) for critical user paths to avoid outages.
2. Use hard caps for non-critical workflows and abuse-prone endpoints.
3. Degrade gracefully: cheaper models, smaller context, shorter outputs.
4. Escalate to hard blocks only after burn-rate confirmation.
5. Review outcomes weekly and retune thresholds by endpoint criticality.

What to send (payload example)

{
  "externalRequestId": "req_01HZXB6MQZ2WQ9D2KCF9M4V2QY",
  "provider": "provider_id",
  "model": "model_id",
  "endpointTag": "billing.guardrail_check",
  "promptVersion": "budget_v1",
  "userId": "tenant_acme_hash",
  "inputTokens": 240,
  "outputTokens": 80,
  "latencyMs": 892,
  "status": "success",
  "dataMode": "real",
  "environment": "prod"
}

Common mistakes

• Setting static thresholds without burn-rate checks.
• No single owner or escalation path for warning/exceeded states.
• Alerting on totals only (missing endpoint and tenant concentration context).
• Including demo/staging traffic in production spend policy decisions.

How to verify in Opsmeter Dashboard

1. Use Overview to confirm spike window and budget posture.
2. Use Top Endpoints to find feature-level concentration.
3. Use Top Users to find tenant-level concentration.
4. Use Prompt Versions to validate deploy-linked cost drift.

Related guides

• Full guide: LLM budget alert policy: thresholds and escalation
• Budget exceeded response playbook
• LLM cost attribution pillar
• No-proxy telemetry quickstart
• Quickstart (5-minute setup)
• Alternatives and tradeoffs
• Prompt deploy regressions pillar

Evaluation resources

For security and procurement reviews, use our trust summary before final tool selection.