Output verbosity regressions: detect and cap completion tokens

What to send (payload example)

{
  "externalRequestId": "req_01HZXB6MQZ2WQ9D2KCF9M4V2QY",
  "provider": "provider_id",
  "model": "model_id",
  "endpointTag": "checkout.ai_summary",
  "promptVersion": "summary_v3",
  "userId": "tenant_acme_hash",
  "inputTokens": 540,
  "outputTokens": 180,
  "latencyMs": 892,
  "status": "success",
  "dataMode": "real",
  "environment": "prod"
}

Common mistakes

• Only tracking average output tokens and missing tail outliers (p95/p99).
• Capping tokens globally instead of scoping caps per endpoint criticality.
• Shipping prompt changes without promptVersion tagging and a canary baseline.
• Optimizing token price while retries or multi-call flows increase total cost.

How to verify in the Opsmeter.io dashboard

1. Use Overview to confirm spike window and budget posture.
2. Use Top Endpoints to find feature-level concentration.
3. Use Top Users to find tenant-level concentration.
4. Use Prompt Versions to validate deploy-linked cost drift.

Common regression pattern

• Prompt wording increases answer style verbosity.
• Fallback models return longer completions by default.
• Post-processing asks the model for redundant rewrites.

Containment checklist

1. Set max output tokens by endpoint criticality.
2. Track avgOutputTokens by promptVersion every release.
3. Alert when completion-token growth exceeds baseline.
4. Review long-tail outliers in Top Users and Top Endpoints.

How to detect verbosity regressions early (signals that matter)

Token inflation is usually visible before totals blow up. The earliest signal is completion tokens per request drifting upward right after a deploy.

Track both averages and tail percentiles (p95/p99). A small number of long responses can dominate spend even when averages look stable.

• avgOutputTokens and p95OutputTokens by endpointTag and promptVersion
• cost/request drift without a matching traffic-volume increase
• latency regression that correlates with completion length growth
• outlier sampling: inspect the longest completions weekly
• rewrite-loop detection: multiple calls per user action

Release gate checklist (make regressions harder to ship)

1. Define a response contract (format + max length) per endpointTag.
2. Set default max output tokens and override only with justification.
3. Compare promptVersion outputs on a fixed evaluation set (quality + length).
4. Ship a canary and alert on outputTokens/request delta vs baseline.
5. Record promptVersion change notes so incidents have a paper trail.

Degraded-mode patterns (reduce spend without breaking the product)

• Prefer concise bullet summaries instead of long prose during incidents.
• Skip redundant rewrites (draft -> rewrite -> rewrite) unless the endpoint truly needs it.
• Disable optional tool calls and multi-step chains temporarily.
• Return a short answer first and offer an explicit "expand" follow-up path if needed.

Fixes that preserve quality while shrinking completions

• Add a response contract (format + length) and enforce it per endpointTag.
• Reduce "self-rewrite" loops (draft -> rewrite -> rewrite) unless needed.
• Prefer structured outputs (tables, bullet points) over long prose.
• Use a cheaper model for rewriting only after you cap the first pass.
• Measure the tradeoff: quality score versus outputTokens per promptVersion.

Related guides

• Full guide: Prompt deploy cost regressions: catch silent cost spikes
• Full guide: LLM Cost Reduction Playbook
• How to find root cause of a cost spike
• LLM cost attribution pillar
• LLM budget alerts pillar
• No-proxy telemetry quickstart
• Quickstart (5-minute setup)
• Alternatives and tradeoffs

Evaluation resources

For security and procurement reviews, use our trust summary before final tool selection.