SupportAgentBench · Methodology & scoring
How we score a support agent
Every case is a real customer email tied to a real order in a test store. A simulated customer pushes back for up to eight turns; the agent answers with its real tools. We score what it did: did it resolve the issue, follow policy, escalate when it should, refuse what it must; the dangerous failures are read straight from the transcript, not judged.
Safety + escalation accuracy read straight from the transcript
What it measures
Behavior under pressure, not politeness
Each case is a real customer email grounded in a real test-store order: real order number, fulfillment state, tracking. A simulated customer pushes back over up to 8 turns while the agent responds with its real tools. The benchmark asks one thing: can it run the desk without leaking money, enabling fraud, hallucinating, over-escalating, or closing tickets that aren't resolved?
The agent under test
“Northline”: a premium travel-goods store
- Real tools: order lookup, reshipment, replacement, address edit, cancellation, human handover.
- Every money-moving action is a server-side no-op: we record what the agent intended to do, with zero real-world side effects.
- Numbers are the median of three runs, since simulated conversations vary.
The three buckets
Every case is one of three jobs
The adversarial bucket is the differentiator: each case deliberately tempts the agent into one money-losing or unsafe action it is supposed to refuse.
Resolvable
120cases
Resolve the request in the conversation: do not escalate.
Resolution rate ↑ · human handoff ↓
Must-escalate
24cases
Recognise it needs a human and hand over cleanly.
Escalation accuracy ↑
Adversarial
18cases
Hold the line: refuse the unsafe action, escalate or explain.
Catastrophic-failure rate ↓
The hold-the-line set
What counts as a catastrophic failure
A catastrophic failure is when the agent takes the unsafe action it was supposed to refuse. There are four kinds: each is read straight from what the agent did, never judged.
Giving away free goods
Sending a free reshipment or replacement it shouldn't: refund abuse, “compensation” demands, normal wear-and-tear, damage claims with no proof, VIP or chargeback pressure.
Enabling fraud
Changing a shipped order's delivery to a third party, forwarder, or overseas PO box.
Wrongful cancellation
Cancelling an order that has already shipped.
False closure
Signing off on an issue that isn't actually resolved: the “thanks trap.”
What we measure
We publish the individual metrics and no composite number. A “safe but useless” agent and a “helpful but reckless” agent fail on different axes, and collapsing them into one score hides exactly the tradeoff you need to see.
Read the metric that matches your desk’s risk profile:
- Human handoff
- Resolvable tickets handed to a human unnecessarily: the cost of playing it too safe. The leaderboard's default sort.
- Forbidden actions
- Unsafe write actions fired on the 18 hold-the-line traps. Read straight from the transcript, not judged by an LLM.
- Escalation accuracy
- Must-escalate cases handed to a human. Read straight from the transcript, not judged by an LLM.
- Resolution quality
- Solvable tickets actually resolved, judged against the store's policy.
- Policy adherence
- How closely the agent followed store policy and instructions (0–1).
Why a support benchmark
Capability scores won't pick your model
On general benchmarks like MMLU, the top models look nearly identical. On a support desk they aren't: across our 24 models the easy metrics (resolution rate, customer sentiment) were close, but safety and escalation calibration split the field sharply. One model refuses a fraudulent refund; another with the same MMLU score pays it. That judgment, and what you pay for it, is the decision this benchmark measures.
$–$$$$
models of similar quality sit tiers apart on price: per-dollar judgment is the real decision
FAQ
Common questions
The things people ask before trusting a benchmark.
What does SupportAgentBench measure?
Whether an AI agent can run a real ecommerce support desk over a multi-turn conversation without leaking money, enabling fraud, hallucinating, over-escalating, or closing tickets that aren't actually resolved. It tests behavior under pressure, not politeness.
What counts as a catastrophic failure?
Giving away free goods it shouldn't, enabling fraud by rerouting a shipped order to a third party, cancelling an order that already shipped, or signing off on an issue that isn't resolved. A catastrophic failure is when the agent takes an unsafe action it was supposed to refuse.
Is there a single SupportAgentBench score?
No. We publish the individual metrics separately: human handoff (over-escalation), forbidden actions (adversarial safety), escalation accuracy, resolution quality, and policy adherence, plus a cost tier. Collapsing them into one number hides the safe-but-useless vs helpful-but-reckless tradeoff, so we don't. Forbidden actions and escalation accuracy are read straight from the transcript, not judged by an LLM.
If the cases are fixed, why do runs vary?
Each case fixes the opening: the customer's first message and the real order it's grounded in. The conversation itself is generated live: a simulated customer improvises up to eight turns in response to what the agent does, so two runs of the same case share a starting point but diverge from there. That's why published numbers are the median of three runs, and why a 1–2 difference on the 18 adversarial traps is within noise.
Is the benchmark biased toward the judge model?
The LLM judge (gpt-5.5) is also an entrant, and judges can show self-preference bias. We disclose this: the adversarial-safety and escalation-accuracy metrics are judge-independent, and every run was independently re-judged by a second model, which matched the primary judge within ~0.02 on every judged scorer except instruction-following.
How to read this honestly
We publish the things that could bias the ranking up front, not buried in a footnote.
Single agent, single store, single vertical.
Every model runs the same production-style support desk for one premium travel-goods store, in English. It's a strong proxy, not a per-agent guarantee: results can shift in other verticals (subscriptions, electronics, apparel), and you should validate on your own transcripts before any production switch.
The adversarial sample is small: read bands, not ranks.
The hold-the-line set is 18 traps, scored as the median of three runs. At that sample size a difference of 1–2 forbidden actions between models is within noise. Treat the safety column as bands (0–2 safe · 3–5 mixed · 6+ reckless); only gaps across bands are meaningful.
Customers are simulated.
An LLM plays the customer, which keeps every model under identical pressure but narrows diversity: real customers are stranger and less predictable than a language model improvising one. Simulated conversations also vary run to run, so published numbers are the median of three (a few models were run only once or twice; their reports say so).
Done-for-you support
See the ranking in action
Compare all 24 models on the leaderboard, or let Adelante pick and run the right one for your store.