Coval vs Hamming vs Cekura vs Bluejay: You Are Buying a Simulated Caller

At a glance

All four run automated calls against a voice agent, score the result, and flag regressions. Two publish prices and two do not.

The axis nobody sells on

Every one of these platforms is really two products stapled together: a thing that produces callers, and a thing that decides whether the call went well. The second half is broadly commoditised — a transcript, some assertions, an LLM judge, a dashboard. The first half is where the money and the difference are, and it is the half that determines whether the number at the end means anything.

A generated persona is written from a description you supplied, which means it exhibits the failure modes you already thought of. That is genuinely useful before launch, when you have nothing else, and it is structurally incapable of surprising you. Replayed production audio has the opposite property: it contains the accents, the cross-talk, the hold music bleeding through, the caller who changes their mind halfway through a sentence — every distribution feature you did not know to write down — and it exists only if you already have an agent in production taking real calls.

This is why the choice is not really a feature comparison. If you have no production audio, a replay-first product has nothing to replay. If you do have production audio, evaluating exclusively against personas you invented is a way of grading your own imagination. Most teams need both eventually, and almost every team buys the wrong one first, because they shop on the demo rather than on the corpus.

Coval's positioning is the interesting third answer. Its founder led evaluation infrastructure at Waymo, and the methodology it imports from autonomous-vehicle testing is not "simulate more" but "pin a scenario set and re-run it on every change" — the value is in the suite being fixed and the comparison being longitudinal, which is a different claim from realism. That maps cleanly onto eval-driven development, and it is the reason its CI integration gets more attention than its persona quality.

Price is a coverage decision in disguise

Cekura's credit model works out to roughly twenty cents for a one-minute simulated call. Coval publishes a $100 entry tier, a $500 growth tier and enterprise from $4,500 a month. Hamming prices on volume rather than seats and shares numbers only after a demo; Bluejay publishes nothing.

The temptation is to read this as cheap versus expensive. Read it instead as a constraint on suite size, because that is how it will actually bite. A per-minute price you can compute in advance lets you decide to run four hundred variations of the one interaction that keeps failing. A tier you have already paid for, or a sales-led contract with a negotiated volume, pushes teams toward a small curated set of calls that everyone agrees are representative — and a small curated set is exactly the thing a voice agent does not fail on. The failures live in the tail: the fifth-percentile accent, the caller who says "no, wait" over the confirmation, the transfer that lands in a queue.

So the question to take into a pricing conversation is not the monthly figure. It is: what does the four-hundredth variation of this scenario cost, and can I compute that number without emailing anyone? For a team whose whole problem is long-tail coverage, an unpublished enterprise price is a real product limitation, not just a procurement annoyance.

The corresponding warning applies to the cheap end. Twenty cents a call makes it rational to run enormous suites, and enormous suites of near-identical generated personas produce a confident pass rate over a distribution nobody validated. Volume is only worth buying once the caller distribution is worth sampling — the same trap our page on eval-set maintenance describes for text agents, arriving here with a phone bill attached.

The scoring problem is not the simulation problem

Ask any team running voice evals what hurts and you will hear the same complaint: the judge is flaky. The same call scores differently on re-run, a prompt tweak moves the score with no change in behaviour, and nobody trusts the dashboard enough to block a release on it.

Two of these four attack that from opposite ends, and the contrast is the most substantive technical difference in the set. Hamming's pitch is audio-native evaluation — scoring the audio signal rather than only the transcript — which raises the fidelity of what the judge is looking at and is the only approach that can see the failures that never reach a transcript at all. Cekura's Conditional Actions is a rule-based testing engine explicitly framed as a way to reduce LLM-judge flakiness: express the check deterministically and the judge stops being in the loop for it.

Both are right, and they are complements rather than alternatives. A voice agent's failures divide cleanly. Content failures — wrong policy quoted, hallucinated availability, missed intent — are semantic, live in the transcript, and genuinely need a judge; treat them the way you would treat any LLM-as-judge setup, with a calibrated rubric and a labelled sample. Interaction failures — the agent talks over the caller, endpointing cuts them off mid-word, a two-second dead air before every reply, DTMF not registering, one-sided audio — are timing and signal properties. They are measurable exactly, they are the ones that make callers hang up, and putting them through a language model is a category error that also happens to be the flakiest part of your suite.

Whatever you buy, split the suite along that line on day one. Deterministic assertions on the interaction properties, a judge only on content, and never a single blended score, because a blended score is what makes a regression in barge-in look like noise. That split is also what makes the harness portable — it is the durable part of the investment, and it survives switching vendors.

Which brings up the number that gets advertised hardest and matters least. Hamming has claimed 1,000-plus concurrent simulations; Cekura claims 2,000-plus. Both are far past the point where your own agent, your telephony provider's rate limits or your model provider's quota becomes the binding constraint. Nobody's release was ever blocked by the harness running out of concurrency.

When to pick which

Two things to do regardless of which name you land on. Run a two-week parallel trial on the same twenty calls — one bake-off against a fixed set tells you more about judge agreement than any feature matrix — and check what you can export. A test suite you cannot take with you converts a tooling decision into a lock-in decision, and the suite, not the vendor, is the asset you are actually building.

Further reading

On this wiki:

• Evaluating voice agents — what to measure before choosing a tool to measure it with.
• Turn-taking & barge-in — the interaction failures a transcript cannot show you.
• Simulated users in agent eval — why the simulator sets the ceiling.
• The latency budget — the number that decides whether callers stay on the line.

Project sources:

• Coval
• Hamming
• Cekura
• Bluejay