Your LLM judge is calibrated against human labels, which makes those labels a measuring instrument — and as of November 2026 not one of these four tools will tell you, on a tier you can self-host for free, whether two of your reviewers scored the same item the same way. The queue is the easy part and all four have it. Agreement is the part that decides whether any of the numbers mean anything, and it is legacy in one, paid in another, and simply absent in two.
At a glance
Read at the versions published on 18 November 2026. The first column is the one to choose on, because it decides whether the tool can represent an agent run at all.
| Tool | Version (18 Nov 2026) | Unit of review | Licence |
|---|---|---|---|
| Argilla | 2.8.0 (10 Mar 2025) | A record: fields + questions | Apache-2.0 |
| Label Studio | 1.23.2 (29 Sep 2026) | A task; agent traces on paid tiers | Apache-2.0 |
| Langfuse | 4.82.0 (17 Nov 2026) | Trace, observation or session | MIT, except ee/ |
| Opik | 2.2.146 (17 Nov 2026) | Trace or thread | Apache-2.0 |
Note the date in Argilla's row. It is not a typo.
The unit of review splits the field in half
These four are usually compared as if they were four implementations of "human feedback on LLM output". They are not. Two of them come from the dataset-annotation tradition, where the object under review is a record — a bundle of fields shown to an annotator with questions attached. Two come from the tracing tradition, where the object is a run — a trace with nested observations, tool calls, retries and a final output.
For a chat completion the distinction barely matters. For an agent it is the whole question, because the thing a reviewer most needs to judge is not the final answer but the path: which tool the agent called, what came back, and where the run went wrong. A record can hold the transcript as text. It cannot hold the structure, which means a reviewer cannot score step four and leave step three alone.
Argilla makes this concrete. Its own documentation lists what a field can be — "plain text and markdown through the TextField, images through the ImageField, chat formatted data through the ChatField and full custom templates through our CustomField" — and there is no trace, observation or span concept anywhere in the product. ChatField gives you role-and-content message lists, which covers a multi-turn conversation but not an execution tree. (Argilla does have a SpanQuestion, which is text-span labelling for named-entity work, not an execution span. The name collision has misled people.)
Langfuse sits at the other end: its annotation queues are "built for domain experts to add scores and comments to traces, observations or sessions", so a reviewer can attach a score to one span inside one run. Opik is close but not identical — its queues hold "traces or threads", one or the other per queue, and its documentation notes that spans are not included in the trace view. So Langfuse is the only one of the four where the unit of review goes all the way down to a single step.
Argilla is in maintenance mode, and has been for 18 months
This is the fact most likely to change a decision, and it is stated in the first lines of the project's own README:
The original authors have moved on to exciting new projects! The codebase is mature and stable, having served users reliably for years. While we won't be adding new features going forward, we're committed to solve bug fixes and publish patches as needed.
The git history dates it precisely. A harsher version of the notice landed on 16 May 2025 and was softened to the wording above on 8 July 2025. The last substantive, non-README commit on develop is 10 March 2025 — the same day 2.8.0 was published to PyPI, and the last release of any kind. The README goes on to ask for maintainers.
None of that makes Argilla a bad tool. A stable, Apache-2.0, well-documented annotation app that does what it says is a perfectly reasonable thing to deploy, and "mature and stable" is the maintainers' own honest framing. It does mean you should not expect it to grow a trace-shaped review surface, and that is the direction this whole field is moving.
Its agreement metrics are in the previous major version
Argilla is also the only one of the four that ever shipped real inter-annotator agreement — and it is in the legacy SDK, not the current one. The v1 package carries agreement_metrics.py ("This module contains metrics to gather information related to inter-Annotator agreement"), implemented as a wrapper over NLTK's AnnotationTask with binary, interval and MASI distances. Search the 2.8.0 SDK and the current docs for "agreement", "kappa" or "Krippendorff" and you get nothing.
What v2 offers instead is overlap control, which is not the same thing: rg.TaskDistribution(min_submitted=2) configures "the number of minimum submitted responses expected for each record". That guarantees two people looked. It does not tell you whether they said the same thing.
Label Studio has the best agent-review surface, behind a paywall
Label Studio 1.23.2 is actively released and its community edition is plain Apache-2.0 — the LICENSE file is the unmodified Apache text with the copyright line filled in, and there are no additional use restrictions. The gating here is by edition, not by licence, and it lands on exactly the features this article is about.
The community templates are genuinely useful for LLM work: "LLM Response Grading", "Side-by-Side LLM Output Comparison", "Multi-Turn Chat Evaluation", "Evaluate RAG with Human Feedback", "Human Preferences collection for RLHF". Then there is the one built for agents, marked with a lock:
This template creates an agent run review Interface for evaluating multi-step agent traces — reasoning, tool calls, subagents, artifacts, and errors.
Reviewers using it "rate individual steps as correct, partial, or incorrect, assign an overall verdict (good / mixed / bad), score the run against a configurable rubric, tag failure modes, and write a free-text critique". That is the best description of trace review in any of these four products. Its docs also say "Interfaces can only be used in Label Studio Enterprise and Starter Cloud".
The review workflow itself is paid, and so is agreement
Two further gates matter. The annotation review workflow — a second pass over annotators' work — is documented as "only available in Label Studio Enterprise Edition". And the edition matrix is explicit about agreement across three tiers: "Agreement metrics" reads ❌ for Community, "Limited" for Starter Cloud, ✅ for Enterprise; "Custom agreement metrics" and "Annotator consensus matrices" are Enterprise only. Enterprise also ships automatic reassignment of tasks with low agreement scores.
One honest caveat on what that buys you. Label Studio's built-in agreement metrics are matching functions — Exact Match, Common Labels Matches, Jaccard Similarity, Numeric Difference, IoU, bounding-box variants — each documented as "Pairwise" or "Consensus" methodology. Neither Cohen's kappa nor Krippendorff's alpha is named among them. These are raw-agreement measures, which do not correct for the agreement you would get by chance. For a rubric with three options and a dominant "good" class, raw agreement of 80% can be close to worthless, and that is precisely the case kappa exists to catch.
Langfuse: queues are free, and the agreement it computes is the wrong pair
Langfuse's licensing is worth getting right because people assume review is the paid part. The repository is MIT except for content under ee/, web/src/ee/, worker/src/ee/ and packages/shared/src/server/ee/. The annotation-queue implementation is not in any of those paths — it lives at web/src/features/annotation-queues/ with its API definition under fern/apis/server/. The pricing table agrees: manual annotation of traces is available on every tier, and queues are "1 queue" on the free Hobby tier, three on Core, unlimited on Pro, and true for self-hosted open source. (The LICENSE copyright holder is now ClickHouse, Inc., which is its own piece of news for anyone tracking vendor risk.)
So Langfuse gives you the deepest unit of review, free, self-hostable. What it does not give you is annotator-versus-annotator agreement. Its Score Analytics feature does compute agreement — "Correlation metrics (Pearson, Spearman)" and "Agreement metrics (Cohen's Kappa, F1, Overall Agreement for categorical/boolean)", with documented kappa interpretation bands — but it compares score against score. The worked example in the docs is "Human vs AI Annotation Agreement": one annotation score against one judge score. It is the right tool for calibrating a judge and the wrong one for validating the humans, because two reviewers' labels on the same item are not two different score names. The feature is also marked Beta and limited to two scores at a time.
That distinction is easy to lose and expensive to lose. Judge-versus-human agreement tells you the judge tracks the human. Human-versus-human agreement tells you whether the human was tracking anything.
Opik: overlap enforced, agreement unmeasured
Opik 2.2.146 was published the same day as the Langfuse release read for this article, and it is Apache-2.0 throughout. Its annotation queues are the newest of the four and the ergonomics are good: "An annotation queue gives them a list of traces or threads to label, in a view that shows each item the way they judge it." Items can be added by hand from the Logs page or automatically when feedback scores meet a condition, which is the closest thing in this set to a triage rule.
Crucially it ships the knob the argument needs — "annotators per item", described as "how many people must score an item before it is done", with lock slots so multiple reviewers can work the same item concurrently. The docs even name the use case, recommending 2 or more for expert agreement when aligning a judge. And then they stop. Nowhere does Opik define an agreement metric or say how one would be computed. You are told to collect the overlapping labels; what you do with them is your problem.
That is the same shape as Argilla's TaskDistribution and it is the pattern across the whole category: three of these four tools let you require that N people review an item, and only the paid tier of the fourth will tell you whether the N agreed.
Why this is the load-bearing gap
The reason to care is that human labels are not the ground truth. They are a measurement of the ground truth, taken with an instrument whose precision nobody reported.
The canonical justification for LLM-as-judge makes this dependency explicit. The MT-Bench authors' own repository states the result as: "Our results show that humans and GPT-4 judge achieve over 80% agreement, the same level of agreement between humans." Read that sentence as a claim about judges and it is reassuring. Read it as a claim about the denominator and it says something sharper: the judge is declared good because it matches humans about as well as humans match each other. That argument is only available to someone who measured human-versus-human agreement. If you have not measured yours, you do not know what level your judge needs to reach, and "the judge agrees with my labeller 85% of the time" is uninterpretable — it could be excellent or it could be two people guessing in the same direction.
The practical consequences are three, and they compound:
- Your eval set inherits the noise. Labels collected without an agreement check go into the golden set permanently, and every future model comparison is run against them. This is the maintenance problem in eval set maintenance, arriving before the set is even built.
- Your judge is calibrated to one person's taste. With a single annotator per item there is no way to separate "the rubric is unclear" from "this reviewer is strict", which is the distinction judge calibration and meta-evaluation exists to make.
- You cannot tell a rubric problem from a reviewer problem. Low agreement is diagnostic: it usually means the rubric has an undefined case, not that your reviewers are careless. Without the number you will keep retraining people instead of fixing the question.
None of which requires buying anything. Agreement on a categorical rubric is a few lines of scikit-learn or NLTK over labels you can already export from all four tools. The gap is not that it is hard; it is that no tool in this category prompts you to do it, so it does not get done.
When to pick which
| Situation | Pick | Why |
|---|---|---|
| Reviewing agent runs, self-hosted, no budget | Langfuse | Only free tier where the unit of review reaches a single observation inside a trace. |
| Reviewing agent runs with a real review workflow and a budget | Label Studio Enterprise | The step-level trace Interface and agreement metrics are the best in the set — both paid. |
| You want triage rules that queue items automatically | Opik | Score-condition-based auto-enqueue plus an explicit annotators-per-item setting, Apache-2.0. |
| Building a labelled dataset for fine-tuning, not reviewing runs | Argilla | Record-shaped review is the right shape for this, and the project is stable — accept that it is feature-frozen. |
| You need defensible agreement numbers | Any of them, plus your own script | Export the overlapping labels and compute kappa or Krippendorff's alpha yourself. Only Label Studio Enterprise does it in-product, and it reports raw agreement rather than chance-corrected. |
FAQ
Is Argilla abandoned?
Not abandoned, but feature-frozen and looking for maintainers. The README says the original authors have moved on and that no new features are coming, while committing to bug fixes and patches. The last release is 2.8.0 from 10 March 2025 and the last substantive commit is the same day.
Are Langfuse annotation queues a paid feature?
No. The implementation sits outside every ee/ path in the repository, so it is covered by the MIT core, and the pricing table lists queues as available to self-hosted open source and to the free Hobby cloud tier, capped at one queue there.
Can Label Studio review agent traces in the community edition?
Not with the purpose-built surface. The agent-run review Interface and the Multi-Turn Evaluation and Chatbot Evaluation templates are marked Enterprise or Starter Cloud. Community users get "Multi-Turn Chat Evaluation" and the conversational-AI templates, which handle dialogue but not a tool-call tree. The annotation review workflow is also Enterprise-only.
Which of these computes inter-annotator agreement?
Only Label Studio, and only on paid tiers — and its built-in metrics are raw matching functions (Exact Match, Jaccard, IoU and similar) rather than chance-corrected statistics; neither Cohen's kappa nor Krippendorff's alpha appears in its built-in list. Argilla had NLTK-backed agreement metrics in its legacy v1 SDK only. Langfuse computes agreement between two scores, not between two annotators. Opik enforces reviewer overlap but defines no metric.
How many reviewers per item do I need?
Enough on a sample to estimate agreement, not on everything. Both Opik and Argilla let you require two or more submissions per item; the cheap pattern is single review for throughput plus a double-reviewed sample held back for the agreement measurement, which keeps the cost discussion in the cost of human review honest.
Further reading
On this wiki:
- Annotation & labeling ops — running the review function, not just picking its tool.
- Judge calibration & meta-evaluation — what the agreement number is for.
- Trajectory & process evaluation — why the unit of review has to reach a single step.
- Retrospective trace review — the workflow these queues are meant to serve.
- PII redaction in agent traces — what you must do before a human opens the trace.
- Oversight capacity — the headcount ceiling that caps every queue above.