构建一个包含 50 道题的数据集。

评估中最难的部分不是测试框架，而是数据集（dataset）。糟糕的数据集让你对一个糟糕的智能体盲目自信。优质的数据集能暴露真实的失败模式。为此花一天时间，别外包出去，也别完全靠 LLM 生成然后照单全收。

四个难度层

将 50 道题按四个难度层分层，让评估能区分不同能力，而非仅仅看平均分。

• 第一层（15 道题）：直查。单一事实，来自单篇文档。"Postgres 默认使用哪个端口？"若智能体答错这类题，说明检索（retrieval）已出问题。
• 第二层（15 道题）：多跳推理。需要综合 2–3 篇文档。"PgBouncer 连接池与预处理语句如何相互影响？"
• 第三层（10 道题）：比较分析。需权衡不同选项的取舍。"何时应使用分区而非分片。"
• 第四层（10 道题）：否定案例。语料库中没有答案的问题。"如何配置 PgBouncer 与 Redis？"智能体应回答"语料库中没有相关信息"，而非编造答案。

模式（schema）

与提供商无关，纯 JSON。

# evals/dataset.jsonl — one question per line
{
  "id": "q001",
  "tier": "lookup",
  "question": "What port does PostgreSQL listen on by default?",
  "expected_answer_contains": ["5432"],
  "expected_chunks": ["runtime-config-connection::0"],
  "forbidden_phrases": ["I don't know", "not sure"],
  "notes": "basic fact, must succeed on every run"
}
{
  "id": "q024",
  "tier": "multi_hop",
  "question": "Why do prepared statements break in PgBouncer transaction mode?",
  "expected_answer_contains": ["session", "transaction"],
  "expected_chunks": ["pgbouncer-modes::1", "sql-prepare::0"],
  "requires_synthesis": true
}
{
  "id": "q045",
  "tier": "negative",
  "question": "How does PostgreSQL integrate with Redis Streams?",
  "expected_behavior": "refuse",
  "expected_answer_contains": ["not in", "corpus"],
  "forbidden_phrases": ["you can use", "the integration"]
}

如何构建题目

按优先级排列，来源有三：

1. 第 1–3 阶段的追踪日志。智能体曾答错的每一道题，都是完美的评估题目。真实的失败案例胜过凭空捏造。
2. 浏览文档。打开语料库，找到你认为非平凡的章节，针对该章节的内容写一道自然语言问题。
3. LLM 生成，再经人工筛选。让模型从语料库中生成 100 道题；你从中保留 30 道真正有实际价值的，其余丢弃。这是 LLM 唯一能帮上忙的环节，且必须有人工把关。

单独评估各组件。

端到端评估噪声很大。检索失败、规划失败、综合失败、验证器失败会被混在一个数字里。要排查问题，就必须知道是哪个组件出了毛病。因此要分开评估各组件。

检索评估：recall@k 与 MRR

对数据集中每道有 expected_chunks 字段的题，运行检索并判断：前 k 个结果中是否包含预期的文档块？

# evals/eval_retrieval.py
import json
from retrieval import retrieve
from retrieval.bm25_index import BM25Index
from retrieval.dense_index import DenseIndex
from retrieval.hybrid import HybridSearch

def load_dataset():
    return [json.loads(line)
            for line in open("evals/dataset.jsonl")]

def eval_retrieval(retriever, dataset, k=5):
    hits, mrr_sum, n = 0, 0.0, 0
    for q in dataset:
        if not q.get("expected_chunks"): continue
        n += 1
        results = retriever(q["question"], top_k=k)
        result_ids = [c.chunk_id for c in results]
        # Recall: at least one expected chunk in top-k
        if any(eid in result_ids
               for eid in q["expected_chunks"]):
            hits += 1
        # MRR: 1 / rank of first expected chunk
        for rank, rid in enumerate(result_ids, 1):
            if rid in q["expected_chunks"]:
                mrr_sum += 1.0 / rank
                break
    return {"recall@k": hits / n, "mrr": mrr_sum / n, "n": n}

数据说明了什么

用三种不同的检索器运行同一个评估——纯 BM25、纯稠密向量、混合——再做比较：

              recall@5   recall@20    MRR    n
BM25 only       0.62       0.81      0.41   40
Dense only      0.68       0.85      0.47   40
Hybrid (RRF)    0.82       0.94      0.58   40
Hybrid+rerank   0.89       0.94      0.71   40

验证器评估：对声明的精确率与召回率

人工标注 30 组（声明、来源、判定）三元组——由你亲手完成，不要交给 LLM——然后运行验证器，检查一致性。

# evals/verifier_gold.jsonl — hand-labeled
{"claim": "Transaction pooling breaks prepared statements",
 "sources": ["pgbouncer-modes::1"],
 "human_verdict": "SUPPORT"}
{"claim": "This is the most common migration failure",
 "sources": ["pgbouncer-modes::1"],
 "human_verdict": "NOT_SUPPORTED"}

def eval_verifier(gold):
    correct = 0
    for ex in gold:
        result = verify(ex["claim"],
                        load_chunks(ex["sources"]))
        v = result["claims"][0]["verdict"]
        if v == ex["human_verdict"]:
            correct += 1
    return {"agreement": correct / len(gold)}

verifier agreement with humans: 0.87  (26 of 30)

inspecting disagreements:
  3 cases where verifier said PARTIAL, human said SUPPORT
    → verifier is slightly too strict on phrasing
  1 case where verifier said SUPPORT, human said PARTIAL
    → genuine bug; source says "X under condition Y"
      but verifier missed the condition

同样的工作，用 RAGAS 术语来说

我们刚刚构建的东西有一套标准名称，从业者现在期望你使用它们。RAGAS 是封装这些指标的常用框架（底层用 LLM 作为评判者，和我们的验证器与评判者是同一思路）：

• 忠实度（faithfulness）——答案是否扎根于检索到的上下文？这正是我们的落地验证器：每条结论都必须能追溯到引用的块。
• 答案（响应）相关性（answer/response relevancy）——答案是否真正回应了问题，而不只是停留在话题上？
• 上下文精确率（context precision）——我们检索到的块中，有多少是真正相关的（top-k 中的信号与噪声之比）？
• 上下文召回率（context recall）——检索是否找全了回答所需的一切？这就是我们的 recall@k，换了个名字。

用 LLM 作为评判者（LLM-as-judge）评估轨迹。

组件评估告诉你每个部件是否正常工作。轨迹评估告诉你整体智能体是否能成功完成任务——而这不只是"最终答案是否正确"。还包括：它的步骤数是否合理？是否引用了正确的来源？是否避免了被禁止的行为？

先做硬检查（廉价、确定性）

# evals/eval_trajectory.py
def hard_checks(q: dict, result: dict) -> dict:
    answer = result.get("answer", "").lower()
    checks = {}

    # Did the answer mention expected terms?
    expected = q.get("expected_answer_contains", [])
    checks["contains_expected"] = all(
        term.lower() in answer for term in expected
    )

    # Avoid forbidden phrases (especially for negatives)
    forbidden = q.get("forbidden_phrases", [])
    checks["no_forbidden"] = not any(
        p.lower() in answer for p in forbidden
    )

    # Citation overlap with expected chunks
    cited = set(result.get("citations", []))
    expected_chunks = set(q.get("expected_chunks", []))
    if expected_chunks:
        checks["citation_overlap"] = bool(cited & expected_chunks)

    # Step efficiency: didn't burn budget
    checks["under_budget"] = result.get("steps_used", 99) <= 8

    # For negative cases: agent must have refused
    if q.get("expected_behavior") == "refuse":
        checks["refused"] = (
            "not in" in answer or
            "corpus" in answer or
            "don't have" in answer
        )

    return checks

再用 LLM 评判者做软检查

对于"答案是否正确且格式良好"这类硬检查无法回答的问题，使用评判者模型。使用与智能体不同的模型来评判（跨模型评判可减少自我确认偏差）。如果智能体运行在 Claude 上，就用 GPT 来评判，反之亦然。

# evals/judge.py — judge with GPT
from openai import OpenAI
judge_client = OpenAI()

JUDGE_PROMPT = """Rate the agent's answer on three axes.

QUESTION: {question}
EXPECTED ANSWER NOTES: {expected_notes}
AGENT'S ANSWER: {answer}
AGENT'S CITATIONS: {citations}

Score each from 1 (terrible) to 5 (excellent):
- correctness: does it answer the question accurately?
- completeness: are key aspects covered?
- groundedness: are claims supported by citations?

Output JSON: {{"correctness": N, "completeness": N,
"groundedness": N, "reasoning": "one sentence"}}"""

def judge(q, result):
    response = judge_client.responses.create(
        model="gpt-5.5",
        input=JUDGE_PROMPT.format(
            question=q["question"],
            expected_notes=q.get("notes", ""),
            answer=result.get("answer", ""),
            citations=result.get("citations", []),
        ),
        text={"format": {"type": "json_object"}},
    )
    return json.loads(response.output_text)

# evals/judge.py — judge with Claude
from anthropic import Anthropic
judge_client = Anthropic()

JUDGE_PROMPT = """Rate the agent's answer on three axes.

QUESTION: {question}
EXPECTED ANSWER NOTES: {expected_notes}
AGENT'S ANSWER: {answer}
AGENT'S CITATIONS: {citations}

Score each from 1 (terrible) to 5 (excellent):
- correctness: does it answer the question accurately?
- completeness: are key aspects covered?
- groundedness: are claims supported by citations?

Output JSON only: {{"correctness": N, "completeness": N,
"groundedness": N, "reasoning": "one sentence"}}"""

def judge(q, result):
    response = judge_client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user",
                   "content": JUDGE_PROMPT.format(
                       question=q["question"],
                       expected_notes=q.get("notes", ""),
                       answer=result.get("answer", ""),
                       citations=result.get("citations", []),
                   )}],
    )
    return json.loads(response.content[0].text)

每道题运行评判者 3 次

LLM 评判者存在噪声。即使 temperature=0，同一道题的评分也会因运行不同而变化。为获得可靠的信号，每次判断运行 3 遍，取中位数。方差本身也是有价值的数据：如果评判者对同一个例子给出 [3, 5, 4] 的评分，说明该题的评估信号较弱。

def judge_with_variance(q, result, n=3):
    scores = [judge(q, result) for _ in range(n)]
    def median(key):
        return sorted(s[key] for s in scores)[n // 2]
    return {
        "correctness": median("correctness"),
        "completeness": median("completeness"),
        "groundedness": median("groundedness"),
        "variance": max(s["correctness"] for s in scores) -
                    min(s["correctness"] for s in scores),
    }

完整评估运行

$ python scripts/eval.py --full

evaluating 50 questions across 4 tiers...

Tier 1 (lookup):     15/15 hard pass    avg judge: 4.7 / 5.0
Tier 2 (multi-hop):  13/15 hard pass    avg judge: 4.2 / 5.0
Tier 3 (comparative): 8/10 hard pass    avg judge: 3.9 / 5.0
Tier 4 (negative):    9/10 hard pass    avg judge: 4.5 / 5.0

OVERALL hard-pass rate: 45/50 = 90.0%
OVERALL judge correctness (median): 4.3 / 5.0
OVERALL judge groundedness (median): 4.6 / 5.0

high-variance questions (judge disagrees with itself):
  q017: scores [3, 5, 4] — answer is partially correct,
        judge unsure which way to score
  q031: scores [2, 5, 3] — answer is verbose; judge
        differs on whether length helps or hurts

failing questions:
  q024: missed expected chunk pgbouncer-modes::1
        (retrieval ranked it at 7)
  q037: cited a chunk that doesn't support the claim
        (verifier should have caught this — bug)
  q045: invented Redis integration details
        (negative case failed)

将评估接入开发循环。

每周运行一次的评估套件是研究；每次提交都运行的评估套件才是工程。让它足够轻量以便你真的会去运行，足够快速以便在开发中使用，且有清晰的通过/失败信号。

两个层级：快速与完整

完整评估需要数分钟；每次代码改动后都等它不现实。拆分为快速层（10 道题，约 30 秒）用于迭代，完整层（50 道题，3–5 分钟）用于确认。

# Makefile
.PHONY: eval-fast eval-full eval-watch

eval-fast:
    python scripts/eval.py --tier fast --n 10

eval-full:
    python scripts/eval.py --full --judge --variance 3

eval-watch:
    # Re-run fast eval whenever agent/ changes
    fswatch -o agent/ retrieval/ | xargs -n1 -I{} make eval-fast

ci:
    make eval-full
    python scripts/eval.py --regression-check

随时间追踪分数

最有价值的单一产出物：一份记录每次评估运行的 CSV，包含日期、git commit 和各项分数。把它画出来。

# evals/scoreboard.csv — append after every run
date,commit,hard_pass,judge_correct,judge_grounded,steps_p50
2026-05-08,a1b2c3d,0.72,3.8,4.1,5.2
2026-05-09,e4f5g6h,0.78,4.0,4.3,4.8   # added reranker
2026-05-10,i7j8k9l,0.84,4.2,4.5,4.5   # tuned chunk size
2026-05-12,m0n1o2p,0.88,4.3,4.6,4.2   # added planner
2026-05-15,q3r4s5t,0.90,4.3,4.6,3.8   # subagent context isolation

现在你有了一个故事。"加入重排序器为硬通过率带来了 6 个百分点的提升。子智能体没有提升正确率，但将中位步骤数从 4.2 降到了 3.8。"这就是你写进 PR 描述和在评审会上呈现的内容。

回归检查

一个简单的门控：上次通过而这次失败的任何题目，都是回归。阻止合并。

def regression_check(current, baseline):
    regressed = []
    for qid, c in current.items():
        b = baseline.get(qid)
        if b and b["passed"] and not c["passed"]:
            regressed.append(qid)
    if regressed:
        print(f"REGRESSION: {regressed}")
        sys.exit(1)
    print("no regressions")