Repo Navigation & Code Context

8 min read

U2
Playbook · Coding & Computer-Use Agents

On a large repo the agent's hard problem is not generation — it is finding the twelve lines that matter.

The code-retrieval question got settled in 2026, and not in favour of the thing most teams built: Cursor removed the semantic index it had shipped for years and replaced it with local grep, and Claude Code never had one. The reason is not that embeddings stopped working — it is that how a search result enters the context turned out to matter more than which retriever produced it. Build your agent's navigation around cheap, exact, inline search it can iterate on, and spend your context budget on reading, not indexing.

Step 1The frontier harnesses converged on search without an index.

This is worth stating as fact rather than preference, because it reverses the architecture a lot of internal coding agents were built on.

  • Claude Code has no codebase index. Its navigation primitives are glob, regex content search and file reads, executed on demand against the working tree. There is no embedding store and no sync job, so the agent always sees the file as it is on disk right now.
  • Cursor removed its semantic index during 2026. The indexing settings — including resync and delete-index — came out of the UI, and retrieval moved to a locally built, locally queried exact-match index that Cursor markets as instant grep. Cursor's stated reason, posted to its own community forum, is that models got good enough at using grep and indexed search that the dedicated semantic path had stopped helping in a meaningful way. Because that index is keyed to a git commit, a non-git directory falls back to plain ripgrep.

Two caveats so you don't over-read this. Which regex binary a given release shells out to has changed more than once and is not a durable fact worth designing around. And at least one comparison claims Cursor still computes server-side encrypted embeddings for purposes other than search, which is contested; the durable, repeatedly stated claim is narrower — search is grep-shaped and local.

What follows describes the state as of November 2026. The direction of travel has reversed once already, in the pro-index direction, and it can reverse again — so the point of this essay is the mechanism, not the scoreboard.

Step 2The delivery channel decided it, not the retriever.

The most useful published result here is also the least intuitive. Is Grep All You Need? How Agent Harnesses Reshape Agentic Search (arXiv:2605.15184, May 2026) held the retrievers fixed and varied how their output reached the model. With search results returned inline in the tool response, grep beat vector retrieval on all ten harness-and-model pairs tested. With the same results written to files the agent then had to open, vector retrieval won five of ten.

Read that as a statement about the loop rather than about similarity math. An inline result is something the model can judge and immediately refine — a bad regex costs one cheap turn and a better regex. A result delivered as a file is a read the model must pay for before it can tell whether the retrieval was any good, which rewards a retriever that is right on the first attempt and punishes iteration. Exact search is the better partner for iteration because its failures are legible: zero hits means the symbol is spelled differently, and four hundred hits means the term is too generic. Embedding search always returns its k nearest neighbours, confidently, whether or not any of them is the answer.

# the loop that makes exact search win
hits = grep("refresh_token")        # 0 hits -> the name is wrong, cost: 1 turn
hits = grep("refreshToken")         # 41 hits -> too broad, still cheap
hits = grep("refreshToken.*expir")   # 3 hits -> now read those

# the loop that makes similarity search expensive
docs = vector_search("token refresh expiry", k=8)
# 8 plausible neighbours, no signal about whether any is right,
# and you must spend 8 reads to find out.

So the design question for your harness is not which index to build. It is whether your search tool returns enough of each hit — path, line, and a few lines of surrounding context — for the model to decide what to open next without opening anything. That one choice is worth more than the retriever swap.

Step 3What the index still buys, and why the gain evaporated.

The semantic gain was real and measured. Cursor's own earlier research reported roughly a 12.5% average accuracy improvement from semantic search over grep alone — which is exactly why the removal is interesting. The gain did not get disproven; it got eaten, by models that learned to drive search tools well enough to reach the same answers with regex and more turns.

That leaves a residual where an index still earns its keep, and it is narrower than it looks:

  • The query where the reader cannot name the thing. "Where do we handle retry backoff" has no shared identifier if the code calls it jitteredDelay. Exact search cannot find a name nobody wrote.
  • Undocumented legacy and poor naming, where structure has stopped telling the truth — metaprogramming, dynamic dispatch, generated code, a monorepo where Handler appears in nine unrelated services.
  • Latency against a very large tree, which is a product concern rather than a recall one. This is the quiet reason index-based products kept their index: a human waiting on autocomplete has a budget an agent doing a four-minute task does not.

Note what all three have in common: none of them is "the agent cannot find the code." They are cases where the cheap path needs more turns than you want to pay for. Treat an index as a latency and naming optimisation with a sync job attached, priced accordingly — the same hybrid logic as hybrid search and reranking, with the balance much further toward lexical than it sits for prose. If you run one, the invalidation problem is the real cost: index freshness and invalidation covers why a stale code index is worse than none, because it returns a line number that no longer exists with full confidence.

Step 4Index symbols and the graph, not just text.

A flat text index cannot answer "find references." A symbol index built from a parser or LSP/ctags/tree-sitter gives the agent go-to-definition and find-references as primitive tools — the same affordances a human engineer uses to move through code. This converts navigation from "read files and hope" into graph traversal: from the issue's surface symbol, hop to its definition, then to its callers, narrowing the candidate edit set with each hop instead of flooding context with whole files.

Structural search is the complement to STEP 2's argument rather than an exception to it, because it has the same property that made grep win: its answers are exact and its failures are legible. One published study of a structural codebase index found that it did not duplicate what competent agentic grep already reaches — and, importantly, did not regress the agent either. That is the honest summary of the structural tier as of late 2026: a real affordance on statically analysable code, not a step change.

# navigation as graph traversal, not file dumping
hits  = repo.grep(symbol)                 # lexical: exact, cheap, verifiable
defn  = repo.goto_def(hits[0])           # structural: one true site
calls = repo.refs(defn)                  # who depends on this?
ctx   = budget.select(defn, calls, k=12)  # keep ~12 spans, not 12 files

Prefer windowed views over whole files. An open(path, line, ±40) viewport keeps the agent on the relevant span and leaves budget for the call graph; dumping a 1500-line module spends the window on noise the agent must then re-skim every turn.

Step 5Searching is cheap; reading is the bill.

Once search is free the binding constraint moves, and most teams aim their optimisation at the wrong half. A regex sweep over a large repository costs a few hundred tokens of output. Opening the six files it pointed at costs tens of thousands, permanently, in every subsequent turn of the run — and a window that is nominally large degrades well before it is full, which is the subject of effective versus advertised context.

Treat the window as a fixed budget split across four competing claimants: the task (issue + repro), the code under edit (the spans you will change), the evidence (callers, tests, traces that justify the change), and the scratchpad (the agent's own reasoning and history). Every token spent on an irrelevant file is a token stolen from evidence. The discipline is ruthless eviction: a file read to test a hypothesis that was refuted should be summarized to one line and dropped, not carried forever.

The structural fix that actually works is to put the expensive reads somewhere else. Claude Code's pattern is a read-only exploration subagent on a smaller model with its own context window: it does the searching and reading, and returns a summary, so the main loop pays for the conclusion rather than the transcript. The trade-offs — what the delegate can and cannot see, and when the summary loses the detail you needed — are in what a subagent inherits and shaping tool results. When the main window fills anyway, context compaction is the fallback, and it is lossy by construction: pin the repo map and the task statement so compaction never eats them.

one navigation turn, order of magnitude:
  grep across 40k files, 12 hits with context     ~   700 tokens
  read the 3 files those hits sit in              ~ 24,000 tokens
  read 40-line windows around the 12 hits instead ~ 3,100 tokens

the search was never the expense.

Step 6The failure mode is a confident wrong localization, not a missed file.

A symbol like handler or config matches hundreds of irrelevant sites; semantic retrieval returns plausible-but-wrong neighbours. Unfiltered, this is context poisoning — the model anchors on a confidently-retrieved wrong file and edits it. Mitigations: rank by structural proximity to the failing stack frame, not just match count; prefer the file the traceback names; and make the agent state its localization hypothesis and confirm it by reproducing the bug before editing, so a bad retrieval is caught by the oracle rather than by the diff. That handoff into patch generation and tests is where navigation quality actually gets measured.

The most expensive failure in U2 is a confident wrong localization: the retrieved file looks right, the patch is clean, the targeted test even passes — and an untested call path elsewhere now breaks. High retrieval recall without structural grounding manufactures exactly this, and it is invisible to any eval that only checks whether the named test went green.

Two cheap habits reduce it more than a retriever change. First, a repo map — a compact ranked outline of top packages, key modules and their public symbols, the technique Aider popularised — gives the agent a persistent skeleton for a few hundred tokens, so navigation starts from "the auth code lives roughly here" instead of from zero. Second, force the agent to cite path and line for every claim about the code, which converts an unfalsifiable assertion into something a reviewer can check in one click.

If you are building this today: make search results inline and rich (path, line, surrounding lines) before you build any index — that is the change the May 2026 harness result says is largest. Add a symbol tier via LSP or tree-sitter for go-to-definition and find-references, because its answers are exact. Give exploration its own subagent and context window so reads do not accumulate in the main loop. Keep the repo map pinned across compaction. And add a semantic index only after you can point at a logged class of queries that exact and structural search failed on — by name, with counts — because its real price is not the embeddings, it is the invalidation.