Notebook & data-science agents.
In a notebook the file on disk is not the program that produced the answer — the kernel is, and nothing writes it down. That single fact breaks the contract every other coding agent runs on: the agent reports a cell that executed cleanly, the reviewer sees the output, and neither of them can tell that it passed because of a variable defined in a cell that was edited an hour ago and no longer exists. Build this agent so that "it ran" is decided by a restart-and-run-all executed by your harness, and everything else in this page follows from that one gate.
The transcript is not the program.
Every other coding agent operates on a filesystem where the artefact and the thing that runs are the same object: the diff you review is the code that executes. A notebook breaks that identity in three ways at once, and they compound.
- Hidden state. The kernel namespace holds objects that no cell currently creates. Delete the cell that defined
dfand every downstream cell keeps working until the kernel restarts — which, on the reviewer's machine, is the first thing that happens. - Out-of-order execution. Cells carry an
execution_count, and the sequence[1, 2, 7, 3]is legal, common, and invisible in a rendered diff. The stored outputs are a record of an execution order that the document no longer describes. - Outputs are committed data. A
.ipynbstores results next to source, so a stale number can survive a source edit indefinitely and looks exactly like a fresh one.
The failure this produces is specific and it is not a crash. The analysis is correct on the machine that produced it and wrong everywhere else, so it fails at the point where somebody acts on it rather than at the point where it is written. Treat a notebook the agent has been editing as a scratchpad whose claims are unverified until re-executed from a cold kernel — the same posture as reproducibility and nondeterminism takes toward a failing run you cannot re-run.
Make restart-and-run-all the only definition of done.
The agent must not be the thing that decides it finished. Give the harness one gate: a fresh kernel executes every cell top to bottom, and the run either completes or the task is not done. This is cheap to build, it is the single highest-value component in the system, and it converts an unfalsifiable claim into a pass or fail.
- Run it in a clean process, not the agent's kernel. If the verification shares a namespace with the exploration, it verifies nothing. Use a separate execution — headless notebook execution is a solved problem in every major runtime.
- Diff the outputs, not just the exit code. A notebook that runs clean but now produces a different number than the one in the write-up is the failure you are actually hunting. Compare the re-executed outputs against the committed ones and surface every numeric divergence.
- Pin the inputs before you blame the code. A re-run that hits a live table will diverge for legitimate reasons. Snapshot the query results, or run against a fixed as-of timestamp, so a divergence means a bug rather than a Tuesday.
- Give the agent the gate as a tool. Let it call the restart-and-run-all itself, mid-task, as many times as it likes. An agent that can check its own work converges; one that finds out at submission time flails. This is the generator–verifier gap in its most favourable form — verification here is genuinely cheap.
Feed the agent the namespace, not the output.
The instinct is to put the cell's output into the transcript, because that is what the human looks at. It is the wrong thing on both axes: a rendered dataframe is enormous and tells the model almost nothing it can act on, while the facts it needs to write the next line — column names, dtypes, null rates, cardinality, the actual range of a key — are compact and almost never present.
- Return a schema card, not a table. For every dataframe the agent touches: shape, column names with dtypes, null count per column, distinct count for low-cardinality columns, and min/max for numerics and dates. That is a few hundred tokens and it prevents most of the errors a printed head of five rows invites.
- Cap output by construction, at the tool boundary. Truncate to a hard token ceiling in the execution tool rather than instructing the model not to print things. One
df.head(50)on a wide frame poisons every subsequent step, for the reasons in agent cost control. - Expose the live namespace as a tool. A
describe(name)call the agent can make on demand beats dumping state into context speculatively — the classic pull-over-push move from context engineering. - Surface the execution order. Put the current
execution_countsequence and the set of names defined-but-not-currently-produced into the agent's view. The model cannot reason about stale state it cannot see.
The dangerous boundary is the warehouse, not the filesystem.
Sandboxing guidance for coding agents is written around the filesystem and the network, and it transfers here unchanged — see sandboxing and execution. But a data-science agent's real reach is the credential in the connection string, and a container boundary does nothing about it. The agent is inside your sandbox and inside your warehouse at the same time.
- Issue a read-only role, scoped to the tables the task names. Not the analyst's role. The most common incident in this category is not a malicious query, it is an agent that wrote a table into the shared schema because that was the obvious way to cache an intermediate.
- Give it a scratch schema and make it the default write target. Agents will materialise intermediates; the choice is whether they do it somewhere you can drop.
- Bound the query, not just the session. Statement timeout, byte-scanned limit, and row limit by default. An unbounded scan on a columnar warehouse is a bill, not an error, and it is exactly the shape of mistake a model makes when it does not know a table's size.
- Sample first as a workflow, not a rule. Have the agent develop against a sampled extract and re-run against the full table once at the end. This cuts cost and iteration latency, and it makes the expensive query a deliberate step rather than an accident repeated forty times.
Read-only is a smaller promise than it sounds. A read-only agent can still exfiltrate — it prints results, and those results land in a transcript and often in a shared notebook. Scope by table, not just by verb, and read blast radius in terms of what the credential can reach rather than what the sandbox can write.
Grade the number, not the code.
Coding-agent evaluation asks whether the tests pass, and it works because the tests are the specification — see evaluating coding agents. Here there are no tests, the deliverable is a claim about the world, and clean code that computes the wrong quantity is the dominant failure. Nobody's harness catches it, because it runs perfectly.
- Build a set of questions with known answers. Thirty real analyses whose numbers you already trust, re-asked from scratch. It is more work than a code benchmark and it is the only thing that measures what you actually deployed.
- Grade on the number and the method separately. A right answer from a wrong method is a coin flip that landed well, and it will not land well on the next question. This is the trajectory versus outcome distinction, and in analysis the trajectory carries most of the signal.
- Score abstention above confident error. "The join key is ambiguous between these two tables; which did you mean?" is a better outcome than a number, and your rubric has to say so explicitly or the model will always produce the number. See uncertainty and calibration.
- Track the semantic errors as their own taxonomy. Wrong grain, silent row loss on an inner join, a filter that drops nulls, a date boundary off by a timezone, a denominator that changed definition last quarter. These recur, they are gradeable, and they are invisible to any check that only asks whether the code ran — failure taxonomy and triage is the machinery.
Ship the analysis out of the notebook.
The notebook is where the agent thinks. It is a poor place for the result to live, and the transition is the step teams skip — which is how a one-off exploration becomes a number three dashboards depend on, still carried by a document whose reproducibility guarantee is a habit.
- Make extraction an explicit task, not a nice-to-have. When an analysis is going to be repeated, have the agent produce a parameterised module plus a thin calling notebook. The module is reviewable and testable in the ordinary way; the notebook becomes presentation.
- Require a data lineage line, generated not remembered. Every published number names its query, its as-of timestamp, and the commit that produced it. An agent can emit this perfectly and a human never does.
- Strip outputs at the commit boundary. Committed outputs are how stale numbers survive; the re-execution gate from Step 2 is what regenerates them. Keep one rendered copy for humans, and keep it out of the path that anything reads programmatically.
- Re-verify on a schedule, not on request. Anything promoted to recurring gets the restart-and-run-all on a timer, so upstream schema drift surfaces as a red job rather than as a wrong number in a meeting — the same argument as third-party tool drift, one layer down.
Build the restart-and-run-all gate first, before the agent, before the tools, before the prompt — hand it to the agent as a callable tool and refuse to accept any result that has not passed it from a cold kernel with pinned inputs. Then do two things that cost an afternoon each: replace printed dataframes with schema cards at the execution-tool boundary, and issue the agent a read-only role scoped to named tables with a byte-scanned cap. Everything else on this page is refinement; those three are what separate a notebook agent you can act on from one that is confidently wrong on somebody else's machine. Related: data & analytics agents for the domain framing, and code as action for why the cell is the tool call.