Design-to-Code Agents

The job is translation into a vocabulary, not transcription of pixels.

Every design carries two layers. There is the visual result, which a screenshot captures completely, and there is the system that produced it — the token that supplied the colour, the component instance the designer dragged in, the spacing step chosen from a scale of six. Only the first survives being flattened into an image, and the second is the one your codebase runs on.

An agent given only pixels does something closer to OCR for layout, and its three characteristic outputs all pass a visual check:

• A component that already exists, rebuilt. The new one has the same appearance and none of the behaviour: no focus ring, no disabled state, no loading spinner, no telemetry, no story. It looks correct until someone changes the real Button and eleven screens do not follow.
• Literals where tokens belong. #d4421e instead of var(--accent), 14px instead of the 16px step. This is invisible in review and fatal to theming — a dark mode that was one variable swap becomes a search-and-replace across a hundred files.
• Plausible values off the scale. Nothing in the image says the spacing scale has six steps, so the agent measures 14 pixels and writes 14 pixels. Every off-scale value is a small permanent debt that no test will ever flag.

None of this is a model capability problem, and a better model does not fix it. It is a context problem: the information required to make the right choice was destroyed before the agent was asked. That is context engineering, and it is the whole job here.

Hand over the inventory, or the agent will invent one.

The agent needs the vocabulary before it needs the picture. Assembling that is ordinary retrieval work against artefacts most codebases already have.

• Tokens, by name and value. A machine-readable list — colour, spacing, radius, typography — so the agent can map an observed value to the nearest legal one and, crucially, notice when there isn't one. "This colour is not in the palette" is a finding, not a licence to invent a hex.
• The component API, not the component list. Names alone produce confident misuse. Props, variants, required children and one sentence on when each component is the right choice. If your components are typed, extract this from the types rather than maintaining prose that drifts — the same argument tool schemas and contracts makes about tools applies to components, because to the agent a component is a tool.
• Canonical usage examples. Three to five real compositions per common component do more than any amount of description, for the same reason few-shot examples beat instructions elsewhere.
• The negative list. Deprecated components, the legacy CSS file nobody should touch, the two utilities that exist only for one old page. Agents are excellent at finding and imitating exactly the code you wish would die.

Do not paste the whole design system into the prompt. A mature system has hundreds of components and the relevant set for one screen is around ten; retrieving those ten is the same problem as repo navigation, with the same failure mode — confident wrong localisation costs more than a miss, because a component that is nearly right gets used.

When all you have is an image, extract structure as its own reviewed step.

Plenty of real work arrives as a PNG in a ticket. The instinct is to hand it to the agent and ask for a component; the better shape is to make structure extraction a separate pass with a human gate, for the same reason a plan is a cheaper gate than a diff.

• First pass: a layout tree, not code. Named regions, nesting, repetition ("this is a list of three identical cards"), and a proposed mapping from each region to a component in the inventory. This is small, readable in two minutes, and it is where the expensive mistakes are visible.
• Review the mapping, not the markup. A designer or engineer scanning "header → PageHeader, filter row → FilterBar, cards → ResultCard" catches the invented component instantly. The same person scanning 300 lines of JSX will not.
• Second pass: code, constrained to the approved mapping. With the mapping fixed, generation becomes filling in props, and the failure modes shrink to ones tests can catch.
• Record what could not be mapped. Anything the agent could not resolve to an existing component is either a genuine gap in the design system or a designer improvising off-system. Both are worth knowing; neither should be resolved silently by generating a new component.

Visual diffing is the wrong oracle, and it rewards exactly the wrong behaviour.

Screenshot comparison is the obvious test and it is worse than useless as the primary signal, because the failure this playbook is about — a one-off component that looks perfect — scores a perfect result. Keep visual regression as a guard against unintended change, and judge the agent on three other numbers.

• Reuse rate. Of the elements rendered, what share are existing components versus newly created ones? This is countable from the AST, it is the single number that predicts whether this programme helps or harms you, and it should be reported per pull request. Set a floor and fail below it.
• Token adherence. Of the colour, spacing, radius and type values in the diff, what share resolve to tokens rather than literals? A lint rule usually gets you most of this for free, and unlike reuse rate it can be enforced mechanically at commit time. Make it a hard gate — it is the cheapest of the three and the one that protects theming.
• State coverage. Designs show the happy path. Production shows empty, loading, error, one item, four hundred items, a name in a language with no spaces, a user who tabs instead of clicking. The agent will not invent these because they are not in the picture, so the acceptance criterion has to name them explicitly, and a test-generation pass is a reasonable way to produce the cases.

Grade the pull request against these before a human reads it, exactly as patch generation and test-driven loops prescribes: the agent should be told it failed the reuse floor and asked to try again, rather than a reviewer discovering it and rewriting by hand.

Accessibility and interaction are, literally, not in the picture.

A mock contains no focus order, no roles, no accessible names, no keyboard behaviour, no announcement of state changes and no motion. An agent asked to reproduce appearance will produce div soup that looks right, and no visual test will ever notice.

• Make semantic components the only path. This is the strongest argument for the inventory-first approach in step 2: if the agent composes from real components, the accessibility work is inherited rather than regenerated. Every hand-rolled element is an accessibility regression waiting to be found by a user.
• Run the automated checks inside the loop, as a hard gate. Axe-style rules catch missing labels, contrast failures and unlabelled controls cheaply and deterministically. Put them in the agent's verify step so a failure is a retry, not a review comment.
• Require interaction spec as a separate input. What is focusable, what happens on submit, what is announced when the list updates, what the escape key does. The designer or engineer supplies this in a sentence each; the agent cannot infer it and will confidently guess.
• Keep a human on the parts machines cannot check. Automated tooling catches a minority of real accessibility defects. Reading order that is visually fine and semantically scrambled is the classic generated-markup failure and it needs a person with a screen reader, not a rule.

Where this pays, and the case where it makes things measurably worse.

The economics are unusually legible here, because the same property decides both.

• Pays: assembling new screens from a mature system. A large inventory, a designer working inside it, and a stream of screens that are recombinations of solved parts. Reuse rate is naturally high, the agent's job is composition, and the time saved is real.
• Pays: the first eighty per cent of a conformant mock. Structure, imports, props and layout as a starting pull request that a human finishes. Treat the output as a draft with a named owner, per background coding agents — the throughput gain is only real if someone merges it.
• Pays: high-volume, low-stakes surfaces. Marketing pages, internal tools, admin screens — places where the cost of a slightly non-canonical component is genuinely low and the volume is high.
• Does not pay: greenfield with no system. With nothing to reuse, every screen invents its own vocabulary, and you end the quarter with forty near-identical buttons. The agent did not cause that; it just did it forty times faster than a human would have.
• Does not pay: bespoke visual work. A campaign page whose entire value is that it looks unlike your product has a reuse rate of zero by definition, and the metric that governs this playbook stops applying.
• Does not pay: anything whose value is the interaction. A drag-and-drop editor, a canvas, a complex form with conditional logic. The picture contains almost none of the specification, so you are not doing design-to-code, you are doing dictation.

Related: generative UI patterns for the opposite case, where the agent assembles the screen at runtime instead of at build time; code review agents for enforcing the reuse and token gates as review rather than as CI; and evaluating coding agents for building the private eval set that tells you whether any of this is improving.