Evaluating AI-Generated Interfaces Beyond Snapshot Tests

Generating an interface is easy to demonstrate. Generating one that can survive production constraints is a different problem.

A screenshot can look convincing while the underlying result fails to compile, ignores the design system, breaks at another viewport, or exposes controls that cannot be used from a keyboard. Even a clean visual diff only tells us that pixels changed. It does not tell us whether the interface is correct.

That distinction matters when generative systems produce working software instead of static inspiration. The evaluation target is no longer a single image. It is a layered product artifact: code, semantics, interaction, visual language, and user outcome.

The most useful evaluation strategy therefore resembles a quality system more than a benchmark leaderboard.

Begin with executable truth

The first layer should be deliberately unglamorous. Can the output become software?

A generated interface should:

• parse and compile without manual repair;
• render without uncaught runtime errors;
• avoid unresolved imports and missing assets;
• satisfy the repository’s type and lint rules; and
• remain inside the requested technical boundaries.

These checks are deterministic, fast, and easy to explain. They should run before subjective scoring because there is little value in judging the visual quality of an artifact that cannot ship.

This layer also protects the rest of the evaluation pipeline. If invalid output reaches browser automation or a visual evaluator, the resulting failure is often noisy and misleading. A short-circuiting build gate produces a much clearer signal.

Evaluate semantics separately from appearance

Visual similarity is not semantic correctness.

A generated button might look right while being rendered as a generic div. A form might resemble the reference while lacking programmatic labels. A dialog might be pixel-perfect and still lose focus behind the overlay.

The semantic layer should inspect:

• landmark and heading structure;
• accessible names, roles, states, and values;
• label-to-control relationships;
• keyboard reachability and focus order;
• status and error communication; and
• the use of native elements before custom behavior.

The WCAG guidance for name, role, and value is a useful baseline because it focuses on what assistive technology must be able to determine. The broader WCAG keyboard criterion makes the interaction requirement equally explicit: functionality should not depend on a pointer.

Automated assertions can cover a meaningful part of this layer. For example, Playwright ARIA snapshots compare the accessible structure of a page rather than its DOM implementation. That makes them useful for checking the interface contract while allowing harmless markup changes.

They are not a substitute for assistive-technology testing, but they catch a different and often more important class of failure than pixels alone.

Treat visual comparison as one instrument

Visual regression still matters. It is particularly good at finding spacing drift, missing content, unexpected wrapping, incorrect colors, and layout changes that are difficult to express as individual assertions.

It also has limits. Playwright’s visual-comparison documentation notes that screenshots vary with operating system, browser version, hardware, fonts, and other environmental details. Stable baselines require a controlled rendering environment.

For generated interfaces, visual checks should answer focused questions:

• Is the layout structurally faithful to the intended composition?
• Are key regions present and visually ordered correctly?
• Does the interface remain coherent across required breakpoints?
• Are light and dark themes both valid?
• Did the generation introduce obvious clipping, overflow, or overlap?

A single full-page similarity score hides too much. Region-level comparisons and explicit layout assertions usually produce more actionable failures.

Make the design system testable

“Looks on brand” is too vague to operate.

If design-system compliance matters, translate it into observable constraints. Some can be hard gates:

• approved components are used for known interaction patterns;
• colors resolve through semantic tokens;
• spacing comes from the supported scale;
• focus indicators remain visible;
• typography follows the established hierarchy; and
• unsupported one-off values are rejected.

Other qualities need a rubric rather than a binary test. Hierarchy, density, balance, and responsiveness can be scored against defined examples with a short explanation attached to each result.

The important move is to evaluate the generated artifact against the system’s rules, not against a vague memory of its aesthetic. That turns the design system from documentation into an executable product constraint.

Test behavior as a sequence

An interface is a state machine wearing pixels.

The evaluation dataset should describe important journeys, not only initial render states. A useful case includes the starting state, the user action, the expected transition, and the observable outcome.

For a form, that may mean:

1. Submit an empty required field.
2. Confirm that focus moves to a useful location.
3. Confirm that the error is visible and programmatically associated.
4. Correct the value.
5. Confirm that the stale error clears.
6. Submit successfully without losing the entered state unexpectedly.

This style of test reveals whether the generated interface understands the product behavior, not merely the component inventory.

Include responsive changes, loading states, empty states, errors, and keyboard interaction in the same model. Happy-path rendering is necessary, but it is rarely where generated software becomes expensive.

Separate gates from scores

Not every result should be averaged.

A missing accessible name, an invalid build, or a broken primary action should fail the artifact even if every other category scores highly. These are gates: minimum conditions for continuing.

Qualities such as visual fidelity or compositional polish are better represented as scores with evidence. They can improve over time without allowing a critical failure to disappear inside an average.

A practical evaluation report might contain:

• pass or fail gates for build, runtime, accessibility, and core behavior;
• category scores for design-system alignment and visual quality;
• the exact artifact, model configuration, and evaluation version;
• failure evidence such as screenshots, traces, and assertion output; and
• a short reviewer rationale when human judgment is involved.

This creates a result that an engineer can debug and a product team can understand.

Design the dataset around failure modes

Random prompts are useful for exploration, but they are a weak regression suite.

Build cases around the ways the system is likely to fail:

• ambiguous requirements;
• incomplete source material;
• conflicting visual and semantic instructions;
• dense content;
• long or localized labels;
• narrow viewports;
• missing data;
• unsupported components; and
• attempts to bypass system constraints.

Keep a small, stable core set for release decisions and a broader rotating set for discovery. When a production issue appears, convert it into a durable case after removing sensitive data.

This aligns with the testing, evaluation, verification, and validation approach collected by the NIST AI Resource Center: evaluation is an operating practice, not a one-time score.

Keep humans at the judgment boundary

Human review is most valuable where the requirement itself contains judgment.

People should not spend their time confirming that code compiles or that every button has an accessible name. Automation can do that consistently. Reviewers should focus on whether the product hierarchy makes sense, whether the interpretation is responsible, and whether the result solves the intended problem.

The rubric matters here. “Would you ship this?” invites preference. “Does the primary task remain obvious at each viewport, and what evidence supports that conclusion?” invites a review that can be compared and improved.

Model-based evaluation can help scale those rubrics, but it should be calibrated against human decisions and should return reasons, not only numbers. A judge that cannot explain its disagreement is difficult to trust and difficult to improve.

Measure the evaluation system too

An evaluation can be stable and still be wrong.

Track the quality of the evaluator alongside the quality of the generator:

• Does a failure reproduce?
• Do automated and human judgments agree?
• Which checks create noisy false positives?
• Which production failures escaped?
• How long does a result take to diagnose?
• Does a benchmark improvement predict a better user outcome?

These questions prevent the evaluation suite from becoming ceremonial. A useful system should make regressions visible, improvements credible, and failures easier to act on.

The goal is confidence, not a perfect score

AI-generated interfaces combine probabilistic creation with deterministic product expectations. Testing only the screenshot evaluates the least reliable layer as if it were the whole product.

A stronger approach moves from executable validity to semantics, visual structure, system compliance, behavior, and finally human judgment. Each layer answers a different question, and each produces evidence appropriate to that question.

The result is not certainty. It is something more useful to an engineering organization: a repeatable basis for deciding what is ready, what failed, and what should improve next.