Validate the Issue, Not the Structure: a Self-Correcting Pipeline for LLM-Generated Games

I run a small side project, Kirotopia, where an agent generates a complete HTML5 mini-game every morning — one prompt in, one playable game out, published to a public marketplace. Amazon Bedrock writes the game, a validation pipeline checks it, and an automated publisher ships it.

At least that was the theory. When I audited the marketplace, 6 of the 7 published games were broken. Not subtly broken — one had a fully transparent canvas and no game loop at all. Games that did not respond to touch. Games where eight seconds of active play changed nothing on screen.

The uncomfortable part: every one of those games had passed validation. My quality gate was green while the product was dead on arrival.

This post is about why that happened, and about the rebuild that followed. The lesson generalizes far beyond games: it applies to any pipeline that publishes LLM-generated artifacts — reports, dashboards, UI components, notebooks — behind automated checks.

Structural checks lie

The original validator ran eight structural checks. For each generated game it verified, roughly: a canvas element exists and has painted pixels, touch handlers are registered, a score variable is present, a game-over mechanism exists in the code, the file is self-contained, and so on.

A game that passed 8/8 looked like this in practice: the physics were inverted, and the player lost automatically less than two seconds after pressing Start. Every capability the checks looked for was present. None of it worked together.

That is the trap. Structural validation inventories capabilities: “a touch handler exists,” “a canvas is painted,” “a lose condition is implemented.” It never asks the only question the player asks: can I actually play this? An LLM is exceptionally good at producing code that has all the right parts — it has seen thousands of games and knows what one is supposed to contain. Whether those parts compose into a coherent, winnable game is exactly the property structural checks cannot see.

If you validate generated content, the rule I now apply everywhere: test the issue, not the structure. Replay the real usage, the way the real consumer will, and require the outcome the consumer requires.

Rebuilding the gate: playtest like a player

The validator now runs 12 checks, and the ones that matter drive a real headless browser (Playwright) through an actual play session. Three design decisions carried most of the value.

1. A progress-signal check instead of capability checks. The core new check plays the game for at least 8 seconds with varied input — taps in different zones, swipes, drags — and requires that something measurable moves: a score, a level, a lives counter, the score display text, or a win/lose screen. If nothing changes under active play, the game is incoherent regardless of what the code contains. This one check caught every “all parts present, nothing works” failure in the audit, including unreachable win conditions (collectibles spawned inside walls) and dead input wiring.

I initially planned a graph-reachability analysis (BFS over the maze to prove the win condition reachable). The generic progress-signal check made it unnecessary — replaying usage subsumes most bespoke coherence proofs, at a fraction of the complexity.

2. Hit-test your synthetic input. My first playtest harness “clicked the Start button” with a naive selector — and silently hit a button covered by an overlay. The scenario never actually started, and the harness reported results about a game it never played. The fix: before every synthetic tap, hit-test with elementFromPoint to confirm the tap will land on the element you think it will. A test harness that lies is worse than no harness.

3. A two-strike rule against difficulty false positives. Some games are just hard — dying once in 8 seconds is not a defect. Losing automatically, twice, without meaningful input is. Requiring two independent premature-failure observations before failing the check eliminated the false positives without letting the auto-lose games through.

The playtest runs on two viewports (phone and tablet), because the marketplace is consumed on iOS: a game that overflows the viewport or ships 20-pixel touch targets is broken for the actual audience, whatever the desktop rendering looks like.

Feed the constraints upstream: prompt contracts

A validator that rejects bad games is necessary but wasteful on its own — every rejection is a full generation you paid for. The second half of the fix is moving the lessons into the generation prompt as explicit contracts.

The prompt now carries two blocks. A mobile contract: touch-only input, full-screen fixed canvas, touch-action: none, controls at minimum 44 px (60 preferred), safe-area insets. And a coherence contract: the win condition must be reachable, mazes must be generated by carving a connected path (not by random wall placement), game state must visibly move within seconds of play, and difficulty must ramp — the player must be able to survive the first 10 seconds.

Each contract clause exists because a validator check failed on a real game. That is the loop worth building: every class of rejection becomes a named constraint in the prompt, so the generator stops producing that failure class instead of the validator catching it forever.

Close the loop: reject, retry, fail closed

With the gate and the contracts in place, generation becomes a self-correcting loop: generate, validate, and on failure regenerate with the failure report injected into the prompt.

The first live proof was satisfying. Asked for a Pac-Man-style game, the pipeline rejected three consecutive broken versions — each rejection citing the specific failed checks — and published the fourth, which passed 12/12. Total cost: a few extra generation calls and some latency. For a daily batch pipeline, an extra 10–30 seconds is nothing; shipping a dead game to real users is not.

Two publication-side rules complete the design:

• Fail closed. If validation fails after the retry budget, nothing is published. An empty slot in the marketplace is strictly better than a broken game — a lesson I learned the hard way when a misconfigured duplicate job once overwrote a working daily game with a completely dead one.
• Verify the consumer’s access path before announcing. The publisher checks that an anonymous (guest) session can actually fetch the uploaded asset before writing the catalog entry. Publishing is announcing; announce only what a real user can reach.

This is a general pattern, not a games story

Strip the games away and the architecture is three components, applicable to any generated-content pipeline:

1. A usage-replaying validator — drive the artifact the way its consumer will (render the report, load the dashboard, execute the notebook) and require the consumer’s outcome, not the presence of parts.
2. Contracts in the prompt — every recurring rejection class becomes an explicit generation constraint, so quality improves at the source.
3. A fail-closed publish loop — retries with failure feedback, and no partial announcements when the gate stays red.

The same failure mode shows up everywhere I have looked. A conversational agent that passes every single-turn eval and is completely amnesiac across turns. A generated report with perfect structure and a wrong headline number. In every case the fix is the same: stop inventorying capabilities and start replaying usage.

Honest recommendation

If you are building a pipeline that auto-publishes LLM-generated artifacts, do not start where I started. Skip the structural-check phase entirely — it produces green dashboards and broken products. Spend that effort on one end-to-end usage replay, even a crude one: it will catch more real defects than a dozen structural assertions, and every defect it catches hands you a constraint to feed back into the prompt. And make the publish step fail closed from day one. You will not regret the empty slot; you will regret the seven-for-seven audit.